Exploring Cloudflare R2 And Request Authorization Using AWS Signature V4
Once I rebuilt my Incident Commander app in ColdFusion, I finally had the ability to upload images and screenshots as supporting evidence of the incident triage investigation. Right now, those uploads are saved to the server—it's what makes the most sense in a free MVP (minimum viable product). In the long run, I'd prefer to save uploads to a remote object store like Amazon Web Services (AWS) S3 or Cloudflare R2.
Both of these platforms use a complex signature for request authorization. Years ago, I looked at generating this authorization signature using the V2 algorithm. But, the V2 algorithm is deprecated in favor of the more robust—and more complex—V4 algorithm. As such, I wanted to look at how the V4 algorithm can be implemented in ColdFusion.
Aside: Cloudflare R2 has a mostly S3-compatible API. Which means, any client library or SDK (Software Developer Kit) that performs basic operations against S3 can also be used to consume R2. Which means, you probably won't ever have to calculate this signature on your own. But, I like to build things to better understand how they work; and, to keep my mind active.
Calculating the V4 signature is rather laborious and contains many steps:
Build up a canonical representation of the request including the case-sensitive normalization and sorting of query-string and header keys.
Hash the canonical representation of the request and fold it into a scoped representation of the request (scoped to the target service, date, and signing algorithm).
Construct a signing key by folding the secret key into a 4-step reduction that uses the
hmac()function to generate an intermediary key for the next step in the reduction.Use the secret key reduction to generate a signature for the scoped representation of the request.
Use the signature to generate an
Authorizationheader or a pre-signed URL.
To their credit, the AWS documentation provides a walk-through of the above algorithm; and shows you what the inputs and outputs of each step look like. This provided me with a way to create a light-weight test harness for the signing process.
The AWS documentation provides four examples of the Authorization-based signing and one example of the query-string-based signing. I've boiled these down to a set of inputs, a method name, and an expected output. I then setup a test runner to apply these assertions against my V4 signing ColdFusion component (AwsSignatureVersion4Testing.cfc):
<cfscript>
assertions = [
// API operation examples from:
// https://docs.aws.amazon.com/AmazonS3/latest/API/sig-v4-header-based-auth.html
{
methodName: "getAuthorizationHeader",
arguments: {
httpMethod: "GET",
resourcePath: "test.txt",
searchParams: {},
responseOverrideParams: {},
headers: {
"Host": "examplebucket.s3.amazonaws.com",
"Range": "bytes=0-9",
"X-AMZ-Date": "20130524T000000Z",
"x-amz-Content-Sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
},
payloadHash: "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", // Empty hash (constant).
timestamp: "20130524T000000Z"
},
expects: "AWS4-HMAC-SHA256 Credential=AKIAIOSFODNN7EXAMPLE/20130524/us-east-1/s3/aws4_request,SignedHeaders=host;range;x-amz-content-sha256;x-amz-date,Signature=f0e8bdb87c964420e857bd35b5d6ed310bd44f0170aba48dd91039c6036bdb41"
},
{
methodName: "getAuthorizationHeader",
arguments: {
httpMethod: "PUT",
resourcePath: "test$file.text",
searchParams: {},
responseOverrideParams: {},
headers: {
"Host": "examplebucket.s3.amazonaws.com",
"Date": "Fri, 24 May 2013 00:00:00 GMT",
"X-AMZ-Date": "20130524T000000Z ",
"X-AMZ-Storage-Class": "REDUCED_REDUNDANCY",
"X-AMZ-content-SHA256": "44ce7dd67c959e0d3524ffac1771dfbba87d2b6b4b4e99e42034a8b803f8b072"
},
payloadHash: "44ce7dd67c959e0d3524ffac1771dfbba87d2b6b4b4e99e42034a8b803f8b072",
timestamp: "20130524T000000Z"
},
expects: "AWS4-HMAC-SHA256 Credential=AKIAIOSFODNN7EXAMPLE/20130524/us-east-1/s3/aws4_request,SignedHeaders=date;host;x-amz-content-sha256;x-amz-date;x-amz-storage-class,Signature=98ad721746da40c64f1a55b78f14c238d841ea1380cd77a1b5971af0ece108bd"
},
{
methodName: "getAuthorizationHeader",
arguments: {
httpMethod: "GET",
resourcePath: "",
searchParams: {
"lifecycle": ""
},
responseOverrideParams: {},
headers: {
"Host": "examplebucket.s3.amazonaws.com",
"X-AMZ-Date": "20130524T000000Z ",
"X-AMZ-Content-Sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
},
payloadHash: "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", // Empty hash (constant).
timestamp: "20130524T000000Z"
},
expects: "AWS4-HMAC-SHA256 Credential=AKIAIOSFODNN7EXAMPLE/20130524/us-east-1/s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date,Signature=fea454ca298b7da1c68078a5d1bdbfbbe0d65c699e0f91ac7a200a0136783543"
},
{
methodName: "getAuthorizationHeader",
arguments: {
httpMethod: "GET",
resourcePath: "",
searchParams: {
"max-keys": 2,
"prefix": "J"
},
responseOverrideParams: {},
headers: {
"Host": "examplebucket.s3.amazonaws.com",
"X-AMZ-Date": "20130524T000000Z ",
"X-AMZ-content-Sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
},
payloadHash: "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", // Empty hash (constant).
timestamp: "20130524T000000Z"
},
expects: "AWS4-HMAC-SHA256 Credential=AKIAIOSFODNN7EXAMPLE/20130524/us-east-1/s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date,Signature=34b48302e7b5fa45bde8084f4b7868a86f0a534bc59db6670ed5711ef69dc6f7"
},
// Query string (presigned URL) examples from:
// https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-query-string-auth.html
{
methodName: "getPresignedUrl",
arguments: {
httpMethod: "get",
resourcePath: "/test.txt",
responseOverrideParams: {},
expiresInSeconds: 86400,
timestamp: "20130524T000000Z"
},
expects: "https://examplebucket.s3.amazonaws.com/test.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIOSFODNN7EXAMPLE%2F20130524%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20130524T000000Z&X-Amz-Expires=86400&X-Amz-SignedHeaders=host&X-Amz-Signature=aeeed9bbccd4d02ee5c0109b86d86835f995330da4c265957d157751f604d404"
}
];
// ------------------------------------------------------------------------------- //
// ------------------------------------------------------------------------------- //
// Hard-coded settings for the authorization examples.
testHarness = new AwsSignatureVersion4Testing(
accessID = "AKIAIOSFODNN7EXAMPLE",
secretKey = "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
host = "examplebucket.s3.amazonaws.com",
region = "us-east-1"
);
for ( assertion in assertions ) {
calculated = invoke( testHarness, assertion.methodName, assertion.arguments );
// Test the calculated response against known response using CASE-SENSITIVE op!
if ( ! compare( calculated, assertion.expects ) ) {
statusEmoji = canonicalize( "&##x2705;", false, false );
statusText = "PASS";
} else {
statusEmoji = canonicalize( "&##x274C;", false, false );
statusText = "FAIL";
}
writeDump([
status: (
repeatString( "#statusEmoji# ", 10 ) &
statusText &
repeatString( " #statusEmoji#", 10 )
),
// arguments: assertion.arguments,
calculated: calculated,
expects: assertion.expects
]);
}
</cfscript>
In this workflow, I'm either passing the arguments to the getAuthorizationHeader() method or the getPresignedUrl() method. And then, I perform a case-sensitive comparison to the expected result; and mark the test as either passed or failed. When we run this ColdFusion code, we get the following output:
All five tests pass—they produce the correct signature based on the given inputs. But, this test harness doesn't exercise all of the nuances that go into the V4 signing process. For example, none of them deal with request overrides of the object response.
The request overrides are represented as a secondary set of query string search parameters which must be incorporated into the signing process. But, they can't be mixed in with the core query string parameters. Instead, they have to be sorted on their own and then appended to the end of the canonical query string (within the canonical request calculation).
To put the signing process through a more real-world scenario, I created another test harness that performs three steps:
PUTan object to my test Cloudflare R2 bucket.GETsaid object and store it back to my ColdFusion server.Render said object as an
<img>tag using a query string authenticated, presigned URL complete with some response overrides.
Here's the ColdFusion code for this test. Note that Cloudflare R2 doesn't use "region" like AWS does; as such, the region is always auto:
<cfscript>
// The provisioned Read/Write object token.
accountID = "0f8e14185b412fbeeb1fa436e5dba78e";
accessID = "ce356bcd498f1c2b07e5f342a5227b75";
host = "bennadel-testing.#accountID#.r2.cloudflarestorage.com";
region = "auto";
// -- SECURITY ALERT --
// This is my SECRET KEY for Cloudflare request signing. This key SHOULD NEVER BE
// SHARED PUBLICLY. However, this is an API token that I've provisioned specifically
// for this demo; and which will be revoked after this demo. As such, it's ok to share
// here in the context of the demo.
secretKey = "[redacted for blog post anyway]";
// ^^-- SECURITY ALERT -^^
testHarness = new AwsSignatureVersion4Testing(
accessID = accessID,
secretKey = secretKey,
host = host,
region = region
);
// Testing PUT.
putObjectFromFile(
resourcePath = "/images/dog.jpg",
filePath = expandPath( "./files/dog.jpg" )
);
// Testing GET.
getObjectToFile(
resourcePath = "/images/dog.jpg",
filePath = expandPath( "./files/dog-copy.jpg" )
);
// Testing query string authorization.
signedUrl = getPresignedUrl( "/images/dog.jpg", 20 );
writeOutput( "<img src='#signedUrl#' width='300'>" );
// ------------------------------------------------------------------------------- //
// ------------------------------------------------------------------------------- //
/**
* I get the Cloudflare R2 object and store it in the given file.
*/
private any function getObjectToFile(
required string resourcePath,
required string filePath
) {
fileWrite( filePath, getObject( resourcePath ) );
}
/**
* I get the Cloudflare R2 object and return it as a binary value.
*/
private binary function getObject( required string resourcePath ) {
resourcePath = testHarness.normalizePath( resourcePath );
// For GET, we use the empty payload hash.
var payloadHash = hash( "", "sha-256" ).lcase()
var timestamp = utcNow().dateTimeFormat( "yyyymmdd'T'HHnnss'Z'" );
var httpMethod = "get";
var searchParams = {};
// Note: I'm just putting these overrides in here to make sure that they don't
// break the request. I don't actually care about them - and they don't actually
// affect the binary that is returned, only the HTTP response headers.
var responseOverrideParams = {
"response-content-disposition": "inline; filename=""override.jpg"";",
"response-content-type": "application/octet-stream"
};
var headers = {
"X-AMZ-Date": timestamp,
"X-AMZ-Content-Sha256": payloadHash
};
cfhttp(
result = "local.httpResponse",
method = httpMethod,
url = "https://#host##resourcePath#",
getAsBinary = "yes"
) {
cfhttpparam(
type = "header",
name = "Authorization",
value = testHarness.getAuthorizationHeader(
httpMethod = httpMethod,
resourcePath = resourcePath,
searchParams = searchParams,
responseOverrideParams = responseOverrideParams,
headers = headers,
payloadHash = payloadHash,
timestamp = timestamp
)
);
for ( var key in headers ) {
cfhttpparam(
type = "header",
name = key,
value = headers[ key ]
);
}
for ( var key in responseOverrideParams ) {
cfhttpparam(
type = "url",
name = key,
value = responseOverrideParams[ key ]
);
}
}
// Note: for the sake of the exploration, I'm not bothering with any response
// validation, normalization, or retry mechanics.
return httpResponse.fileContent;
}
/**
* I put the Cloudflare R2 object from the given file.
*/
private void function putObjectFromFile(
required string resourcePath,
required string filePath
) {
resourcePath = testHarness.normalizePath( resourcePath );
var payload = fileReadBinary( filePath );
var payloadHash = hash( payload, "sha-256" ).lcase();
var timestamp = utcNow().dateTimeFormat( "yyyymmdd'T'HHnnss'Z'" );
var httpMethod = "put";
var searchParams = {};
var responseOverrideParams = {};
var headers = {
"Content-Type": "image/jpeg",
"X-AMZ-Date": timestamp,
"X-AMZ-Content-Sha256": payloadHash,
// Any custom data can be added with "x-amz-meta-" prefix.
"x-amz-meta-author": "Benny Kablamo",
"x-amz-meta-version": "v12"
};
cfhttp(
result = "local.httpResponse",
method = httpMethod,
url = "https://#host##resourcePath#",
getAsBinary = "yes"
) {
cfhttpparam(
type = "header",
name = "Authorization",
value = testHarness.getAuthorizationHeader(
httpMethod = httpMethod,
resourcePath = resourcePath,
searchParams = searchParams,
responseOverrideParams = responseOverrideParams,
headers = headers,
payloadHash = payloadHash,
timestamp = timestamp
)
);
for ( var key in headers.keyArray() ) {
cfhttpparam(
type = "header",
name = key,
value = headers[ key ]
);
}
cfhttpparam(
type = "body",
value = payload
);
}
// Note: for the sake of the exploration, I'm not bothering with any response
// validation, normalization, or retry mechanics.
}
/**
* I get a presigned URL that grants temporary access to the given Cloudflare object.
*/
private string function getPresignedUrl(
required string resourcePath,
required numeric expiresInSeconds
) {
return testHarness.getPresignedUrl(
httpMethod = "get",
resourcePath = testHarness.normalizePath( resourcePath ),
responseOverrideParams = {
"response-content-disposition": "inline; filename=""override.jpg"";",
"response-cache-control": "max-age=86400"
},
expiresInSeconds = expiresInSeconds,
timestamp = utcNow().dateTimeFormat( "yyyymmdd'T'HHnnss'Z'" )
);
}
/**
* I provide the current date/time in UTC.
*/
private date function utcNow() {
return dateConvert( "local2utc", now() );
}
</cfscript>
This ColdFusion code doesn't perform any validation of the CFHttp response; and, it doesn't contain any retry mechanics for failed requests. But, it does put the V4 signing algorithm through a more robust set of tests. And, when we run this ColdFusion code, we get the following output:
As you can see, we were able to upload (PUT) the image up to Cloudflare R2 and then render it using the presigned URL. And, from the network activity, we can see that the response overrides were successfully incorporated into the presigned URL.
Now that we know that everything is working as expected, here's my ColdFusion implementation of the AWS S3 / Cloudflare R2 V4 Signature algorithm. I'm sure there's a lot of room for performance improvement; but, I love the adage, "make it work, make it right, make it fast."
component
output = false
hint = "I provide a set of methods that contain implementation details that run against the AWS signature v4 examples from the AWS documentation."
{
/**
* I initialize the testing component.
*/
public void function init(
required string accessID,
required string secretKey,
required string host,
required string region
) {
variables.accessID = arguments.accessID;
variables.secretKey = arguments.secretKey;
variables.host = arguments.host;
variables.region = arguments.region;
variables.algorithm = "AWS4-HMAC-SHA256";
variables.newline = chr( 10 );
}
// ---
// PUBLIC METHODS.
// ---
/**
* I get the HTTP Authorization header for the given operation.
*/
public string function getAuthorizationHeader(
required string httpMethod,
required string resourcePath,
required struct searchParams,
required struct responseOverrideParams,
required struct headers,
required string payloadHash,
required string timestamp // compact ISO format.
) {
var dateString = left( timestamp, 8 );
// Build up the canonical request.
var canonicalHttpMethod = buildCanonicalHttpMethod( httpMethod );
var canonicalUri = buildCanonicalUri( resourcePath );
var canonicalQueryString = buildCanonicalQueryString( searchParams, responseOverrideParams );
var canonicalHeaders = buildCanonicalHeaders( headers );
var canonicalSignedHeaders = buildCanonicalSignedHeaders( headers );
var canonicalRequest = arrayTolist(
[
canonicalHttpMethod,
canonicalUri,
canonicalQueryString,
canonicalHeaders,
canonicalSignedHeaders,
payloadHash
],
newline
);
var canonicalRequestHash = hash( canonicalRequest, "sha-256" )
.lcase()
;
// Build up the string to sign.
var credentialScope = "#dateString#/#region#/s3/aws4_request";
var stringToSign = arrayToList(
[
algorithm,
timestamp,
credentialScope,
canonicalRequestHash
],
newline
);
// Build up the signature.
var signingKey = buildSigningKey( dateString );
var signature = hmac( stringToSign, signingKey, "HmacSHA256" ).lcase();
// Build up the Authorization header.
return "#algorithm# Credential=#accessID#/#credentialScope#,SignedHeaders=#canonicalSignedHeaders#,Signature=#signature#";
}
/**
* I get the presigned URL (query-string authorized) for the given operation.
*/
public string function getPresignedUrl(
required string httpMethod,
required string resourcePath,
required struct responseOverrideParams,
required numeric expiresInSeconds,
required string timestamp // Compact ISO format.
) {
var dateString = left( timestamp, 8 );
var credentialScope = "#dateString#/#region#/s3/aws4_request";
var searchParams = {
"X-Amz-Algorithm": algorithm,
"X-Amz-Credential": "#accessID#/#credentialScope#",
"X-Amz-Date": timestamp,
"X-Amz-Expires": expiresInSeconds,
"X-Amz-SignedHeaders": "host"
};
var headers = {
"host": host
};
// Build up the canonical request.
var canonicalHttpMethod = buildCanonicalHttpMethod( httpMethod );
var canonicalUri = buildCanonicalUri( resourcePath );
var canonicalQueryString = buildCanonicalQueryString( searchParams, responseOverrideParams );
var canonicalHeaders = buildCanonicalHeaders( headers );
var canonicalSignedHeaders = buildCanonicalSignedHeaders( headers );
var canonicalRequest = arrayTolist(
[
canonicalHttpMethod,
canonicalUri,
canonicalQueryString,
canonicalHeaders,
canonicalSignedHeaders,
"UNSIGNED-PAYLOAD"
],
newline
);
var canonicalRequestHash = hash( canonicalRequest, "sha-256" )
.lcase()
;
// Build up the string to sign.
var stringToSign = arrayToList(
[
algorithm,
timestamp,
credentialScope,
canonicalRequestHash
],
newline
);
// Build up the signature.
var signingKey = buildSigningKey( dateString );
var signature = hmac( stringToSign, signingKey, "HmacSHA256" ).lcase();
return "https://#host##canonicalUri#?#canonicalQueryString#&X-Amz-Signature=#awsEncodeUriComponent( signature )#";
}
/**
* I normalize the given path for use in object operations. Providing this as a public
* method allows the external use of paths to align with the internal signing.
*/
public string function normalizePath( required string resourcePath ) {
return buildCanonicalUri( resourcePath );
}
// ---
// PRIVATE METHODS.
// ---
/**
* I encoded the given input as a URI.
*/
private string function awsEncodeUri( required string input ) {
// For AWS, we have to keep literal slashes in the path.
return awsEncodeUriComponent( input )
.replace( "%2F", "/", "all" )
;
}
/**
* I encoded the given input as a URI component.
*/
private string function awsEncodeUriComponent( required string input ) {
// Note: In older implementations of ColdFusion, where we used urlEncodedFormat(),
// the encoding was too aggressive and we ended up having to decode some of the
// patterns. The encodeForUrl() is more spec-compliant. As such, we don't have to
// back-out of _most_ of the encodings.
return encodeForUrl( input )
.replace( "+", "%20", "all" )
;
}
/**
* I build the headers for the canonical request.
*/
private string function buildCanonicalHeaders( required struct headers ) {
return headers
.keyArray()
.sort( "textnocase" )
.map(
( key ) => {
// Note: every header entry must end with a newline, even the last
// entry. As such, we have to include it in each mapping and can't
// rely on the toList() operation delimiter.
return "#key.lcase()#:#trim( headers[ key ] )##chr( 10 )#";
}
)
.toList( "" )
;
}
/**
* I build the http method for the canonical request.
*/
private string function buildCanonicalHttpMethod( required string httpMethod ) {
return httpMethod.ucase();
}
/**
* I build the query string for the canonical request.
*/
private string function buildCanonicalQueryString(
required struct searchParams,
required struct responseOverrideParams
) {
var baseQueryString = searchParams
.keyArray()
.sort( "textnocase" )
.map(
( key ) => {
return "#awsEncodeUriComponent( key )#=#awsEncodeUriComponent( searchParams[ key ] )#";
}
)
.toList( "&" )
;
var overrideQueryString = responseOverrideParams
.keyArray()
.sort( "textnocase" )
.map(
( key ) => {
return "#awsEncodeUriComponent( key )#=#awsEncodeUriComponent( responseOverrideParams[ key ] )#";
}
)
.toList( "&" )
;
if ( baseQueryString.len() && overrideQueryString.len() ) {
// Note: When override the HTTP response headers using the override params,
// the set of overrides must come AFTER the core set of query string params
// (and each set of params is ordered independently).
return "#baseQueryString#&#overrideQueryString#";
}
// Note: there's no "&" delimiter since we know we only have one of the values.
// This way, we don't have to care which one is populated.
return "#baseQueryString##overrideQueryString#";
}
/**
* I build the signed header for the canonical request.
*/
private string function buildCanonicalSignedHeaders( required struct headers ) {
return headers
.keyArray()
.sort( "textnocase" )
.toList( ";" )
.lcase()
;
}
/**
* I build the request URI for the canonical request.
*/
private string function buildCanonicalUri( required string input ) {
return awsEncodeUri( "/" & input )
// Technically, the object paths are allowed to have nonsensical slashes in
// them because the buckets only "looks like" a directory structure. In
// reality, they act more like a key-value store so the "path" is really just
// an arbitrary "key" and doesn't care if there are crazy slashes. That said,
// allowing crazy slashes will almost certainly cause more issues in the long-
//run and should be avoided.
.reReplace( "/{2,}", "/", "all" )
;
}
/**
* I iteratively build up the signing key starting with the given date.
*/
private binary function buildSigningKey( required string dateString ) {
// Derive the signing key by folding the secret key across several inputs in which
// each subsequent key is computed by the previous hmac() operation. The
// ColdFusion hmac() function returns a HEX-encoded value. But, in order for this
// to work, we have to make sure that we decode the intermediary keys back into
// binary values before we perform the next hmac() operation.
return arrayReduce(
[
dateString,
region,
"s3",
"aws4_request"
],
( reduction, input ) => {
return binaryDecode( hmac( input, reduction, "HmacSHA256" ), "hex" );
},
charsetDecode( "AWS4#secretKey#", "utf-8" )
);
}
}
In the end, this whole post is mostly for my own reference. I was hoping that Cloudflare R2 would have a dramatically simplified model for authorization; but, I suppose being AWS S3 compatible makes the R2 product more appealing to an entrenched developer-base.
I might try incorporating R2 into my Incident Commander (IC) app since they have a fairly generous free tier (which is suitable for my free app). I'd have to first clean-up my IC code—the file system details leak too far into the logic of the current application. I'd have to create a better abstraction before I can change the underlying implementation.
Want to use code from this post? Check out the license.
Reader Comments
Post A Comment — ❤️ I'd Love To Hear From You! ❤️
Post a Comment →