Generating Pre-Signed (Query String Authentication) Amazon S3 Urls With User-Specific Data
Over the weekend, I started to look at using the Amazon Simple Storage Service (S3) with ColdFusion. One of the features that I looked at was generating pre-signed URLs in order to provide temporary access to the otherwise secure objects stored in your S3 buckets. When I found out about this feature, my first thought was, "How can I use user-specific data to generate the Query String Authentication (QSA) signature?" After learning more about S3, I don't think there is actually any real need to incorporate user-specific data in my pre-signed signatures; but, since I found nothing on Google, I figured I write up something quickly.
In my current app, requests for assets are routed through ColdFusion using URL rewriting. The ColdFusion app then looks at the request cookies, determines the requesting user, and checks the database to see if the given user has the proper permissions to access the given asset. If so, it serves up the asset using X-SendFile; if not, it returns a 404.
Coming from this context, I think it was a natural inclination to want to incorporate user-specific data into my S3 pre-signed URLs. So, I tried adding the "userID" value to the URL. But, Amazon ignores any "normal" query string parameters that you attach to your pre-signed URLs.
After much reading and experimentation, I came to figure out that the solution was to use Amazon meta-headers. That is, to use headers in the form of:
"x-amz-meta-"
These meta-headers allow you to apply custom data to your authorized requests. Amazon won't actually validate the value of the header; but, it will incorporate the header in the expected signature.
With a non-GET style request, these Amazon headers would be sent as HTTP headers; however, in a GET request, such as that being used in the pre-signed URLs, we can't rely on the client to define HTTP headers. As such, Amazon allows these headers to be added to the query string of the pre-signed URLs.
So, if I want to incorporate user-specific data in my Amazon S3 pre-signed URLs, I have to use it in both the signature generation and in the URL generation. To see this in action, take a look at the following code - I am incorporating the Amazon header, "x-amz-meta-user-id" and the actual ID of the user making the request:
<!---
Creates a structure with the secretKey and accessID so that I
don't have to have them in the blog post.
--->
<cfinclude template="../credentials.cfm" />
<!---
This is the resource for which we want to generate an access URL.
This URL will only be viable for an explicit amount of time.
--->
<cfset resource = "/testing.bennadel.com/signed-urls/helena.jpg" />
<!--- ----------------------------------------------------- --->
<!--- ----------------------------------------------------- --->
<!--- The URL will only be valid for a short amount of time. --->
<cfset nowInSeconds = fix( now().getTime() / 1000 ) />
<!--- Add 10 seconds. --->
<cfset expirationInSeconds = ( nowInSeconds + 10 ) />
<!---
When preparing the string for the Hmac, we might normally leave
the "Amazon Headers" portion blank. In this case, however, I'm
going to add the "x-amz-meta-" values that we are ALSO going to
be passing in the pre-signed URL.
In this case, we are passing in the USER ID of the user requesting
access to the resource.
--->
<cfset stringToSignParts = [
"GET",
"",
"",
expirationInSeconds,
"x-amz-meta-user-id:4",
resource
] />
<!--- Collapse the parts into a newline-delimited list. --->
<cfset stringToSign = arrayToList( stringToSignParts, chr( 10 ) ) />
<!--- Generate the Hmac-Sha-1 signature. --->
<cfset signature = new Crypto().hmacSha1(
aws.secretKey,
stringToSign,
"base64"
) />
<!---
Make sure the signature is properly encoded for use in the query
string of a GET request.
--->
<cfset urlEncodedSignature = urlEncodedFormat( signature ) />
<!--- ----------------------------------------------------- --->
<!--- ----------------------------------------------------- --->
<!---
Since the URL is rather long, I'll build it as a list (for easier
viewing). Notice that we have to include the "x-amz-meta-user-id"
value in the URL since have ALSO included it in the signature
generation.
NOTE: The "Amazon Header" values in the query string (ie.
x-amz-meta-user-id) must be in all lowercase to match the
normalized casing used to calculate the Hmac-Sha1 signature.
--->
<cfset urlParts = [
"https://s3.amazonaws.com#resource#?AWSAccessKeyId=#aws.accessID#",
"Expires=#expirationInSeconds#",
"Signature=#urlEncodedSignature#",
"x-amz-meta-user-id=4"
] />
<cfoutput>
<img src="#arrayToList( urlParts, "&" )#" />
</cfoutput>
This code results in a pre-signed URL, based on a given User ID, that is valid for 10 seconds.
Now, with all that said, does it actually make sense to incorporate user-specific data in your Amazon S3 pre-signed URLs? For logging purposes, yes; while Amazon won't validate "x-amz-meta" headers (other than in the context of the URL signature), it will add the values to log files (from what I've read). As such, you can use these meta-headers to make your S3 logs a bit more insightful.
But, logging wasn't my original intent. As I explained before, I am coming from a user-specific request context in which each request is evaluated by my own ColdFusion application. But with Amazon S3, I don't get to evaluate the request. Once I generate the pre-signed URLs, the rest of that request lifecycle is out of my hands. If I create a URL for a user and that user gives it to a friend, I won't know about it. At that point, the only thing I can do is leverage the expiration date of the pre-signed URLs in order to limit possible exposure.
So, in the end, other than for logging purposes, incorporating user-specific data in a pre-signed URL is probably not worthwhile. All it does is make the URL more complex without adding any additional security measures.
Want to use code from this post? Check out the license.
Reader Comments
Hi Ben,
What is credentials.cfm?
Rgds
@Nc,
That just contains my Amazon AWS bucket, access ID, and secret key - which I don't want in my actual code (for security purposes).
Hi Ben,
Thanks for your post; found it while implementing S3 pre-signed URLs with very short expiration times and noticed this: The URLs didn't expire consistently. I was using times from 5-10 seconds and saw them expire as late as 1 minute, very inconsistent. Did you notice anything like this with your 10-second expiration?
Thanks -Scott