Uploading Files To Amazon S3 Using Plupload And ColdFusion
Earlier this week, I posted an experiment in which I used an ordinary Form POST to upload files directly to Amazon's Simple Storage Service (S3) without using my server as an upload proxy. Once I had that working, I really wanted to see if I could do the same thing using Plupload. I've talked about Plupload's excellent drag-and-drop file uploads before; but, I was uploading files directly to my server. Now, with Amason's POST functionality, I want to do the same thing, only using my S3 bucket as my target URL.
Project: View this project on my GitHub.
The Plupload repository has an Amazon S3 demo; but, even with the demo in hand, it took me a few days to get this working. I kept running into small hurdles around data serialization, unique naming, policy generation, redirects, etc.. Finally, however, I think I came up with a stable solution that seems to work with everything that I've thrown at it.
In the following demo, I have it programmed to only accept image files. This filtering happens at the JavaScript level (ignoring the selection of non-image files); and, it happens at the Amazon level - uploading non-image files will cause Amazon to reject the POST due to the specifications in the Policy document.
Once Amazon has stored the image, it returns the location information in an XML response. The JavaScript must then parse this XML document and report the upload information back to the ColdFusion server; remember, now that you're uploading directly from the client to Amazon S3, you have to use the client as a sort of go-between for your ColdFusion server and the S3 bucket.
NOTE: Using a normal Form POST allows for a "success redirect", which removes this communication burden from the client (to some degree).
In this demo, the "upload report" is being implemented as an IMG src attribute. This image URL both reports the upload to the ColdFusion server and then consumes a pre-signed query string request authentication URL to the private S3 resource.
That said, here is the main page of the demo - the one that generates the Amazon S3 upload Policy and then renders the Plupload widget. Remember, since this user-interface (UI) maybe be visible for a good amount of time without any client-side page refresh, we have to create a distant expiration date for the Policy. Welcome to the world of rich client-side applications.
index.cfm - Our Main Plupload Demo
<cfscript>
// Include our ColdFusion 9 -> ColdFusion 10 migration script so
// that I can work on this at home (CF10) and in the office (CF9).
include "cf10-migration.cfm";
// Include the Amazon Web Service (AWS) S3 credentials.
include "aws-credentials.cfm";
// The expiration must defined in UCT time. Since the Plupload
// widget may be on the screen for a good amount of time,
// especially if this is a single-page app, we probably need to
// put the expiration date into the future a good amount.
expiration = dateConvert( "local2utc", dateAdd( "d", 1, now() ) );
// NOTE: When formatting the UTC time, the hours must be in 24-
// hour time; therefore, make sure to use "HH", not "hh" so that
// your policy don't expire prematurely.
// ---
// NOTE: We are providing a success_action_status INSTEAD of a
// success_action_redirect since we don't want the browser to try
// and redirect (won't be supported across all Plupload
// environments). Instead, we'll get Amazon S3 to return the XML
// document for the successful upload. Then, we can parse teh
// response locally.
policy = {
"expiration" = (
dateFormat( expiration, "yyyy-mm-dd" ) & "T" &
timeFormat( expiration, "HH:mm:ss" ) & "Z"
),
"conditions" = [
{
"bucket" = aws.bucket
},
{
"acl" = "private"
},
{
"success_action_status" = "2xx"
},
[ "starts-with", "$key", "pluploads/" ],
[ "starts-with", "$Content-Type", "image/" ],
[ "content-length-range", 0, 10485760 ], // 10mb
// The following keys are ones that Plupload will inject
// into the form-post across the various environments.
[ "starts-with", "$Filename", "pluploads/" ],
[ "starts-with", "$name", "" ]
]
};
// ------------------------------------------------------ //
// ------------------------------------------------------ //
// The policy will be posted along with the FORM post as a
// hidden form field. Serialize it as JavaScript Object notation.
serializedPolicy = serializeJson( policy );
// When the policy is being serialized, ColdFusion will try to turn
// "201" into the number 201. However, we NEED this value to be a
// STRING. As such, we'll give the policy a non-numeric value and
// then convert it to the appropriate 201 after serialization.
serializedPolicy = replace( serializedPolicy, "2xx", "201" );
// Remove up the line breaks.
serializedPolicy = reReplace( serializedPolicy, "[\r\n]+", "", "all" );
// Encode the policy as Base64 so that it doesn't mess up
// the form post data at all.
encodedPolicy = binaryEncode(
charsetDecode( serializedPolicy, "utf-8" ) ,
"base64"
);
// ------------------------------------------------------ //
// ------------------------------------------------------ //
// To make sure that no one tampers with the FORM POST, create
// hashed message authentication code of the policy content.
// NOTE: The hmac() function was added in ColdFusion 10.
hashedPolicy = hmac(
encodedPolicy,
aws.secretKey,
"HmacSHA1",
"utf-8"
);
// Encode the message authentication code in Base64.
encodedSignature = binaryEncode(
binaryDecode( hashedPolicy, "hex" ),
"base64"
);
</cfscript>
<!--- ----------------------------------------------------- --->
<!--- ----------------------------------------------------- --->
<!--- Reset the output buffer and set the page encoding. --->
<cfcontent type="text/html; charset=utf-8" />
<cfoutput>
<!doctype html>
<html>
<head>
<meta charset="utf-8" />
<title>
Uploading Files To Amazon S3 Using Plupload And ColdFusion
</title>
<link rel="stylesheet" type="text/css" href="./assets/css/styles.css"></link>
</head>
<body>
<h1>
Uploading Files To Amazon S3 Using Plupload And ColdFusion
</h1>
<div id="uploader" class="uploader">
<a id="selectFiles" href="##">
<span class="label">
Select Files
</span>
<span class="standby">
Waiting for files...
</span>
<span class="progress">
Uploading - <span class="percent"></span>%
</span>
</a>
</div>
<ul class="uploads">
<!--
Will be populated dynamically with LI/IMG tags by the
uploader success handler.
-->
</ul>
<!-- Load and initialize scripts. -->
<script type="text/javascript" src="./assets/jquery/jquery-2.0.3.min.js"></script>
<script type="text/javascript" src="./assets/plupload/js/plupload.full.js"></script>
<script type="text/javascript">
(function( $, plupload ) {
// Find and cache the DOM elements we'll be using.
var dom = {
uploader: $( "##uploader" ),
percent: $( "##uploader span.percent" ),
uploads: $( "ul.uploads" )
};
// Instantiate the Plupload uploader. When we do this,
// we have to pass in all of the data that the Amazon
// S3 policy is going to be expecting. Also, we have
// to pass in the policy :)
var uploader = new plupload.Uploader({
// Try to load the HTML5 engine and then, if that's
// not supported, the Flash fallback engine.
// --
// NOTE: For Flash to work, you will have to upload
// the crossdomain.xml file to the root of your
// Amazon S3 bucket.
runtimes: "html5,flash",
// The upload URL - our Amazon S3 bucket.
url: "http://#aws.bucket#.s3.amazonaws.com/",
// The ID of the drop-zone element.
drop_element: "uploader",
// To enable click-to-select-files, you can provide
// a browse button. We can use the same one as the
// drop zone.
browse_button: "selectFiles",
// For the Flash engine, we have to define the ID
// of the node into which Pluploader will inject the
// <OBJECT> tag for the flash movie.
container: "uploader",
// The URL for the SWF file for the Flash upload
// engine for browsers that don't support HTML5.
flash_swf_url: "./assets/plupload/js/plupload.flash.swf",
// Needed for the Flash environment to work.
urlstream_upload: true,
// NOTE: Unique names doesn't work with Amazon S3
// and Plupload - see the BeforeUpload event to see
// how we can generate unique file names.
// unique_names: true,
// The name of the form-field that will hold the
// upload data. Amason S3 will expect this form
// field to be called, "file".
file_data_name: "file",
multipart: true,
// Pass through all the values needed by the Policy
// and the authentication of the request.
// --
// NOTE: We are using the special value, ${filename}
// in our param definitions; but, we are actually
// overriding these in the BeforeUpload event. This
// notation is used when you do NOT know the name
// of the file that is about to be uploaded (and
// therefore cannot define it explicitly).
multipart_params: {
"acl": "private",
"success_action_status": "201",
"key": "pluploads/${filename}",
"Filename": "pluploads/${filename}",
"Content-Type": "image/*",
"AWSAccessKeyId" : "#aws.accessID#",
"policy": "#encodedPolicy#",
"signature": "#encodedSignature#"
},
max_file_size : "10mb" // 10485760 bytes
});
// Set up the event handlers for the uploader.
uploader.bind( "Init", handlePluploadInit );
uploader.bind( "Error", handlePluploadError );
uploader.bind( "FilesAdded", handlePluploadFilesAdded );
uploader.bind( "QueueChanged", handlePluploadQueueChanged );
uploader.bind( "BeforeUpload", handlePluploadBeforeUpload );
uploader.bind( "UploadProgress", handlePluploadUploadProgress );
uploader.bind( "FileUploaded", handlePluploadFileUploaded );
uploader.bind( "StateChanged", handlePluploadStateChanged );
// Initialize the uploader (it is only after the
// initialization is complete that we will know which
// runtime load: html5 vs. Flash).
uploader.init();
// ------------------------------------------ //
// ------------------------------------------ //
// I return the content type based on the filename.
function getContentTypeFromFilename( name ) {
if ( /\.jpe?g$/i.test( name ) ) {
return( "image/jpg" );
} else if ( /\.png/i.test( name ) ) {
return( "image/png" );
} else if ( /\.gif/i.test( name ) ) {
return( "image/gif" );
}
// If none of the known image types match, then
// just use the default octet-stream.
return( "application/octet-stream" );
}
// I handle the before upload event where the meta data
// can be edited right before the upload of a specific
// file, allowing for per-file Amazon S3 settings.
function handlePluploadBeforeUpload( uploader, file ) {
console.log( "File upload about to start." );
// Generate a "unique" key based on the file ID
// that Plupload has assigned. This way, we can
// create a non-colliding directory, but keep
// the original file name from the client.
var uniqueKey = ( "pluploads/" + file.id + "/" + file.name );
// Update the Key and Filename so that Amazon S3
// will store the resource with the correct value.
uploader.settings.multipart_params.key = uniqueKey;
uploader.settings.multipart_params.Filename = uniqueKey;
}
// I handle the init event. At this point, we will know
// which runtime has loaded, and whether or not drag-
// drop functionality is supported.
function handlePluploadInit( uploader, params ) {
console.log( "Initialization complete." );
console.info( "Drag-drop supported:", !! uploader.features.dragdrop );
}
// I handle any errors raised during uploads.
function handlePluploadError() {
console.warn( "Error during upload." );
}
// I handle the files-added event. This is different
// that the queue-changed event. At this point, we
// have an opportunity to reject files from the queue.
function handlePluploadFilesAdded( uploader, files ) {
console.log( "Files selected." );
// Make sure we filter OUT any non-image files.
for ( var i = ( files.length - 1 ) ; i >=0 ; i-- ) {
if ( ! isImageFile( files[ i ] ) ) {
console.warn( "Rejecting non-image file." );
files.splice( i, 1 );
}
}
}
// I handle the queue changed event.
function handlePluploadQueueChanged( uploader ) {
console.log( "Files added to queue." );
if ( uploader.files.length && isNotUploading() ){
uploader.start();
}
}
// I handle the upload progress event. This gives us
// the progress of the given file, NOT of the entire
// upload queue.
function handlePluploadUploadProgress( uploader, file ) {
console.info( "Upload progress:", file.percent );
dom.percent.text( file.percent );
}
// I handle the file-uploaded event. At this point,
// the resource had been uploaded to Amazon S3 and
// we can tell OUR SERVER about the event.
function handlePluploadFileUploaded( uploader, file, response ) {
var resourceData = parseAmazonResponse( response.response );
var li = $( "<li><img /></li>" );
var img = li.find( "img" );
// When passing the uploaded Key back to the
// success page, we have to be sure to fullly
// encode the key so that it doesn't confuse
// any other parts of the normal URL component
// system (ex. hashes in a filename can be
// misinterpreted as the URL hash).
img.prop(
"src",
( "./success.cfm?key=" + encodeURIComponent( resourceData.key ) )
);
dom.uploads.prepend( li );
}
// I handle the change in state of the uploader.
function handlePluploadStateChanged( uploader ) {
if ( isUploading() ) {
dom.uploader.addClass( "uploading" );
} else {
dom.uploader.removeClass( "uploading" );
}
}
// I determine if the given file is an image file.
// --
// NOTE: Our policy requires us to send image files.
function isImageFile( file ) {
var contentType = getContentTypeFromFilename( file.name );
// Make sure the content type starts with the
// image super-type.
return( /^image\//i.test( contentType ) );
}
// I determine if the upload is currently inactive.
function isNotUploading() {
var currentState = uploader.state;
return( currentState === plupload.STOPPED );
}
// I determine if the uploader is currently uploading a
// file (or if it is inactive).
function isUploading() {
var currentState = uploader.state;
return( currentState === plupload.STARTED );
}
// When Amazon S3 returns a 201 reponse, it will provide
// an XML document with the values related to the newly
// uploaded Resource. This function extracts the two
// values: Bucket and Key.
function parseAmazonResponse( response ) {
var result = {};
var pattern = /<(Bucket|Key)>([^<]+)<\/\1>/gi;
var matches = null;
while ( matches = pattern.exec( response ) ) {
var nodeName = matches[ 1 ].toLowerCase();
var nodeValue = matches[ 2 ];
result[ nodeName ] = nodeValue;
}
return( result );
}
})( jQuery, plupload );
</script>
</body>
</html>
</cfoutput>
Right before the upload occurs, in the Plupload "BeforeUpload" event, I am dynamically adjusting the POST data for the current file. In this case, I am using the Plupload-generated file ID in order to place the upload in a unique "directory" within the S3 bucket. In this way, I can upload files with the same name and avoid collisions. This allows me to handle possible naming collisions in the business logic on my ColdFusion server, and not on the Amazon S3 bucket.
NOTE: Amazon S3 doesn't really have "directories;" merely, resource keys that can mimic a hierarchical directory structure.
Once Plupload has posted the file to Amason S3, I then parse the response and report back to the ColdFusion server using a newly prepended IMG element. Here is the ColdFusion page that accepts the report and generates a pre-signed query string request authentication URL (that will eventually render the upload in the IMG element back on the client-side).
success.cfm - Our ColdFusion Success Handler
<cfscript>
// Include our ColdFusion 9 -> ColdFusion 10 migration script so
// that I can work on this at home (CF10) and in the office (CF9).
include "cf10-migration.cfm";
// Include the Amazon Web Service (AWS) S3 credentials.
include "aws-credentials.cfm";
// ------------------------------------------------------ //
// ------------------------------------------------------ //
// We are expecting the key of the uploaded resource.
// --
// NOTE: This values will NOT start with a leading slash.
param name="url.key" type="string";
// Since the key may have characters that required url-encoding,
// we have to re-encode the key or our signature may not match.
urlEncodedKey = urlEncodedFormat( url.key );
// Now that we have the resource, we can construct a full URL
// and generate a pre-signed, authorized URL.
resource = ( "/" & aws.bucket & "/" & urlEncodedKey );
// The expiration is defined as the number of seconds since
// epoch - as such, we need to figure out what our local timezone
// epoch is.
localEpoch = dateConvert( "utc2local", "1970/01/01" );
// The resource will expire in +1 day.
expiration = dateDiff( "s", localEpoch, ( now() + 1 ) );
// Build up the content of the signature (excluding Content-MD5
// and the mime-type).
stringToSignParts = [
"GET",
"",
"",
expiration,
resource
];
stringToSign = arrayToList( stringToSignParts, chr( 10 ) );
// Generate the signature as a Base64-encoded string.
// NOTE: Hmac() function was added in ColdFusion 10.
signature = binaryEncode(
binaryDecode(
hmac( stringToSign, aws.secretKey, "HmacSHA1", "utf-8" ),
"hex"
),
"base64"
);
// Prepare the signature for use in a URL (to make sure none of
// the characters get transported improperly).
urlEncodedSignature = urlEncodedFormat( signature );
// ------------------------------------------------------ //
// ------------------------------------------------------ //
// Direct to the pre-signed URL.
location(
url = "https://s3.amazonaws.com#resource#?AWSAccessKeyId=#aws.accessID#&Expires=#expiration#&Signature=#urlEncodedSignature#",
addToken = false
);
</cfscript>
There's a lot of detail in the upload script - more than I can go into in this one blog post. Hopefully the comments in the actual markup will elucidate some of the decisions I made. I'll also try to follow-up with some smaller blog posts about some very specific problems that I had when trying to figure this all out. But, in the end, I was super pumped up to finally get Plupload uploading files directly to Amazon S3!
Want to use code from this post? Check out the license.
Reader Comments
Ben, this is awesome! I'm looking forward to having this implemented in our application. It should make file uploads faster and take some load off the server. Everything feels logical and secure. Great job!
@Josh,
Thanks! It's pretty exciting. Once you upload to Amazon S3, you may still need to pull it back down to the ColdFusion server (such as for image thumbnailing, validation, etc.). BUT, you don't have to subject the ColdFusion server to *slow* client connections. That's huge! Let Amazon deal with keeping a connection open for 300 seconds... once it's on Amazon S3, the ColdFusion server can pull it down (over the massive internet backbone) in a matter of (milli)seconds.
@Ben,
I hadn't thought about thumbnailing. I was trying to think about how we could keep all of the user file activity off of the server. I wonder if it'd be possible to create the thumbnail on the client using canvas/JavaScript? Even "versions" could essentially reside on S3 by copying the object on S3 and just storing its key in our database. Asset thumbnailing is a whole different story though...
Very nice work, Ben!
@Josh - it might be worth setting up the infrastructure for your ColdFusion server (or whatever server-side infrastructure you use) to talk to S3 and pull files down as needed. S3 is awesome, but pretty limited in terms of file manipulation. If you ever want to dynamically ZIP files, for example, you can't do that on S3. You have to pull them all down to the server, create the ZIP, and then re-upload to S3 (or serve the ZIP from your server). There are JS solutions for cropping and applying filters to images, but you're probably going to get better options and performance on the server -- for now at least!
@Josh,
Yeah, there's definitely certain image assets that can just be copied. Plus, S3 offers Copy/Move commands that can move resources from the "temp upload" keys to the application keys without pulling them down.
@Brian,
Thanks! And yeah, the ZIPing up of images (well logical collections of a number of things) is on our soon-to-be-done roadmap. I was just thinking about that as well. Even so, however, pulling things from S3 to the data-center is probably going to be significantly fast. Even if we had to download 200MB of stuff (which is larger than we would have to, most likely), I don't think that will add a significant wait time. At least not in some small amount of testing that I've done.
"BUT, you don't have to subject the ColdFusion server to *slow* client connections. That's huge! Let Amazon deal with keeping a connection open for 300 seconds... "
PLUpload works in chunks, so uploading even the largest file isn't going to suck up a thread for 300 seconds solidly...
And if your server connection to S3 is that slow, you'll have other issues if you ever need to do anything else on AWS.
So I'm not sure of the benefit of doing all the extra client-side heartache vs just uploading it from CF once the final chunk arrives...
@Tom,
I think the issue is not so much that the server connection is slow, it's that the client-server connection is slow. And, I think this is perhaps very specific to file-upload. I know that when I am at the office, uploads are super fast because we have some sort of business-grade internet connection. However, when I am at home using my persona internet connection, uploading the same file can take like 4-5 times as long.
During that extended upload time, I can see (from Fusion Reactor) that a ColdFusion thread is being used to read in that file. So, if a user's connection causes them to need 20-seconds to upload a file, that's 20-seconds that ColdFusion needs to be watching that POST.
Now, if we can, instead have people upload directly to Amazon S3, then S3 needs to worry about that 20-second upload time. Once the file is uploaded, however, pulling the file down from S3 to our production server (if we need to) will probably take milliseconds (due to the massive data-center connections).
@Ben,
Well, that's why we chunk.
If a 5 meg file takes 20 seconds, it's not going to take much longer in blocks of 128k or whatever, but you don't see a scary spike for average time in FusionReactor :-)
Oh, and of course you've got upload feedback for chunked uploads in PLUpload too, which users find reassuring.
@Tom,
To be honest, I don't really understand what Chunking is and what implications it has. If I chunk a file in Plupload, do I have to manually glue it back together on the server? Or is that something that is hidden away from me (ie. handled by the ColdFusion infrastructure)?
@Ben,
The target page has to handle the temporary storage and reassembly at the end, but there is example ColdFusion code.
https://gist.github.com/jsteenkamp/1116037 via http://www.plupload.com/punbb/viewtopic.php?pid=7936#p7936
@Tom,
Thanks for the link, I'll check it out. I vaguely remembering having to join files together when using the XStandard API. So, hopefully this will make sense :)
How can i upload my files from specific folder to amazon s3(with access key)? Please suggest
@Raju,
I am not sure I understand your question? When using Plupload, the user has to explicitly select files from their computer. So, you can't really do anything automated (via Plupload).
Regarding chunking, the S3 API supports multi-part uploads. http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingRESTAPImpUpload.html
Multi-part is important for at least 2 reason. It allows you to multi-thread uploads of large files more quickly and also improves resiliency. S3 ingest speed improves with more threads, assuming you have bandwidth to support it, plus if you chunk uploads, and a piece fails, you can restart just that chunk.
Does this still work?
I'm having difficult trying to implement this, I changed this line
As amazon seems to have switched the order to
But difficulty getting any error to show up and upload isn't completing.
I've not read anything about Amazon changing their URL formats. The easiest way to see errors might be to capture the network traffic...
@J, @Tom,
I've recently been having issues with the S3 direct-uploads; but, I believe I have narrowed it down to my VPN software as my current machine cannot run my demo... OR, other demos that I've come across (even ones that are hosted on other people's servers).
As an experiment, try to disabling the "html5" runtime and just doing the "flash" runtime. This worked for me. This is how I was able to narrow the issue down to html5 and the OPTIONS preflight request. It seems that on my computer all OPTIONS requests get automatically "Aborted".
Every time I try to run an OPTION request, I see an error show up in my console.app (Mac) log output. After some Googling, it looks like this error is related to my Cisco AnyConnect VPN software:
https://discussions.apple.com/thread/3507331
Anyway, not to be long-winded, but I believe the demo still works, but may have complications with the VPN if you have one running.
@All,
I just uninstalled my Cisco AnyConnect VPN client and my CORS requests started working again ! I will now try to re-installed with fewer options.
This is awesome work. Anyone know of a similar approach using PLUpload on the Microsoft stack? We're on MVC4 (for better or worse) and would like to implement the direct stream to S3.