Skip to main content
Ben Nadel at InVision In Real Life (IRL) 2018 (Hollywood, CA) with: Sara Dunnack
Ben Nadel at InVision In Real Life (IRL) 2018 (Hollywood, CA) with: Sara Dunnack

Zipping Image Archives With DEFLATE And STORE Compression Methods In Lucee CFML 5.2.9.31

By
Published in — Comments (5)

For the last few years, one of my teammates - David Bainbridge - has been suggesting that we switch our zipping / archiving algorithms over to use STORE instead of DEFLATE when creating an archive of images. The idea being that most images file-formats are already compressed; which means that attempting to compress the images further during the zip operation does nothing but use unnecessary CPU. Now my team is planning to add some new zipping functionality, I thought it would be a good time to start looking at the difference between DEFLATE and STORE in Lucee CFML 5.2.9.31.

Traditionally, ColdFusion has not offered the ability to set any type of compression method in its various zipping features - it just uses DEFLATE, and that's the end of it. That said, it looks like Zac Spitzer recently added compressionMethod to the CFZip tag in Lucee CFML 5.3.3.x. So that's very exciting!

Of course, as luck would have it, I'm still on Lucee CFML 5.2.x at work; so, in order to experiment with different storage compression methods, I'm going to use the zip package that you can install via apt-get.

To get started, I created a simple Dockerfile that uses the Ortus Solutions CommandBox Docker image and installs the zip binary using apt-get:

FROM ortussolutions/commandbox:lucee5

RUN apt-get update && \
	apt-get install -y \
		zip && \
	apt-get clean

Then, I created a docker-compose.yml file to bring this experimental Lucee CFML container online:

version: "2.4"

services:

  cfml:
    build: "." # Build our Dockerfile.
    ports:
      - "8080:8080"
    volumes:
      - "./:/app"
    environment:
      cfconfigfile: "/app/.cfconfig.json"
      APP_DIR: "/app/wwwroot"
    healthcheck:
      test: "echo hello"

Once I had this Lucee CFML container running with the zip binary installed, I went about testing the different compression methods. The test is simple: I have a directory of images (about 17Mb worth); and, I create an archive of the images, once using the default compression method, DEFALTE, and once using the compression method, STORE.

ASIDE: I'm relatively new to the zip binary. As such, getting this to work properly with the CFExecute tag required a good deal of trial-and-error. I used the man page extensively, supplemented by various StackOverflow threads. For reasons unclear to me, executing the zip binary from the server terminal resulted in a different behavior when compared to the CFExecute tag, even when all the paths in the command were absolute.

<cfscript>

	// Reset demo on subsequent executions.
	cleanupFile( "./images-zip.zip" );
	cleanupFile( "./images-zip-0.zip" );

	// ------------------------------------------------------------------------------- //
	// ------------------------------------------------------------------------------- //

	// First, let's try archiving the images using the DEFAULT COMPRESSION settings.
	timer
		type = "outline"
		label = "Default ZIP Settings"
		{

		execZip([
			"-r", // Recurse the input directory.
			"-j", // Junk file paths (only store filenames, resulting in flat directory).
			expandPath( "./images-zip.zip" ), // Output file.
			expandPath( "./images" ), // Input directory.
			"-x *.DS_Store" // Don't include files in zip.
		]);

	}

	echo( "<br />" );
	echo( "Zip file size: " );
	echo( numberFormat( getFileInfo( "./images-zip.zip" ).size ) & " bytes" );
	echo( "<br /><br />" );

	// ------------------------------------------------------------------------------- //
	// ------------------------------------------------------------------------------- //

	// Second, let's try archiving the images using NO COMPRESSION (ie, just storing the
	// files as an archive, but not attempting to reduce the file size at all). Since we
	// are working with images, the contents are already compressed; in most cases, the
	// zip algorithm won't be able to remove nay size.
	timer
		type = "outline"
		label = "Zero-Compression ZIP Settings"
		{

		execZip([
			// Regulate the speed of compression: 0 means NO compression. This is setting
			// the compression method to STORE, as opposed to DEFLATE, which is the
			// default method. This will apply to all files within the zip - if we wanted
			// to target only a subset of file-types, we could have used "-n" to white-
			// list a subset of the input files (ex, "-n .gif:.jpg:.jpeg:.png").
			"-0",
			"-r", // Recurse the input directory.
			"-j", // Junk file paths (only store filenames, resulting in flat directory).
			expandPath( "./images-zip-0.zip" ), // Output file.
			expandPath( "./images" ), // Input directory.
			"-x *.DS_Store" // Don't include files in zip.
		]);

	}

	echo( "<br />" );
	echo( "Zip file size: " );
	echo( numberFormat( getFileInfo( "./images-zip-0.zip" ).size ) & " bytes" );
	echo( "<br /><br />" );

	// ------------------------------------------------------------------------------- //
	// ------------------------------------------------------------------------------- //

	/**
	* I execute the zip command-line utility using the given arguments. Standard output
	* is printed to page. If an error is returned, the page request is aborted.
	* 
	* @zipArguments I am the command-line arguments for zip.
	*/
	public void function execZip( required array zipArguments ) {

		execute
			name = "zip"
			arguments = zipArguments.toList( " " )
			variable = "successOutput"
			errorVariable = "errorOutput"
			timeout = 10
			terminateOnTimeout = true
		;

		if ( len( errorOutput ?: "" ) ) {

			dump( errorOutput );
			abort;

		}

		echo( "<pre>" & ( successOutput ?: "" ) & "</pre>" );

	}


	/**
	* I delete the given file if it exists.
	* 
	* @filename I am the file being deleted.
	*/
	public void function cleanupFile( required string filename ) {

		if ( fileExists( filename ) ) {

			fileDelete( filename );

		}

	}

</cfscript>

In the second call to the CFExecute tag, I am passing in -0 as the first command-line argument. This moderates the compression speed, where -0 tells zip to use the compression method, STORE. This will apply to all files in the zip - if we wanted to be more targeted, we could have used the -n command-line argument to apply STORE to a set of white-listed file-extensions.

That said, if we run the above Lucee CFML code, we get the following browser output:

Speed comparison of zip binary using DEFALTE and STORE compression methods in Lucee CFML.

As you can see, when using the STORE algorithm (via the -0 compression speed setting), the archive of images is generated significantly faster with essentially the same file-size. Of course, the speed of execution is going to vary a bit on each page request; however, the STORE method was consistently about twice-as-fast as the DEFALTE method.

It looks like David Bainbridge was right - using the STORE method when creating image archives is going to be the smart choice. I'm excited to see that compressionMethod was added to the recent releases of Lucee CFML. However, since I'm still on an older version of Lucee, at least I can fallback to the zip binary; or, who knows, maybe dive down into the Java layer.

Want to use code from this post? Check out the license.

Short link: https://bennadel.com/3802

Reader Comments

15,848 Comments

@Zac,

To be honest, I have no idea what a tar file is :D This may be a silly question, but can any [common] computer open up a tar file? For example, if I was generating a zip/tar file for the user to download it, would double-clicking on it work for them? Or do you have to be more tech-savvy to have tar capabilities?

15,848 Comments

@All,

When I was looking at using zip with CFExecute, one of the hurdles is that you cannot tell CFExecute to use any particular working directory. And, unfortunately, the way to get relative folder paths in the resultant archive is to use a working directory in combination with a relative folder input. As such, I wanted to revisit this approach using Java's ProcessBuilder:

www.bennadel.com/blog/3810-executing-command-line-processes-from-a-working-directory-using-processbuilder-in-lucee-cfml-5-2-9-31.htm

I had never heard of the ProcessBuilder class until Brad Wood mentioned it on Twitter a few weeks back (during my GraphicsMagick exploration, I think). It seems pretty cool; and affords us ways to set working directories, define environment variables, and manipulate inputs and outputs.

15,848 Comments

@All,

I just realized that I wasn't properly scoping my local variables in the exeZip() method. The success + error variable names should have been prefixed with local.. So, currently it is:

execute
	name = "zip"
	arguments = zipArguments.toList( " " )
	variable = "successOutput"
	errorVariable = "errorOutput"
	timeout = 10
	terminateOnTimeout = true
;

... and it should be:

execute
	name = "zip"
	arguments = zipArguments.toList( " " )
	variable = "local.successOutput"
	errorVariable = "local.errorOutput"
	timeout = 10
	terminateOnTimeout = true
;

Without the local., the variables get stored in the variables scope, not the local scope.

As an aside, you can always set the localMode of the function to modern if you want to change the way unscoped variable assignments work:

www.bennadel.com/blog/3678-using-function-localmode-modern-to-more-safely-render-coldfusion-templates-in-lucee-5-3-2-77.htm

Oh my chickens, this post is old!
I believe in love. I believe in compassion. I believe in human rights. I believe that we can afford to give more of these gifts to the world around us because it costs us nothing to be decent and kind and understanding. And, I want you to know that when you land on this site, you are accepted for who you are, no matter how you identify, what truths you live, or whatever kind of goofy shit makes you feel alive! Rock on with your bad self!
— Ben Nadel