Generating Rich Server-Side Reports In Lucee CFML 5.3.6.61
When it comes to rendering "Views" and / or "API Responses" in a ColdFusion application, I feel like I have a good-enough handle on where things are supposed to live within the application architecture (somewhat dictated by whatever framework I'm using). But, when it comes to generating rich, possibly interactive, reports, I feel a bit like I've wandered off the map. They're not quite "views", so they don't live in the "View" layer of the application. And, they're not really part of the client-side code, so they don't live in the Single-Page Application (SPA) layer. Reporting assets have this ambiguous, mixture of characteristics. As such, I wanted to noodle on where they might live and how they might all come together in Lucee CFML 5.3.6.61.
To explore this problem space, I'm going to generate a ZIP file that contains an HTML file (aka, the rich, interactive report) with embedded CSS and JavaScript and a directory of images that are referenced by this report. To do this, I need:
- The data for the report.
- Something to render the HTML template.
- The CSS files.
- The JavaScript files.
- Some sort of "scratch" directory in which to put intermediary work files.
- Something to ZIP it all up and serve it to the user.
I am sure there are a million-and-one ways to solve this problem. So, please take my approach as little more than an experiment. But, here's what I've come up with so far; and - and least for today - I feel like it works pretty well.
The more I program, the more that I've come to believe that "cohesive functionality" should be stored in a cohesive place. This makes the functionality easier to understand and maintain over time. As such, I want to take all the aspects of this report-generation and collocate them on the server. What I'm going to end up with a directory structure that looks like this:
./report/assets/- The directory of "client-side" assets (CSS and JavaScript) that will be required by the report../report/templates/- The directory of ColdFusion templates (CFML files) that will be used to generate the rich, interactive report on the server-side../report/ReportGenerator.cfc- The ColdFusion Component that contains all of the implementation details regarding how to generate and package this report.
The ReportGenerator.cfc is really the star of this show. But, before we look at the details, let's look at how I'm consuming it. For the sake of this demo, the ReportGenerator.cfc ColdFusion component is being handed all of its data. Meaning, it doesn't look any data up - it has no notion of data persistence. But, I don't feel strongly about this point. I would have zero objection to a Report Generator knowing how to look-up the data needed for the report generation. It just so happened that, in this case, I'm passing the data in because it was the easiest and decoupled thing for the demo.
The ReportGenerator.cfc has a single public method - .generateReport() - that takes the data and returns a binary payload that for the ZIP archive file. Here's the consuming request that invokes this file:
<cfscript>
// Setup the DEMO DATA for our report generation.
project = {
id: 1,
name: "My Important Report"
};
photos = [
{
id: 329,
clientFilename: "vicky_ryder.jpg",
remoteUrl: "https://bennadel.com/images/header/photos/vicky_ryder.jpg"
},
{
id: 362,
clientFilename: "vicky_ryder_2.jpg",
remoteUrl: "https://bennadel.com/images/header/photos/vicky_ryder_2.jpg"
}
];
// When setting up the report generator, we need to give it a "scratch" directory in
// which it can store all of the intermediary files as it prepares the ZIP archive. I
// could have used something like getTempDirectory(); but, I like have an explicit
// folder so that I can check the scratch directory as part of the debugging process.
// --
// NOTE: This could be an application-cached singleton (it is stateless). But, for
// the sake of the demo, I'm just re-creating it on every request.
generator = new report.ReportGenerator( expandPath( "./temp" ) );
// Generate a report for the given Project and Photos.
// --
// NOTE: There are other ways to do this, like returning a file-path or a remote URL
// at which the ZIP has been stored; but, for the sake of the demo, I'm just going to
// have the report generator return the BINARY payload for the ZIP.
zipContent = generator.generateReport( project, photos );
// Stream ZIP content back to user.
header
name = "content-disposition"
value = "attachment; filename=""my-important-project.zip"";"
;
content
type = "application/zip"
variable = zipContent
;
</cfscript>
Here, I'm going to be downloading a few photos that I took with my good friend, Vicky Ryder - the queen of Codebass radio - and presenting them in an HTML file that also has some embedded CSS and JavaScript. But, all of that complexity is hidden from this consuming page - this script just calls:
zipContent = generator.generateReport( project, photos )
... passes-in the data and takes the returned ZIP binary and serves it up to the user.
CAUTION: Generating and passing around large binary variables could be problematic for your server, depending on how large the files are. Another possibility would be to stream the file up to a transient storage system, like an Amazon S3 bucket with a define TTL (Time-To-Live); and then, forwarding the user to a pre-signed URL location.
Without looking at the entirety of the ReportGenerator.cfc component, let's first just look at the main method - .generateReport() - as this layouts out the algorithm nicely:
<cfscript>
/**
* I generate a report for the given data and returns a ZIP payload for the report.
*
* @project I am the project data.
* @photos I am the photos to be downloaded and included in the report.
*/
public binary function generateReport(
required struct project,
required array photos
) {
var zipFile = withTempDirectory(
( tempDirectory ) => {
// NOTE: The "Report Name" folder will become the root folder that is
// exposed when the user expands the ZIP archive file on their local file
// system. As such, the name of this folder should be human-readable and
// meaningful to the user.
var reportName = "report-#dateFormat( now(), 'yyyy-mm-dd' )#";
var workingDirectory = ( tempDirectory & "/" & reportName );
var photosDirectory = ( workingDirectory & "/photos" );
directoryCreate( workingDirectory );
directoryCreate( photosDirectory );
// STEP 1: Download the photo binaries.
downloadPhotos( photos, photosDirectory );
// STEP 2: Render the HTML report template.
renderReport( project, photos, workingDirectory );
// STEP 3: Generate and return the ZIP archive (binary).
return( generateZip( tempDirectory, workingDirectory ) );
}
);
return( zipFile );
}
</cfscript>
The algorithm for generating the report creates intermediary files. As such, it performs its work inside a "temp directory" using the component method withTempDirectory(). We'll see this later; but for now, understand that it generates a temp directory, exposes it the callback above, and then cleans-up the intermediary files automatically. This leaves the .generateReport() method to the task of actually doing the meaningful work:
Step 1 - Download the photos that will be placed in the report.
Step 2 - Render the report, processing the server-side CFML files and generating client-side HTML files.
Step 3 - Compress the "working directory" of intermediary files as a ZIP archive and return it.
To download the photos, I'm going to use one of Lucee CFML's most absurdly awesome features: parallel iteration. In fact, downloading files is one the most money cases for parallel iteration since the performance bottleneck is the data-transfer itself. This far outweighs the cost of spawning a new Java thread, making the parallel iteration a huge win. Here's the function that implements Step 1:
<cfscript>
/**
* I download the given photos into the given directory. Each photo will be stored
* with the clientFilename associated with the photo record. The results of the
* download operations are returns (each with a "success" flag).
*
* @photos I am the collection of photos to download.
* @photosDirectory I am the directory into which the photos will be downloaded.
*/
private array function downloadPhotos(
required array photos,
required string photosDirectory
) {
// For performance reasons, we're going to download the photos in parallel (using
// Lucee's parallel iteration features). However, so as not to overwhelm the
// server, we're going to use a limited number of threads).
var results = photos.map(
( photo ) => {
try {
// NOTE: We're using Lucee's VIRTUAL FILE SYSTEM features to read the
// source file from a REMOTE URL and then copy that file to the local
// file-system.
fileCopy(
photo.remoteUrl,
( photosDirectory & "/" & photo.clientFilename )
);
return({
photo: photo,
success: true
});
} catch ( any error ) {
return({
photo: photo,
success: false,
error: error
});
}
},
// Perform map using parallel iteration.
true,
// Maximum parallel threads (20 is default).
6
);
return( results );
}
</cfscript>
As you can see, we're using the Array .map() function on the collection of photos - using up to 6 parallel threads - to download the photos individually. Of course, going over the network is problematic; so, we're returning a download result rather than just blowing up if something were to go wrong.
ASIDE: In this demo, I'm not doing anything with that result data. But, it could be that in a real-world situations, the report generation should continue even if one of the downloads fails (for example if the remote file is unavailable).
So far, Step 1 is pretty straightforward. So, let's look at Step 2, which is responsible for rendering server-side CFML files and generating client-side HTML files. This is where the non-conventional stuff comes into play:
<cfscript>
/**
* I render the main report template (consuming a CFML file to create an HTML file)
* for the given projects and photos.
*
* @project I am the project data.
* @photos I am the photos data.
* @workingDirectory I am the working directory into which the HTML file is saved.
*/
private void function renderReport(
required struct project,
required array photos,
required string workingDirectory
) {
// This will tell the Lucee Compiler to preserve the key-casing for the following
// config object, which we are going to embed in the HTML report page. While
// ColdFusion is not case-sensitive, JavaScript very much is. As such, it's
// important that our config object have predictable and consistent key-casing.
processingDirective preserveCase = true {
// Since this object is going to be inlined in the HTML, we have to be
// careful to create a sanitized version of the data. We don't want to leak
// sensitive information.
var config = {
project: {
id: project.id,
name: project.name
},
photos: photos.map(
( photo ) => {
return({
id: photo.id,
clientFilename: photo.clientFilename
});
}
)
};
}
// In order to prevent variables within the CFML template from "leaking" into the
// report-generator page scope, we're going to use an IIFE (Immediately Invoked
// Function Expression) with an explicit LOCALMODE so that we "trap" any unscoped
// variables (such as those created in a CFLoop tag).
var reportHtml = (
function() localmode = "modern" {
savecontent variable = "local.reportContent" {
include "./templates/index.cfm";
}
return( reportContent );
}
)();
fileWrite( ( workingDirectory & "/index.htm" ), reportHtml );
}
</cfscript>
This sub-algorithm for Step 2 does two main things: It prepares the JSON payload to be embedded within the report; and, it renders a CFML template into a variable which it then writes to an HTML file - the "report file". This method is doing a couple of cool things:
It's using the
CFProcessingDirectiveto make sure that the JSON payload, embedded within the report, is going to have an expected key-case making it inter-operable with the JavaScript embedded within the report.It's using an Immediately-Invoked Function Expression (IIFE) in order to safely render the CFML template to a String buffer using
localmode="modern".
This is actually one of my favorite uses of the localmode concept in Lucee CFML, which locally-scopes unscoped variable assignments. Which means, any unscoped variables declared within the ./templates/index.cfm CFML file will be scoped to the IIFE closure, and not leak out into the variables scope of the ColdFUsion component.
And, piling onto that dollop of that awesome-sauce, by CFInclude-ing the CFML template into the ColdFusion component, we're creating a ColdFusion Mixin that has access to the rest of the component methods. Which means, our report template can consume other private methods within ReportGenerator.cfc just as if it were a part of the ColdFusion component.
With that said, let's look at the ./index.cfm CFML file which renders the report:
<cfoutput>
<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8" />
<title>
#encodeForHtml( config.project.name )#
</title>
<!--- Slurp the CSS files into the report content. --->
#inlineStyles()#
</head>
<body>
<h1>
#encodeForHtml( config.project.name )#
</h1>
<cfloop index="photo" array="#config.photos#">
<h2>
#encodeForHtml( photo.clientFilename )#
</h2>
<p class="photo">
<a
href="./photos/#encodeForHtmlAttribute( photo.clientFilename )#"
target="_blank">
<img
src="./photos/#encodeForHtmlAttribute( photo.clientFilename )#"
alt="#encodeForHtmlAttribute( photo.clientFilename )#"
/>
</a>
</p>
</cfloop>
<!---
Inline the CONFIG object into the page (window.config) so that it can be
consumed by the embedded JavaScript files. We don't actually need that for
this demo; but, it showcases the ability to embed data.
--->
<script type="text/javascript">
window.config = JSON.parse( "#encodeForJavaScript( serializeJson( config ) )#" );
</script>
<!--- Slurp the JavaScript files into the report content. --->
#inlineScripts()#
</body>
</html>
</cfoutput>
As you can see, this looks, feels, and acts just like a normal ColdFusion "view" template: I'm using the <CFOutput> tag to interpolate strings; and, I'm using the encodeForJavaScript() function embed the JSON payload within the report. The main difference here is that this "view" is being executed as part of a server-side report generation that is being captured into a String-buffer - not served to the browser.
Notice that the CFML here makes references to non-built-in function, inlineStyles() and inlineScripts(). As I alluded to before, these are just private methods on the ReportGenerator.cfc component. They read-in the content from the ./report/assets/ folder and returns them as Strings that get interpolated into the report:
<cfscript>
/**
* I return the Script content for the report (reading in the files from the scripts
* directory within the "client side" assets).
*/
private string function inlineScripts() {
var inlinedContent = directoryList( assetsDirectory & "/scripts" )
.map(
( filePath ) => {
return(
"<!-- #encodeForHtml( getFileFromPath( filePath ) )#. -->" &
"<script type=""text/javascript"">" &
fileRead( filePath ) &
"</script>"
);
}
)
.toList( chr( 10 ) )
;
return( inlinedContent );
}
/**
* I return the CSS content for the report (reading in the files from the styles
* directory within the "client side" assets).
*/
private string function inlineStyles() {
var inlinedContent = directoryList( assetsDirectory & "/styles" )
.map(
( filePath ) => {
return(
"<!-- #encodeForHtml( getFileFromPath( filePath ) )#. -->" &
"<style type=""text/css"">" &
fileRead( filePath ) &
"</style>"
);
}
)
.toList( chr( 10 ) )
;
return( inlinedContent );
}
</cfscript>
Now, once we have the downloaded photos and the CFML rendered as HTML with in-lined CSS and JavaScript, the only thing we have left to do is ZIP it all together and return it. And, that's what Step 3 does:
<cfscript>
/**
* I generate and return the ZIP binary of the working directory.
*
* @tempDirectory I am the scratch directory for the report generator.
* @workingDirectory I am the directory being archived.
*/
private binary function generateZip(
tempDirectory,
workingDirectory
) {
var zipFile = ( tempDirectory & "/report.zip" );
// CAUTION: In production, I wouldn't necessarily use "compress" for this since
// it will waste time and CPU resources trying to compress the image binaries
// (which are already a compressed file-format). But, for the sake of the demo,
// this is the easiest way to produce the archive file.
compress(
format = "zip",
source = workingDirectory,
target = zipFile,
includeBaseFolder = true
);
return( fileReadBinary( zipFile ) );
}
</cfscript>
Here, we're just using Lucee CFML's easy-peasy compress() function to zip-up the whole working directory and return it.
ASIDE: In a real-world scenario, zipping "images" is not a worthwhile use of CPU time. As such, you can use the
zipCLI tool to "store" images without compressing them. Also, Zac Spitzer has added this to theCFZiptag in an upcoming release of Lucee CFML
And that's all there is to it! Now, if we run the report generation, we get the following output:
Woot! Look at those wonderful people! As you can see from the Mac Finder screen shot, the ZIP archive expanded to include an index.htm file (which is what's being rendered in the Chrome browser) and a director of photos that are being linked-to from the report.
I'm not going to bother showing the CSS and JavaScript files; but, you can tell from the formatting within the report that the CSS is clearly included (you can watch the video).
Now remember, there are a million ways to solve this problem. This just happens to be the way that I'm trying to solve it today. And, hopefully by stepping through the process, things don't seem too complicated.
To tie it all together, here's the ReportGenerator.cfc in its entirety:
component
output = false
hint = "I generate an HTML report."
{
/**
* I initialize the report generator to use the given scratch directory for its
* temporary files.
*
* @scratchDirectory I am the temporary scratch folder.
*/
public void function init( required string scratchDirectory ) {
variables.scratchDirectory = arguments.scratchDirectory;
// As part of the report generation, "client-side" files, like CSS and JavaScript
// files, will be inlined within the report payload. The assets directory is
// where these client-side files live.
variables.assetsDirectory = ( getDirectoryFromPath( getCurrentTemplatePath() ) & "assets" );
}
// ---
// PUBLIC METHODS.
// ---
/**
* I generate a report for the given data and returns a ZIP payload for the report.
*
* @project I am the project data.
* @photos I am the photos to be downloaded and included in the report.
*/
public binary function generateReport(
required struct project,
required array photos
) {
var zipFile = withTempDirectory(
( tempDirectory ) => {
// NOTE: The "Report Name" folder will become the root folder that is
// exposed when the user expands the ZIP archive file on their local file
// system. As such, the name of this folder should be human-readable and
// meaningful to the user.
var reportName = "report-#dateFormat( now(), 'yyyy-mm-dd' )#";
var workingDirectory = ( tempDirectory & "/" & reportName );
var photosDirectory = ( workingDirectory & "/photos" );
directoryCreate( workingDirectory );
directoryCreate( photosDirectory );
// STEP 1: Download the photo binaries.
downloadPhotos( photos, photosDirectory );
// STEP 2: Render the HTML report template.
renderReport( project, photos, workingDirectory );
// STEP 3: Generate and return the ZIP archive (binary).
return( generateZip( tempDirectory, workingDirectory ) );
}
);
return( zipFile );
}
// ---
// PRIVATE METHODS.
// ---
/**
* I download the given photos into the given directory. Each photo will be stored
* with the clientFilename associated with the photo record. The results of the
* download operations are returns (each with a "success" flag).
*
* @photos I am the collection of photos to download.
* @photosDirectory I am the directory into which the photos will be downloaded.
*/
private array function downloadPhotos(
required array photos,
required string photosDirectory
) {
// For performance reasons, we're going to download the photos in parallel (using
// Lucee's parallel iteration features). However, so as not to overwhelm the
// server, we're going to use a limited number of threads).
var results = photos.map(
( photo ) => {
try {
// NOTE: We're using Lucee's VIRTUAL FILE SYSTEM features to read the
// source file from a REMOTE URL and then copy that file to the local
// file-system.
fileCopy(
photo.remoteUrl,
( photosDirectory & "/" & photo.clientFilename )
);
return({
photo: photo,
success: true
});
} catch ( any error ) {
return({
photo: photo,
success: false,
error: error
});
}
},
// Perform map using parallel iteration.
true,
// Maximum parallel threads (20 is default).
6
);
return( results );
}
/**
* I generate and return the ZIP binary of the working directory.
*
* @tempDirectory I am the scratch directory for the report generator.
* @workingDirectory I am the directory being archived.
*/
private binary function generateZip(
tempDirectory,
workingDirectory
) {
var zipFile = ( tempDirectory & "/report.zip" );
// CAUTION: In production, I wouldn't necessarily use "compress" for this since
// it will waste time and CPU resources trying to compress the image binaries
// (which are already a compressed file-format). But, for the sake of the demo,
// this is the easiest way to produce the archive file.
compress(
format = "zip",
source = workingDirectory,
target = zipFile,
includeBaseFolder = true
);
return( fileReadBinary( zipFile ) );
}
/**
* I return the Script content for the report (reading in the files from the scripts
* directory within the "client side" assets).
*/
private string function inlineScripts() {
var inlinedContent = directoryList( assetsDirectory & "/scripts" )
.map(
( filePath ) => {
return(
"<!-- #encodeForHtml( getFileFromPath( filePath ) )#. -->" &
"<script type=""text/javascript"">" &
fileRead( filePath ) &
"</script>"
);
}
)
.toList( chr( 10 ) )
;
return( inlinedContent );
}
/**
* I return the CSS content for the report (reading in the files from the styles
* directory within the "client side" assets).
*/
private string function inlineStyles() {
var inlinedContent = directoryList( assetsDirectory & "/styles" )
.map(
( filePath ) => {
return(
"<!-- #encodeForHtml( getFileFromPath( filePath ) )#. -->" &
"<style type=""text/css"">" &
fileRead( filePath ) &
"</style>"
);
}
)
.toList( chr( 10 ) )
;
return( inlinedContent );
}
/**
* I render the main report template (consuming a CFML file to create an HTML file)
* for the given projects and photos.
*
* @project I am the project data.
* @photos I am the photos data.
* @workingDirectory I am the working directory into which the HTML file is saved.
*/
private void function renderReport(
required struct project,
required array photos,
required string workingDirectory
) {
// This will tell the Lucee Compiler to preserve the key-casing for the following
// config object, which we are going to embed in the HTML report page. While
// ColdFusion is not case-sensitive, JavaScript very much is. As such, it's
// important that our config object have predictable and consistent key-casing.
processingDirective preserveCase = true {
// Since this object is going to be inlined in the HTML, we have to be
// careful to create a sanitized version of the data. We don't want to leak
// sensitive information.
var config = {
project: {
id: project.id,
name: project.name
},
photos: photos.map(
( photo ) => {
return({
id: photo.id,
clientFilename: photo.clientFilename
});
}
)
};
}
// In order to prevent variables within the CFML template from "leaking" into the
// report-generator page scope, we're going to use an IIFE (Immediately Invoked
// Function Expression) with an explicit LOCALMODE so that we "trap" any unscoped
// variables (such as those created in a CFLoop tag).
var reportHtml = (
function() localmode = "modern" {
savecontent variable = "local.reportContent" {
include "./templates/index.cfm";
}
return( reportContent );
}
)();
fileWrite( ( workingDirectory & "/index.htm" ), reportHtml );
}
/**
* I create and manage a temp directory for the current report. The given callback
* will be invoked with the temp directory as its argument. Then, once the callback
* completed, the temp directory will be deleted.
*
* @callback I am the Function to invoke with the temp directory.
*/
private any function withTempDirectory( required function callback ) {
var name = ( scratchDirectory & "/report-#createUniqueId()#" );
var CREATE_PATH_IF_NOT_EXISTS = true;
var RECURSE_DIRECTORIES = true;
directoryCreate( name, CREATE_PATH_IF_NOT_EXISTS );
try {
return( callback( name ) );
} finally {
directoryDelete( name, RECURSE_DIRECTORIES );
}
}
}
Hopefully you've seen a technique or two here in my Lucee CFML code that has been interesting - Closures, IIFEs, localmode, String Buffers, Processing Directives, Parallel Iteration, CFML Mixins, Callback-based workflows, ZIP compression .... Lucee CFML is truly a magical language. It makes this stuff not only possible, but enjoyable!
Epilogue On Variable Mutation
When I was done writing this code and then reviewing it, I noticed something kind of fascinating: I didn't mutate a single variable! That just struck me as being super interesting. I mean, technically, the <cfloop> index variable is re-assigned; but, it's happening declaratively, but ColdFusion, not imperatively by me. Huh. Cool beans!
Want to use code from this post? Check out the license.
Reader Comments