Managing Shared Secret Token Rotation Across Systems In Lucee CFML 5.3.7.47
When two systems interact, a shared secret is often included with inter-system communication in order to make sure that the given calls are both authorized and have not been tampered with. For various reasons, those shared secrets need to be rotated over time. And, since multiple systems - that need to agree on which secrets are valid - cannot be deployed at the exact same moment, we need to have a token rotation strategy that allows for different systems to rotate tokens at different times. Since I've recently had to deal with this type of token rotation in my ColdFusion applications, I thought it would be good to codify my thoughts in a small demo in Lucee CFML 5.3.7.47.
As I discussed in Episode 46 of the Working Code podcast: Secret Management vs. Premature Optimization, InVision didn't have robust platform management in the very early days; "share secrets" were often hard-coded right into the CFML code. As part of the remediation for a recent Penetration Test (aka, Pen Test), we've had to move those hard-coded secrets into ENV (environment) variables. And, rotate the values at the same time.
Rotating a shared secret is particularly tedious because it's impossible to deploy multiple systems at the exact same moment. As such, shared secrets cannot be simultaneously changed in all locations. Instead, rotating a shared secret has to take place incrementally across systems. Which means, at any given moment, one system may be using "old" tokens while another system may be using "new" tokens. And, the target system has to consider all of those tokens "valid" when authorizing incoming API requests / WebHook calls.
For security purposes, once an ENV value is defined in production, it can never be read again by an engineer. After all, it's a "secret"; and, if it could be read by anyone, it wouldn't an effective secret. What this means is that a production ENV value can only be created or entirely overwritten.
When it comes to rotating secret tokens, the term "rotating" might conjure up images of moving one ENV value into another ENV value. However, doing this would require knowing the ENV value that is being moved. And, as I mentioned above, once a production secret is in place, it can never be read again. As such, "rotating" ENV values really means "creating" new ENV values and then updating the ColdFusion code to read those new ENV values.
In my ColdFusion applications, I've been implementing this strategy with a simple Array of tokens. This Array is a "moving window" over the collection of shared secrets that are currently "active" for a given API / WebHook end-point. The "moving window" is defined as version-suffixed values:
<cfscript>
application.secrets.apiTokens = coalesceStrings([
env( "API_TOKEN" ),
env( "API_TOKEN_V2" ),
env( "API_TOKEN_V3" )
// ...
// env( "API_TOKEN_V4" )
// env( "API_TOKEN_V5" )
// env( "API_TOKEN_V{N}" )
// ...
]);
</cfscript>
The env() function, which I posted about the other day, reads the given environment variable; or, falls-back to using a default value (in this case, an empty string). The coalesceStrings() function then filters that collection down to non-empty values.
The values in this Array are the values that can be used to authorize an incoming request. As a thought experiment, let's assume that the most current ENV value is API_TOKEN_V3. If we had to rotate this token across two systems, we'd have to take the following steps:
Define a new ENV value -
API_TOKEN_V4- for the ColdFusion service.Add said ENV token -
API_TOKEN_V4- to the Array in the ColdFusion code (outlined above).Deploy the ColdFusion service, which will now use both
API_TOKEN_V3andAPI_TOKEN_V4as valid values.Update / overwrite the shared secret ENV value in the calling service environment (to use the same secret we just added to the ColdFUsion app).
Re-deploy the calling service so that it picks-up the redefined ENV value.
Remove the old ENV token -
API_TOKEN_V3- from the ColdFusion code (removing it from the Array).Re-deploy the ColdFusion service, which will now use
API_TOKEN_V4as the only valid token value.
As you can see, rotating the token is quite tedious. And, if you have multiple calling services that all use the same shared secret, this becomes even more tedious. But, at least it allows us to rotate shared secrets across systems without any down-time.
Of course, defining these secrets is just one part of the equation - next, we have to use these secrets in order to authorize incoming requests. Authorizing a request can take several different forms: it might be a simple comparison; it might be used to generate a one-way hash; or, it might be used as part of a public/private key encryption. For the sake of simplicity in this demo, we're going to consume these tokens as simple, case-sensitive comparisons.
And, lucky for us, ColdFusion provides the array.find() member method, which performs a case-sensitive look-up of a value within an array. Such as the Array that holds are secrets! Here's a simple example of a ColdFusion template that expects a secret to be provided in the URL - url.apiToken - and then validates that URL token against the array of secret tokens:
<cfscript>
param name="url.apiToken" type="string";
param name="form.message" type="string";
// SECURITY: Each WebHook request contains a secret that is shared by both THIS app
// and the EXTERNAL calling app.
testApiToken( url.apiToken );
logMessage( form.message );
// ------------------------------------------------------------------------------- //
// ------------------------------------------------------------------------------- //
/**
* I inspect and validate the given API token. If it is valid, this method quietly
* exits; if it is invalid, this method throws an error.
*/
public void function testApiToken( required string apiToken ) {
// Compare the incoming token to all of the "active" tokens. Since this is a
// SHARED secret with an external application, we have to use multiple tokens in
// order to allow for asynchronous rotation of tokens across multiple systems.
if ( application.secrets.apiTokens.find( apiToken ) ) {
return;
}
throw(
type = "InvalidApiToken",
message = "Message could not be logged due to invalid API token value.",
detail = "Invalid token: [#apiToken#]"
);
}
/**
* I log the given message provided by the webhook.
*/
public void function logMessage( required string message ) {
systemOutput( "[MESSAGE]: #message#", true );
}
</cfscript>
As you can see, in our simple example, the testApiToken() function is just making sure that the incoming API token is in the collection of tokens that the ColdFusion application is currently supporting. And, as different tokens are added / removed from ColdFusion codebase, this template remains unchanged.
For the sake of completeness, here's the Application.cfc ColdFusion framework component that I am using for this exploration:
component
output = false
hint = "I define the application settings and event handlers."
{
// Define the application settings.
this.name = "WebHookTokenDemo";
this.applicationTimeout = createTimeSpan( 0, 1, 0, 0 );
this.sessionManagement = false;
this.setClientCookies = false;
// ---
// LIFE-CYCLE METHODS.
// ---
/**
* I get called once when the ColdFusion application is being bootstrapped. This
* method is single-threaded and automatically synchronized by the server.
*/
public void function onApplicationStart() {
application.secrets = {};
// This approach for SECRET MANAGEMENT is based on the idea that once a secret is
// in production, it can NEVER AGAIN BE READ BY AN ENGINEER. As such, we have to
// operate under the constraint that ENV values can only ever be CREATED or fully
// OVERWRITTEN. To deal with SHARED secrets, we're going to create a collection
// of tokens that act as a "moving window" over the "active" token versions. As
// new tokens are needed, they are added as subsequent "_V{N}" ENV values. And,
// as old tokens are deprecated, they are simply removed from this collection.
application.secrets.apiTokens = coalesceStrings([
env( "API_TOKEN" ),
env( "API_TOKEN_V2" ),
env( "API_TOKEN_V3" )
// ...
// env( "API_TOKEN_V4" )
// env( "API_TOKEN_V5" )
// env( "API_TOKEN_V{N}" )
// ...
]);
}
// ---
// PRIVATE METHODS.
// ---
/**
* I return a filtered array that contains only populated (non-empty) values.
*/
private array function coalesceStrings( required array values ) {
var filteredValues = values.filter(
( value ) => {
return( value.len() );
}
);
return( filteredValues );
}
/**
* I return the given environment variable value; or, the fallback if the variable is
* either UNDEFINED or EMPTY.
*/
private string function env(
required string name,
string fallbackValue = ""
) {
// In Lucee CFML, we can access the environment variables directly from the
// SERVER SCOPE.
var value = ( server.system.environment[ name ] ?: "" );
// For the sake of the demo, we're treating an EMPTY value and a NON-EXISTENT
// value as the same thing, using the given value only if it is populated.
return( value.len() ? value : fallbackValue );
}
}
Modern best practices require us to store secrets outside of the code so that they can truly be "secret" even if the code itself were to become compromised. This blackbox nature of secrets makes things a bit more complicated when shared secrets need to be rotated. But, using simple constructs like Arrays and array.find() means that we can keep the relative complexity quite low in our Lucee CFML applications.
When is This Secret Management Approach Relevant?
This approach to secret management is used to deal with the race condition of having to deploy multiple services that all use the same value. As such, this approach is really only meaningful if a secret is shared across services. If a secret is wholly owned and consumed by a single service, using an array of values would be overkill - overwriting a single ENV value would be more than sufficient.
That said, it's not always obvious where a system boundary is drawn. Take, as an example, URLs that are embedded within a transactional email. If a URL, within a transactional email, includes a secret, this secret is now shared between the ColdFusion application and the Email Client. As such, naively updating said secret in the ColdFusion application would render all email-based links that are "out in the wild" as broken.
Want to use code from this post? Check out the license.
Reader Comments
Hi Ben
When you reference:
Are you talking about Web Server environment variables or Application Server environment variables. I am guessing it must be the latter, as I don't know whether the Application Server has the ability to access the Web Server. If so, where are these actually stored? I must say I never new these kind of variables existed?
@Charles,
These are the same kind of variables that would be available in something like
process.envin Node.js. Or, that you can provide when calling a command-line binary:I am not sure if that helps at all.
Hi Ben
What I mean, is do you explicitly write these values in the Lucee server.json file or something?
Or do you add them somewhere, like you would with a Windows Environment Variable, like:
https://docs.oracle.com/en/database/oracle/machine-learning/oml4r/1.5.1/oread/creating-and-modifying-environment-variables-on-windows.html
Or can you just create them like:
My apologies, in advance, if I am being dumb!
It is nearly Friday and my brain 🧠 is gradually beginning to melt 😬
Although the last option I gave is probably incorrect, because you wouldn't store secret keys like this, due to security considerations.
Personally, I usually store secret keys in the DB! 😀
@Charles,
So, we are using Docker locally and Kubernetes (K8) in production. Locally, the ENV values get defined in the
docker-compose.ymlfile. Then, in production - honestly, I have no idea how K8 really works, or how the ENV values get injected into the containers. We deploy to several-hundred environments, so the ENV values are managed by the platform tooling (that I'm not even allowed to look at for security reasons).I think the location of the ENV values really depends on how your code is getting deployed.
OK. I might do a bit of research into this, as I am intrigued.
I don't know much about Docker.
Privately, I run a VPS monolith in production, so I would be interested to find out, how this all works with a Windows Server 2012R2, running Lucee & IIS etc
I will let you know, if I find anything interesting!
@Charles,
FWIW, Lucee has some great documentation on environment variables and how to set them for both Windows and Linux environments. See https://docs.lucee.org/guides/Various/system-properties.html
@Hasse
Thanks Hasse…
I still haven't quite got my head around this issue.
I will take a look! 👍