Creating A Generic Proxy For Retry Semantics In ColdFusion
This morning, I took a look at implementing retry behavior for transaction lock wait timeout errors in ColdFusion and MySQL. When I was done with that experiment, I realized that the approach could be made more generic if the error check was factored out into its own behavior. Meaning, I could add retry semantics to any type of ColdFusion component if I factored out the logic that determined which types of errors would warrant a retry attempt. As such, I wanted to see what such an approach might look like.
In the recent Circuit Breaker implementation that I built for Node.js, the Circuit Breaker State had to be able to diffrent "failures," such as network and disk failures, from "domain errors," such at a "Not Found" API response. To do this, the Circuit Breaker State accepts, as part of its configuration, an isError() function that accepts the given Error instance as an argument and returns a Boolean value, indicating that the Error represents a failure.
This approach is light-weight and affective; so, I figured I'd try the same thing with ColdFusion. In my generic RetryProxy.cfc component, the constructor requires a target (to proxy) and a Function or Closure to determine which errors are "transient" (ie, worthy of a retry). Here's what it might look like to consume:
<cfscript>
// Setup some general error-checking functions (closures work as well). Each of
// these function accepts the Error instance in question and must return a boolean
// indicating that the error is transient (true) or non-retriable (false).
function isMySqlLockTimeoutError( required any error ) {
// Read more: https://dev.mysql.com/doc/refman/5.7/en/error-messages-server.html#error_er_lock_deadlock
return(
( error.type == "Database" ) &&
( error.errorCode == "40001" )
);
}
function isSqlServerLockTimeoutError( required any error ) {
// Read more: https://technet.microsoft.com/en-us/library/cc645860(v=sql.105).aspx
return(
( error.type == "Database" ) &&
( error.errorCode == "1222" )
);
}
function isAlwaysTransientError( required any error ) {
return( true );
}
function isNeverTransientError( required any error ) {
return( false );
}
// ------------------------------------------------------------------------------- //
// ------------------------------------------------------------------------------- //
// Create our retry proxy using the given transient error test.
proxy = new RetryProxy( new TestTarget(), isAlwaysTransientError );
try {
writeDump( proxy.works() );
writeDump( proxy.breaks() );
} catch ( any error ) {
// This should be the "breaks()" method error.
writeDump( error );
}
</cfscript>
As you can see, each error function just accepts an error and returns a Boolean. And, while I'm not using most of them in this demo, I've included several in order to illustrate the point. In this example, I'm using the "always" transient behavior, which will retry the .breaks() method. So, when we run this code, we get the following output:
As you can see, the .breaks() method was attempted 3 times before the RetryProxy.cfc finally gave up and threw an error.
Here's what the RetryProxy.cfc looks like. It's more or less the same as the RetriableGateway.cfc in my earlier blog post; with a few more options for including or excluding certain methods from the retry logic. I am not sure that this kind of differentiation is necessary (given the differentiation on the error object); but, I wanted to experiment with it anyway.
component
output = false
hint = "I provide automatic retry functionality around the target component."
{
/**
* I initialize the retry proxy with the given target component. Retries will
* only be applied to "transient" errors. And, since the proxy doesn't know which
* errors are transient / retriable, it must check with the isTransientError()
* function.
*
* @target I am the component being proxied.
* @isTransientError I determine if the thrown error is safe to retry (returns a Boolean).
* @retryCount I am the number of retries that will be attempted before throwing an error.
* @includeMethods I am the collection of method names for which to explicitly apply retry semantics.
* @excludeMethods I am the collection of method names for which to explicitly omit retry semantics.
* @output false
*/
public any function init(
required any target,
required function isTransientError,
numeric retryCount = 2,
array includeMethods = [],
array excludeMethods = []
) {
variables.target = arguments.target;
variables.isTransientError = arguments.isTransientError;
variables.retryCount = arguments.retryCount;
generateProxyMethods( includeMethods, excludeMethods );
return( this );
}
// ---
// PUBLIC METHODS.
// ---
// ... proxy methods will be duplicated and injected here ...
// ---
// PRIVATE METHODS.
// ---
/**
* I inspect the target component and create local, public proxy methods that match
* the invocable methods on the target component. All target methods will be proxied;
* however, the proxy will be a RETRY proxy or a BLIND proxy based on the include /
* exclude method name collections.
*
* @includeMethods I am the collection of method names for which to explicitly apply retry semantics.
* @excludeMethods I am the collection of method names for which to explicitly omit retry semantics.
* @output false
*/
private void function generateProxyMethods(
required array includeMethods,
required array excludeMethods
) {
// Look for public methods / closures on the target component and create a
// local proxy method for each invocable property. By explicitly stamping out
// clones of the proxy method, we don't have to rely on the onMissingMethod()
// functionality, which I personally feel makes this a cleaner approach.
for ( var publicKey in structKeyArray( target ) ) {
var publicProperty = target[ publicKey ];
if ( isInvocable( publicProperty ) ) {
// Determine if the given method is being implicitly or explicitly
// excluded from the proxy's retry semantics.
var isIncluded = ( ! arrayLen( includeMethods ) || arrayFindNoCase( includeMethods, publicKey ) );
var isExcluded = arrayFindNoCase( excludeMethods, publicKey );
this[ publicKey ] = ( isIncluded && ! isExcluded )
? proxyRetryTemplate
: proxyBlindTemplate
;
}
}
}
/**
* I return the back-off duration, in milliseconds, that should be waited after
* the given attempt has failed to execute successfully.
*
* @attempt I am the attempt number (starting at zero) that just failed.
* @output false
*/
private numeric function getBackoffDuration( required numeric attempt ) {
return( 1000 * ( attempt + rand() ) );
}
/**
* I determine if the given value is invocable.
*
* @value I am the public property that was plucked from the target component.
* @output false
*/
private boolean function isInvocable( required any value ) {
return( isCustomFunction( value ) || isClosure( value ) );
}
/**
* I provide the template for "blind pass-through" proxy methods. These implement
* no retry logic.
*
* @output false
*/
private any function proxyBlindTemplate( /* ...arguments */ ) {
// Gather the proxy invocation parameters. Since the proxyBlindTemplate() has
// been cloned for each public method on the target, we can get the name of the
// target method by introspecting the name of "this" method.
var methodName = getFunctionCalledName();
var methodArguments = arguments;
return( invoke( target, methodName, methodArguments ) );
}
/**
* I provide the template for "retry" proxy methods.
*
* @output false
*/
private any function proxyRetryTemplate( /* ...arguments */ ) {
// For the purposes of the error message, we'll record the duration of the
// attempted proxy execution.
var startedAt = getTickCount();
// Gather the proxy invocation parameters. Since the proxyRetryTemplate() has
// been cloned for each public method on the target, we can get the name of the
// target method by introspecting the name of "this" method.
var methodName = getFunctionCalledName();
var methodArguments = arguments;
for ( var attempt = 0 ; attempt <= retryCount ; attempt++ ) {
try {
return( invoke( target, methodName, methodArguments ) );
} catch ( any error ) {
// If this is not a retriable error, then rethrow it and let it bubble
// up to the calling context.
if ( ! isTransientError( error ) ) {
rethrow;
}
// If this was our last retry attempt on the target method, throw an
// error and let it bubble up to the calling context.
if ( attempt >= retryCount ) {
throw(
type = "RetryError",
message = "Proxy method failed even after retry.",
detail = "The proxy method [#methodName#] could not be successfully executed after [#( retryCount + 1 )#] attempts taking [#numberFormat( getTickCount() - startedAt )#] ms.",
extendedInfo = serializeJson( duplicate( error ) )
);
}
// Since we're encountering a transient error, let's sleep the thread
// briefly and give the underlying system time to recover.
sleep( getBackoffDuration( attempt ) );
}
}
// CAUTION: Control flow will never get this far since the for-loop will either
// return early or throw an error on the last iteration.
}
}
With a Database gateway, it's easy to see that a retry attempt would be helpful. Heck, MySQL even tells you to do this in it's error messages: "try restarting transaction". But, other types of failures - such as 3rd-party APIs consumed over an HTTP request - could also benefit from automatic retry behavior. As such, I think there's value in this kind of experimentation.
Want to use code from this post? Check out the license.
Reader Comments
Very interesting POC Ben, will absolutely play with this code next week!
@Harry,
Very cool - I hope you enjoy it :D