Revisiting Circuit Breakers As Explicit Action Proxies In ColdFusion
Earlier this week, I posted my first exploration of the Circuit Breaker pattern in ColdFusion. While I've heard about Circuit Breakers here and there over the past few years, this was the first time that I had really looked at them in a code sense. And, after I posted my experiment, I came to realize that my first approach to Circuit Breakers was flawed. As such, I wanted to revisit the Circuit Breaker pattern as a more explicit action proxy in ColdFusion.
In my first approach, I viewed the Circuit Breaker as a completely transparent proxy to a given component. When the Circuit Breaker wrapped an origin component, it used meta-programming techniques and reflection to expose public methods that mimicked the methods on the origin component. I did this because I thought it was clever for the Circuit Breaker to be seamlessly swapped into situations that expected the origin component.
Upon further reflection and some thought-provoking conversation at work, I came to undrestand that this was not the right approach. For one, you could argue that a transparent proxy violates the Principle of Least Surprise. But really, the surprise is due to a fact that I wasn't thinking properly about "Duck Typing". Part of what makes a Circuit Breaker different than its origin component is that it can throw different kinds of errors. And, since the landscape of errors are part of what gives an Object its Type, I was inappropriately equating the origin Type to the Circuit Breaker Type.
In other words, the concept of a Circuit Breaker should be an explicit part of the control flow and should be called-out specifically in the code. And, in fact, if you look at other examples of Circuit Breakers, such as Microsoft's write-up in their Architecture and Design docs, you will see that this is the case - that the Circuit Breaker is explicitly leveraged in the execution of actions.
To follow this pattern, I removed the transparent proxy functionality and replaced it with a few execution delegates:
- CircuitBreaker.applyMethod( target, methodName [, methodArguments ] )
- CircuitBreaker.callMethod( target, methodName [, arg1 [, arg2 ... [, argN ] ] ] )
- CircuitBreaker.execute( callback )
- CircuitBreaker.execute( target, methodName [, methodArguments ] )
Ultimately, each of these methods just calls .execute() under the hood. But, I was inspired by JavaScript's .call() and .apply() methods for function invocation; the .callMethod() function allows for inline arguments where as the .applyMethod() function expects an explicit collection of invocation arguments. But like I said, these are just convenience methods for the .execute() call.
As in my previous exploration, part of what makes the Circuit Breaker unique is that it throws unique errors, based on the state of the Circuit Breaker. The difference in this exploration, though, is that the calling code can actually leverage these errors since the calling code now knows that it is dealing with a Circuit Breaker. To see this in action, I have a small demo that runs the Circuit Breaker in a try/catch block and has an explicit catch for the "CircuitBreakerOpen" error:
<cfscript>
// Create an instance of our Circuit Breaker with the given configuration.
breaker = new CircuitBreaker(
failedRequestThreshold = 8,
activeRequestThreshold = 12,
openStateTimeout = ( 2 * 1000 )
);
// Create a our target component - this is the component for which we will be
// routing the method invocation through the Circuit Breaker.
testGateway = new TestGateway();
try {
writeOutput( "[ Breaker is Closed: #breaker.isClosed()# ] <br />" );
// The .callMethod() allows optional inline arguments.
result = breaker.callMethod( testGateway, "makeGoodCall", "Hello world (call)!" );
writeOutput( "#result# <br />" );
// The .applyMethod() allows an optional arguments collection.
// --
// NOTE: Since this ultimately calls the native invoke() under the hood, the
// optional arguments collection can be an array or a struct.
result = breaker.applyMethod( testGateway, "makeGoodCall", [ "Hello world (apply)!" ] );
writeOutput( "#result# <br />" );
// The .execute() method can accept a Closure or a Function.
result = breaker.execute(
function() {
return( "Hello world (closure)!" );
}
);
writeOutput( "#result# <br />" );
// --------------------------------------------------------------------------- //
// --------------------------------------------------------------------------- //
// Now, try to trip the breaker with bad calls.
// --
// NOTE: Doing this inside a thread so as not to invoke the page-level try / catch.
thread
action = "run"
name = "breaker-test"
{
// Trip the 8-failed request threshold.
for ( var i = 1 ; i<= 10 ; i++ ) {
try {
breaker.execute( testGateway, "makeBadCall", [ "Meh." ] );
} catch ( any error ) {
// .... we are expecting these calls to fail.
}
}
}
// Join the thread to ensure the "bad" requests all finish.
thread action = "join";
// --------------------------------------------------------------------------- //
// --------------------------------------------------------------------------- //
// At this point, we are expecting the Circuit Breaker to have been tripped
// and for the circuit to be opened.
writeOutput( "[ Breaker is Closed: #breaker.isClosed()# ] <br />" );
// As such, we are expecting this call to fail as well (due to circuit state).
result = breaker.callMethod( testGateway, "makeGoodCall", "After the errors." );
} catch ( CircuitBreakerOpen error ) {
writeOutput( "#error.message# <br />" );
}
</cfscript>
As you can see, in this approach, rather than wrapping the TestGateway.cfc inside the CircuitBreaker.cfc, we are keeping the two separate. But, rather than invoking methods directly on the TestGateway.cfc, I'm using the CircuitBreaker.cfc as an explicit action proxy, allowing the Circuit Breaker to execute the target methods on my behalf.
In this demo, the first few calls will complete successfully. Then, we spawn an asynchronous thread that exhausts the error threshold, thereby tripping the Circuit Breaker open. This should cause the last method invocation to fail with a "CircuitBreakerOpen" error. And, when we run the above code, we get the following page output:
[ Breaker is Closed: YES ]
Hello world (call)!
Hello world (apply)!
Hello world (closure)!
[ Breaker is Closed: NO ]
Target invocation failing fast due to open circuit breaker.
As you can see, after the Circuit Breaker is tripped, the last call fails do to a "CircuitBreakerOpen" error.
Here's the refactored version of my CircuitBreaker.cfc:
component
output = false
hint = "I marshal the invocation of actions (closures or method calls), providing circuit-breaker protection."
{
/**
* I initialize the Circuit Breaker with the given options.
*
* @failedRequestThreshold I am the number of requests that can fail before the circuit it opened.
* @activeRequestThreshold I am the number of parallel requests that can be concurrently active before the circuit is opened.
* @openStateTimeout I am the time (in milliseconds) that the circuit will remain open until the target is tested.
* @output false
*/
public any function init(
numeric failedRequestThreshold = 10,
numeric activeRequestThreshold = 10,
numeric openStateTimeout = ( 60 * 1000 )
) {
// Store the properties.
variables.failedRequestThreshold = arguments.failedRequestThreshold;
variables.activeRequestThreshold = arguments.activeRequestThreshold;
variables.openStateTimeout = arguments.openStateTimeout;
// NOTE: There is no "half-open" state. The half-open pseudo state will be entered
// by a single request in which a true state change isn't necessary.
states = {
CLOSED: "CLOSED",
OPENED: "OPENED"
};
// Default to a closed (ie, flowing) state.
state = states.CLOSED;
// Reset the counters.
activeRequestCount = 0;
failedRequestCount = 0;
// Reset the timers - each of these store millisecond values.
checkTargetHealthAtTick = 0;
lastFailedRequestAtTick = 0;
// All access to the shared state of the circuit breaker will be synchronized
// using this lock name. Errors are aggregated across all requests to execute a
// target function or method.
lockName = "CircuitBreaker-#createUUID()#";
return( this );
}
// ---
// PUBLIC METHODS.
// ---
/**
* I invoke the given method on the given target with the [optional] arguments. The
* arguments can be either an array or a struct (they are ultimately passed to the
* ColdFusion native invoke() function that will accept either an arrays or a struct
* as the arguments collection).
*
* NOTE: This is just a convenience method for execute().
*
* @target I am the component to which the message is being sent.
* @methodName I am the name of the method being sent to the target component.
* @methodArguments I am the array / struct being used as the method invocation arguments.
* @output false
*/
public any function applyMethod(
required any target,
required string methodName,
any methodArguments = []
) {
return( execute( target, methodName, methodArguments ) );
}
/**
* I invoke the given method on the given target with the optional arguments. Each
* passed-in argument after the target and methodName will be considered an invocation
* argument for the target:
*
* callMethod( target, MethodName [, arg1, arg2, arg3, ... argN ] )
*
* NOTE: This is just a convenience method for execute().
*
* @target I am the component to which the message is being sent.
* @methodName I am the name of the method being sent to the target component.
* @output false
*/
public any function callMethod(
required any target,
required string methodName
) {
// If there are any "rest" arguments, slice them off and use them during the
// invocation of the target method.
var methodArguments = ( arrayLen( arguments ) > 2 )
? arraySlice( arguments, 3 )
: []
;
return( execute( target, methodName, methodArguments ) );
}
/**
* I execute the given action inside the Circuit Breaker. The action can be a closure
* or function; or, it can be a ColdFusion component. If the target is a component,
* the methodName and methodArguments must be supplied.
*
* @target I am the function or component being executed.
* @methodName I am the message being sent to the target (if it's a component).
* @methodArguments I am the message arguments being sent to the target (if it's a component).
* @output false
*/
public any function execute(
required any target,
string methodName = "",
any methodArguments = []
) {
// CAUTION: All reading-from and writing-to the shared state of the circuit
// breaker is being SYNCHRONIZED with exclusive locking. While this does incur
// some overhead, no heavy processing should being done inside the locks. As such
// the duration of any lock should be negligible. Each request has two locks:
// one before the target execution to test state and one after the target
// execution to clean up state.
lock
name = lockName
type = "exclusive"
timeout = 1
throwOnTimeout = true
{
var currentTick = getTickCount();
// If the circuit breaker is currently closed (ie, flowing), check to see if
// we're about to go over the active request threshold.
if ( ( state == states.CLOSED ) && ( activeRequestCount == activeRequestThreshold ) ) {
// There are too many concurrent requests still pending a response from
// the target - trip the breaker and open the circuit.
state = states.OPENED;
// Keep the breaker open until some time in the future.
checkTargetHealthAtTick = ( currentTick + openStateTimeout );
}
// If the circuit breaker is currently open (ie, not flowing), then we either
// want to fail-fast or perform a single test on the target to see if we can
// close (ie, allow flow on) the circuit.
if ( state == states.OPENED ) {
// If we are currently in the opened-timeout, fail fast.
if ( currentTick < checkTargetHealthAtTick ) {
throw(
type = "CircuitBreakerOpen",
message = "Target invocation failing fast due to open circuit breaker.",
detail = "The circuit is open and therefore the requested action could not be executed."
extendedInfo = "Active request count: [#activeRequestCount#], Failed request count: [#failedRequestCount#], Testing health in [#( checkTargetHealthAtTick - currentTick )#]."
);
}
// If we made it this far, the circuit break is open; but, we want to
// allow a single test (the current request) to be run against the target
// in order to see if the target has reached a healthy state (the circuit
// can be closed again). To make sure that no parallel requests try to
// perform the same test, push out the timeout.
checkTargetHealthAtTick = ( currentTick + openStateTimeout );
}
activeRequestCount++;
} // END: Lock.
try {
// Try to execute the requested action.
var result = ( isClosure( target ) || isCustomFunction( target ) )
? target()
: invoke( target, methodName, methodArguments )
;
lock
name = lockName
type = "exclusive"
timeout = 1
throwOnTimeout = true
{
activeRequestCount--;
// If we made it this far, it means that the target method invocation has
// completed successfully. As such, we can clean up any opened state as
// long as the open state is not being held open by active requests.
if ( ( state == states.OPENED ) && ( activeRequestCount <= activeRequestThreshold ) ) {
state = states.CLOSED;
// Now that we received a healthy response from the target, let's
// reset the failure count
failedRequestCount = 0;
}
} // END: Lock.
// The target method may not return a defined value, even in a successful
// invocation. As such, we have to check to see if the result exists before
// we try to return the result upstream.
if ( structKeyExists( local, "result" ) ) {
return( result );
} else {
return; // void.
}
// Catch any errors thrown by target invocation.
} catch ( any error ) {
lock
name = lockName
type = "exclusive"
timeout = 1
throwOnTimeout = true
{
activeRequestCount--;
currentTick = getTickCount();
// If the last error occurred in the distant past (ie, a time greater
// than the open-state timeout), reset the error count before we record
// the current failure.
if ( ( currentTick - openStateTimeout ) > lastFailedRequestAtTick ) {
failedRequestCount = 0;
}
lastFailedRequestAtTick = currentTick;
// If we made it here, the invocation of the target method failed (ie,
// threw an error); as such, we need to check to see if this failure
// pushed us past the failure capacity of the circuit breaker.
if ( ++failedRequestCount > failedRequestThreshold ) {
state = states.OPENED;
// Keep the breaker open until some time in the future.
checkTargetHealthAtTick = ( currentTick + openStateTimeout );
}
} // END: Lock.
rethrow;
}
}
/**
* I determine if the Circuit Breaker is in a closed state.
*
* @output false
*/
public boolean function isClosed() {
return( state == states.CLOSED );
}
/**
* I determine if the Circuit Breaker is in an open state.
*
* NOTE: The half-open state is considered open for our purposes.
*
* @output false
*/
public boolean function isOpen() {
return( state != states.CLOSED );
}
}
While this version is fundamentally different in the way it thinks about object relationships, the core "proxy" functionality is basically the same. The difference is that the Circuit Breaker proxies method invocation on a passed-in target rather than an internal target. This doesn't really change the amount of code that needs to be written, which is nice. But, it makes the use of the Circuit Breaker explicit, which makes the code easier to follow and less astonishing. And, by allowing the target to be passed-in, we make way for the ability to execute non-object actions such as those encapsulated in a Function or a Closure reference.
Want to use code from this post? Check out the license.
Reader Comments
@All,
I finally took all of my noodling on the concept of Circuit Breakers and turned it into a GitHub project. While it's not the end of the journey, this forced me to clean it up and add unit tests:
www.bennadel.com/blog/3190-coldfusion-circuit-breaker-project-on-github.htm
Now, I'll have a more directed way to continue evolving my understanding of the concept.