Considering The Separation Of Concerns When Invoking A Remote API In ColdFusion
When dealing with a local database in ColdFusion, the levels of abstraction and the separations of concern feel somewhat second nature. Yes, I've wrestled with some irrational guilt over returning Query objects from my DAL (Data Access Layer); but, on balance, I love the query object's simplicity and power; and, returning it from the DAL makes life easy. Lately, however, I've had to start consuming some remote APIs (microservices). And, when it comes to making HTTP calls, the separation of concerns is less clear in my head - it seems that so much more can go wrong when consuming a remote API. To help me think through the responsibilities of each layer, I wanted to look at consuming the Datamuse API in ColdFusion.
ASIDE: Datamuse is a word-finding API for developers. It's what I used to build the Angular HTTP client for BigSexy Poems, a poem authoring application that automatically counts syllables and facilitates the selection of rhymes and synonyms.
Before we look at my attempt at a remote API implementation, I wanted to reflect on the characteristics of a database-driven data access layer (DAL). It is not lost on me that I am communicating with the database over a remote network call; so, I am sure that a lot of my confusion is strictly emotional. As such, I want to look at the ergonomics of the database DAL and figure out how to make the HTTP DAL provide a similar experience.
A database DAL almost always returns a standard data structure: the ColdFusion query object (though it sometimes returns Booleans and numbers for aggregate-based questions).
A database DAL returns an empty query when the desired rows cannot be found - it does not throw an error.
A database DAL can easily be swapped with an alternative implementation because the only logic that the DAL encapsulates is the low-level translation of inputs into SQL statements (and the invocation of the
CFQuerytag).A database DAL throws an error if it cannot uphold its contract. For example, if there is a "communications link failure" or a "transaction deadlock", the DAL's database driver throws an error instead of attempting to return a query object.
When I started to think about the remote API data access layer, my first instinct was to make it very general. But, as I started to flesh-out that concept, I realized that I was doing little more than reinventing the CFHttp tag. This was causing the HTTP semantics to leak too much into the higher layers of the application.
For example, I was having to pass arguments like method="GET" into the data access layer. But, that's exactly the sort of thing that the DAL should be encapsulating.
That said, we are ultimately making HTTP calls. And there's a wide range of HTTP responses that have semantic meaning in the context of a remote API. As such, I think the HTTP semantics will necessarily "leak" a bit into the layer above the DAL. But, we want to limit this leakage as much as we can; and, hopefully, only expose information that the service layer can actually make use of.
After shifting information back-and-forth between the layers, the standard data structure that I eventually landed on looks like this:
statusCode- the numeric HTTP status code (or0if the network call failed to connect).statusText- the HTTP status text (less the status code).isOk- a Boolean value indicating that the HTTP status code fell in the 2xx-range of response codes.mimeType- the payload type reported by theCFHttptag (ex,application/json).NOTE: This property is primarily for logging and debugging purposes. If a response comes back as "OK", then the type of content should already be known by the application (since this is part of the remote API contract). However, if the content-type is unexpected, this property just helps us identify where the miscommunication is taking place.
data- the payload returned by theCFHttptag. If the payload is JSON (JavaScript Object Notation) and themimeTypeindicates JSON, the payload will be deserialized before it is returned. Otherwise, it will be returned as a String.NOTE: Remember, this is not a generalized DAL - this is specific to the given remote API - I'm not reinventing the
CFHttptag. This is an opinionated layer. And, "JSON" or "String" is the only data type that I'm allowing for at this time.
Yes, this does include HTTP-related information in addition to the actual data payload. But, hopefully, this is the bare-minimum that the "service layer" needs in order to know how to properly handle the response.
The role of the data access layer then becomes how to translate inputs into API calls into this standard data structure. For this exploration, I've created two data access methods:
getWordsThatRhymeWith()getWordsThatMeanLike()
These are the two types of API calls that I make in BigSexy Poems; so, I have some familiarity with how they work. And, as you will see below, translating inputs into HTTP requests has a bit of a learning curve in the Datamuse API. That said, both of these methods ultimately turn around and invoke the private method - makeRequest() - which makes the actual HTTP call and constructs the standardized data structure for the gateway response:
component
output = false
hint = "I provide low-level network calls for the Datamuse API."
{
// The Datamuse API uses some esoteric query-string parameters. I'm creating some
// reflected structs here to give the parameters slightly more intuitive names.
variables.qs = {
METADATA: "md",
MEANS_LIKE: "ml",
RHYMES_WITH: "rel_rhy",
SOUNDS_LIKE: "sl",
SPELLED_LIKE: "sp",
QUERY_ECHO: "qe"
};
// MD: Metadata to include in response.
variables.md = {
PARTS_OF_SPEECH: "p",
SYLLABLE_COUNT: "s"
};
/**
* I initialize the Datamuse API gateway with the given settings.
*
* Documentation: https://www.datamuse.com/api/
*/
public void function init(
required string datamuseDomain,
numeric defaultTimetout = 5
) {
variables.datamuseDomain = arguments.datamuseDomain
// Strip off any trailing slash - we expect the "resource" value to contain
// a leading slash and we don't want to double-up on the slashes.
.reReplace( "/+$", "" )
;
variables.defaultTimetout = arguments.defaultTimetout;
}
// ---
// PUBLIC METHODS.
// ---
/**
* I get words that rhyme with the given word.
*/
public struct function getWordsThatRhymeWith(
required string word,
required numeric limit,
boolean includeWordInResults = false,
boolean includePartsOfSpeech = false,
boolean includeSyllableCount = false
) {
var urlParams = [
"#qs.RHYMES_WITH#": word,
max: limit,
"#qs.METADATA#": ""
];
if ( includeWordInResults ) {
// Include/echo the input word as the first result.
urlParams[ qs.QUERY_ECHO ] = qs.RHYMES_WITH;
}
if ( includePartsOfSpeech ) {
urlParams[ qs.METADATA ] &= md.PARTS_OF_SPEECH;
}
if ( includeSyllableCount ) {
urlParams[ qs.METADATA ] &= md.SYLLABLE_COUNT;
}
return(
makeRequest(
requestResource = "/words",
urlParams = urlParams
)
);
}
/**
* I get the words that have a related meaning to the given word. These are NOT
* synonyms (which are words that mean the SAME thing).
*/
public struct function getWordsThatMeanLike(
required string word,
required numeric limit,
boolean includeWordInResults = false,
boolean includePartsOfSpeech = false,
boolean includeSyllableCount = false
) {
var urlParams = [
"#qs.MEANS_LIKE#": word,
max: limit,
"#qs.METADATA#": ""
];
if ( includeWordInResults ) {
// Include/echo the input word as the first result.
urlParams[ qs.QUERY_ECHO ] = qs.MEANS_LIKE;
}
if ( includePartsOfSpeech ) {
urlParams[ qs.METADATA ] &= md.PARTS_OF_SPEECH;
}
if ( includeSyllableCount ) {
urlParams[ qs.METADATA ] &= md.SYLLABLE_COUNT;
}
return(
makeRequest(
requestResource = "/words",
urlParams = urlParams
)
);
}
// ---
// PRIVATE METHODS.
// ---
/**
* I return the given fileContent value as a string.
*
* NOTE: Even though we always ask ColdFusion to return a Binary value in the HTTP
* response object, the Type is only guaranteed if the request comes back properly.
* If something goes terribly wrong (such as a "Connection Failure"), the fileContent
* will still be returned as a simple string.
*/
private string function getFileContentAsString( required any fileContent ) {
if ( isBinary( fileContent ) ) {
return( charsetEncode( fileContent, "utf-8" ) );
} else {
return( fileContent );
}
}
/**
* I determine if the given response content represents a JSON payload.
*/
private boolean function isJsonResponse(
required string mimeType,
required string data
) {
// NOTE: We're purposefully NOT USING the isJson() function here because 1) we
// don't want to protect against invalid JSON - we want to throw an error and 2)
// it would mean (essentially) parsing the payload twice in the "happy path".
return( mimeType.reFindNoCase( "\b(json)$" ) && data.len() );
}
/**
* I make a request to the target API and return a normalized response. If the response
* type is JSON, the data field will be deserialized before being returned:
*
* - statusCode
* - statusText
* - isOk
* - mimeType
* - data
*/
private struct function makeRequest(
required string requestResource,
struct urlParams = [:],
numeric timeout = defaultTimetout
) {
http
result = "local.httpResponse"
method = "GET"
url = "#datamuseDomain##requestResource#"
getAsBinary = "yes"
charset = "utf-8"
timeout = timeout
{
for ( var name in urlParams ) {
httpParam
type = "url"
name = name
value = urlParams[ name ]
;
}
}
// Prepare normalized return properties.
var statusCode = val( httpResponse.statusCode );
var statusText = ( statusCode )
? httpResponse.statusCode.listRest( " " )
: httpResponse.statusCode
;
var isOk = ( ( statusCode >= 200 ) && ( statusCode < 300 ) );
var mimeType = httpResponse.mimeType.lcase();
var data = getFileContentAsString( httpResponse.fileContent );
// If there was a connection failure, we know that the response file content
// cannot be consumed successfully.
if ( ! statusCode ) {
throw(
type = "DatamuseGateway.ConnectionFailure",
message = "Connection failure, no response available.",
extendedInfo = serializeJson([
status: "#statusCode# #statusText#",
endpoint: "GET: #datamuseDomain##requestResource#",
urlParams: urlParams,
fileContent: data
])
);
}
// Only attempt to parse JSON if the response indicates JSON.
if ( isJsonResponse( mimeType, data ) ) {
try {
data = deserializeJson( data );
} catch ( any error ) {
throw(
type = "DatamuseGateway.InvalidJson",
message = "Response body could not be parsed as JSON.",
extendedInfo = serializeJson([
status: "#statusCode# #statusText#",
endpoint: "GET: #datamuseDomain##requestResource#",
urlParams: urlParams,
fileContent: data
])
);
}
}
return({
statusCode: statusCode,
statusText: statusText,
isOk: isOk,
mimeType: mimeType,
data: data
});
}
}
For the most part, the internal makeRequest() method is a translation layer, taking inputs and returning outputs. But, it does use a heavy-hand in some cases: when the network call failed or when the JSON response is invalid. In these two cases, it throws an error because it knows that the response cannot possibly be correct, nor can it be inspected in a meaningful way.
I didn't want to build too much logic into the "gateway" layer because the layer above it (the "service" layer) will almost certainly have a lot of business logic in it as well. And, I didn't want to start spreading related logic across two layers. This is why the gateway needs to return the HTTP status code along with the data - the service layer will understand how to process non-success status codes.
In the next layer up - the "service" layer - I expose the same two methods:
getWordsThatMeanLike()getWordsThatRhymeWith()
... though, these ones have a much simpler signature since they are even more opinionated than the "gateway" layer methods. And, like the gateway layer, the service layer is also a translation layer; but, the translation responsibilities are slightly different. The service layer must normalize data and wrap API errors in application / domain errors.
When looking at the service layer below, note that I do not have a try/catch around the gateway method invocation. Any error that the gateway throws is an error that the service layer cannot recover from (in the current implementation). As such, there's no value-add in catching-and-wrapping the error as it bubbles-up. The service layer can only add value to responses that it understands.
That said, if the underlying API returns data that the service layer doesn't understand, it will throw an error. But, it will be of a type that the next layer up (the "workflow" layer) might handle in some way (ex, logging).
component
accessors = true
output = false
hint = "I provide high-level calls for the Datamuse API."
{
// Define properties for dependency-injection.
property gateway;
// ---
// PUBLIC METHODS.
// ---
/**
* I get the words that have a related meaning to the given word. These are NOT
* synonyms (which are words that mean the SAME thing).
*/
public array function getWordsThatMeanLike(
required string word,
numeric limit = 100
) {
var results = gateway.getWordsThatMeanLike(
word = word,
limit = limit,
includeWordInResults = true,
includePartsOfSpeech = true,
includeSyllableCount = true
);
if ( results.isOk ) {
return( normalizeWordData( results.data ) );
}
// Handle failure codes.
// --
// NOTE: At this time, the Datamuse API has no documented non-success error codes.
// As such if the response was not OK, there's nothing we can do to recover. That
// said, I'm including a 429 (rate limiting) for the sake of exploration.
switch ( results.statusCode ) {
case 429:
throwTooManyRequestsError( arguments, results, "Rate limit reached when getting words that mean like." );
break;
default:
throwUnexpectedResponseError( arguments, results, "Could not get words that mean like." );
break;
}
}
/**
* I get words that rhyme with the given word.
*/
public array function getWordsThatRhymeWith(
required string word,
numeric limit = 100
) {
var results = gateway.getWordsThatRhymeWith(
word = word,
limit = limit,
includeWordInResults = true,
includePartsOfSpeech = true,
includeSyllableCount = true
);
if ( results.isOk ) {
return( normalizeWordData( results.data ) );
}
// Handle failure codes.
// --
// NOTE: At this time, the Datamuse API has no documented non-success error codes.
// As such if the response was not OK, there's nothing we can do to recover. That
// said, I'm including a 429 (rate limiting) for the sake of exploration.
switch ( results.statusCode ) {
case 429:
throwTooManyRequestsError( arguments, results, "Rate limit reached when getting words that rhyme with." );
break;
default:
throwUnexpectedResponseError( arguments, results, "Could not get words that rhyme with." );
break;
}
}
// ---
// PRIVATE METHODS.
// ---
/**
* I normalize the word-related data responses to include short-hand access to
* properties such as the "part of speech" as separate Booleans.
*/
private array function normalizeWordData( required array data ) {
var normalizedData = data.map(
( item ) => {
// Not all results contain "tags" even when requested.
var tags = ( item.tags ?: [] );
var isQuery = !! tags.contains( "query" );
var isNoun = !! tags.contains( "n" );
var isVerb = !! tags.contains( "v" );
var isAdjective = !! tags.contains( "adj" );
var isAdverb = !! tags.contains( "adv" );
var isProper = !! tags.contains( "prop" );
return([
word: item.word,
score: item.score,
syllableCount: item.numSyllables,
isQuery: isQuery,
isNoun: isNoun,
isVerb: isVerb,
isAdjective: isAdjective,
isAdverb: isAdverb,
isProper: isProper
]);
}
);
return( normalizedData );
}
/**
* I throw a rate limiting error for the given inputs and outputs.
*/
private void function throwTooManyRequestsError(
required any context,
required struct results,
required string detail
) {
throw(
type = "Datamuse.TooManyRequests",
message = "Datamuse API rejected request due to rate limiting.",
detail = detail,
extendedInfo = serializeJson({
context: context,
results: results
})
);
}
/**
* I throw a normalized error for the given inputs and outputs.
*/
private void function throwUnexpectedResponseError(
required any context,
required struct results,
required string detail
) {
throw(
type = "Datamuse.Error",
message = "Unexpected error response from Datamuse API.",
detail = detail,
extendedInfo = serializeJson({
context: context,
results: results
})
);
}
}
As you can see, there is some HTTP-related information in the service layer. But, not so much that the gateway calls are super generic; and, not so much that the gateway layer couldn't easily be swapped with another implementation. Ultimately, the service layer just depends on the standard data structure returned by the remote API gateway; and, it doesn't much care how the gateway was implemented.
Consuming the Datamuse API then just requires a little dependency-injection and we're good to go:
<cfscript>
datamuse = new DatamuseService()
.setGateway( new DatamuseGateway( "https://api.datamuse.com/" ) )
;
rhymes = datamuse.getWordsThatRhymeWith( "quickly", 3 );
dump( rhymes );
similar = datamuse.getWordsThatMeanLike( "sassy", 3 );
dump( similar );
</cfscript>
And, when we run this ColdFusion code, we get the following output:
To be clear, this is all very much a work in progress for me! And, the details will change from remote API to remote API since the layers involved here are opinionated, not generic. For example, in my current makeRequest() method, there is no option to pass-in the HTTP Method - GET is hard-coded because that's the only method that the Datamuse API allows for. The goal of the "gateway" layer isn't to solve for all remote APIs - it's to solve for a very specific remote API.
And, there are problems here that I know I haven't solved. Take "rate limiting" for example. Often times, a remote API will return HTTP headers that contain rate limiting information. Does the gateway layer need to return that information to the service layer somehow? Or, would the gateway layer throw an error when being rate-limited?
In my exploration, I'm simply looking for a 429 status code in order to "handle" rate limiting. But, I'm not convinced that this is the correct approach.
Figuring out which aspects of the code have which responsibilities is a big part of what makes programming challenging. Failure to draw those boundaries well is what creates a "big ball of mud" that becomes increasingly hard to maintain over time. When it comes to database interactions, I feel like I have developed good instincts. But, when it comes to remote APIs, I still very much feel like a novice. Taking time to reflect on the separation of concerns surrounding a remote API call is a good step; but, I know it's only an early step in my evolution as a ColdFusion developer.
I'd be very curious to hear how other developers handle remote calls - and what balance they strike between their layers of abstraction.
Want to use code from this post? Check out the license.
Reader Comments
I've had pretty good luck with this function, for ALL transmission requests outside the system. (At least for JSON-based operations.)
Though thinking about the semantics of 200/404/500 status code types, I don't have a facility in my default result struct to support that, so that the outside calling service can utilize it. An improvement for certain. Unless you consider that a remote API call and a DB query are both DAL features. In theory, the handler (for example), shouldn't know the data is API- or DB-based. Just "here's data" or "not" and "here's an error I came across". Just some stuff to consider.
@Will,
It looks like we're taking fairly similar approaches. I'm assuming that your
getResultStruct()function is doing something like mine, where the "data" is just one of the keys that you return. That said, the point you make is where my brain gets stuck:That's definitely where my mind was when I started thinking about this problem. My first attempt really tried to completely hide away the fact that a remote API was being called.
I wonder if my outlook is being colored (it likely is) by the fact that ColdFusion doesn't really have a robust "Error Object". It's not like in JavaScript (or Java or other languages) where you
throw()an actual error object. Instead, ColdFusion errors are all just String-based, which makes it harder to effectively communicate error information up the stack.In lieu of that, I am communicating more meta-information in the resultant Struct that I return from the data access layer. But, like I said in the post, this is just the first step in this journey. I am sure my thoughts will continue to evolve as I get more hands-on experience.
Ben, yes, "data" is one of the keys.
I do format a little differently for the (heavy) use of DataTables, since it has a specific return format, but that's all done in the handlers.
You mention that CFML doesn't have a "robust" error object. a) I would say "it sorta does", or, optionally b) make your own model object. I'm a Coldbox fan, it does have an Exception model of some sort. (Perhaps you could use it w/o a full Coldbox install.)
@Will,
To be clear, I do use errors a lot in ColdFusion :D But, all the targeting is done on
type; and, it's awesome that ColdFusion allows fortype-basedcatchblocks 💪 so, perhaps I am being too critical.Post A Comment — ❤️ I'd Love To Hear From You! ❤️
Post a Comment →