JsonSerializer.cfc - A Data Serialization Utility For ColdFusion
ColdFusion, as you probably know, is a case-insensitive language. Sure, there are methods for executing case-sensitive string comparisons or searches; but, when it comes to variable names and other identifiers, ColdFusion is case-insensitive. This "feature" can cause a huge headache during data serialization, especially if you're passing the serialized data into a case-sensitive context like JavaScript. To help ease this transition, I've created JsonSerializer.cfc. This ColdFusion component performs a deep-copy of your data structure, converting values and making sure to create struct-keys with the explicitly defined casing.
NOTE: While ColdFusion is case-insensitive, please, for the love of all that is awesome, use consistent casing! Don't be sloppy.
Project: See this project on my GitHub account.
While the JsonSerializer.cfc ColdFusion component takes care of the serialization process, it still relies on you - the developer - to tell it how to treat the keys. Specifically, it requires that you define each key that you want serialized; and, what kind of data to expect in that key-value. While this sounds like a lot of work, it's really quite easy, especially if your application is consistent in its naming.
There are only a few public methods for outlining the serialization rules:
- asAny( key ) - Simply defines the key-casing, without any data conversion.
- asBoolean( key ) - Attempts to force the value to be a true boolean.
- asDate( key ) - Converts the date to an ISO 8601 time string.
- asFloat( key ) - Attempts to force the value to be a true float.
- asInteger( key ) - Attempts to force the value to be a true integer.
- asString( key ) - Forces the value to be a string (including numeric values).
- exclude( key ) - Will exclude the key from the serialization process.
These methods provide two functions: one, they allow you to provide the explicit casing for the keys; and two, they tell the serializer how to convert data values for consistent data types between contexts.
For example, if you tell the serializer to treat "age" as a string:
.asString( "age" )
... and your data structure defines age as the numeric value, 38, the serialization process will force the value to be serialized as the string, "38," not the number.
To see this in action, take a look at the following code. Here, we are serializing a ColdFusion struct and then deserializing it in a case-sensitive, JavaScript context:
<cfscript>
// Set up our serializer, setting up the key-casing and the value
// conversion rules.
serializer = new lib.JsonSerializer()
.asInteger( "age" )
.asAny( "createdAt" )
.asDate( "dateOfBirth" )
.asString( "favoriteColor" )
.asString( "firstName" )
.asString( "lastName" )
.asString( "nickName" )
.exclude( "password" )
;
// Imagine that these keys are all upper-case because they came
// out of a database (or some other source in which the keys may
// have been entered without proper casing).
tricia = {
FIRSTNAME = "Tricia",
LASTNAME = "Smith",
DATEOFBIRTH = dateConvert( "local2utc", "1975/01/01" ),
NICKNAME = "Trish",
FAVORITECOLOR = "333333",
AGE = 38,
CREATEDAT = now(),
PASSWORD = "I<3ColdFusion&Cookies"
};
</cfscript>
<cfcontent type="text/html; charset=utf-8" />
<cfoutput>
<!doctype html>
<html>
<head>
<meta charset="utf-8" />
<title>
JsonSerializer.cfc Example
</title>
</head>
<body>
<h1>
JsonSerializer.cfc Example
</h1>
<h3>
Tricia:
</h3>
<p>
<!--- Add spaces after commas to enable word-wrap. --->
#htmlEditFormat(
replace(
serializer.serialize( tricia ),
",",
", ",
"all"
)
)#
</p>
<h3>
View the JavaScript Console:
</h3>
<script type="text/javascript">
var tricia = JSON.parse(
"#JsStringFormat( serializer.serialize( tricia ) )#"
);
// At this point, the JavaScript object contains the
// date-of-birth as an ISO 8601 time string. Now, we can
// overwrite the key, converting the value from a string
// into an actual JavaScript date object.
tricia.dateOfBirth = new Date( tricia.dateOfBirth );
console.dir( tricia );
</script>
</body>
</html>
</cfoutput>
When we run the above code, we get the following ColdFusion output (for our serialized structure):
{"firstName":"Tricia", "age":38, "dateOfBirth":"1975-01-01T05:00:00.0Z", "createdAt":"{ts '2013-08-07 09:16:59'}", "favoriteColor":"333333", "lastName":"Smith", "nickName":"Trish"}
Within this output, there are few things to notice:
- All the keys have the appropriate casing.
- The key, "password" was excluded from the serialization process.
- The date-of-birth was converted to the ISO 8601 time string (which can be parsed as a JavaScript date).
- The favoriteColor value was kept as a string (not converted to a number).
At this point, the serialized value can be deserialized in a JavaScript context and used without any problems.
This approach isn't perfect; but, it does allow you a lot of freedom in how you define your ColdFusion data structures; instead of using quoted keys throughout your entire application (in order to maintain case), you can define your data structures naturally and then worry about casing only at the last point of serialization.
Want to use code from this post? Check out the license.
Reader Comments
Nice one Ben! ColdFusion's inability to deal with this stuff properly has been a huge headache for me for the last few months. Will be taking a look at this stuff this evening...
--
Adam
@Adam,
Yeah, it's a huge pain! Case "in"sensitivity is one of those things that has never made any sense to me. I don't think people actually use it - or if they did, I don't think anyone would care about losing that "feature."
There are some personal philosophies baked into this approach. For one, Query objects are converted to arrays-of-structs, NOT a struct-of-arrays. This indicates my personal preference for query-serialization.
I should also note that it will only convert data types IF it can. So, let's say you define "id" as an integer (maybe as a pkey on table):
.asInteger( "id" )
... and then the serializer comes across a UUID-based ID:
id = "xxxx-xxxx-x...."
... it won't "break." It will simply see that you asked for an integer, which isn't possible; so, it will just convert the UUID-based ID as a string.
Basically, as a fallback, all simple-values become strings.
@Ben,
> Basically, as a fallback, all simple-values become strings.
That is def the best approach.
Good work.
--
Adam
Excellent! This was on my list of things to do in my free time, that I never seem to get around to once I push the xbox power button...
@Adam,
Groovy! Well, if you have any feedback, I'm all ears!
@Dana,
Ha ha, I know what you mean :) My weakness is Law & Order reruns. Speaking of which, I think Law & Order UK is on tonight. I have never seen that - better set a reminder :D
What's the best way to set up the serializer dynamically/automatically using the results from a query object passed to GetMetaData()?
(NOTE: ColdFusion returns "TypeName" in upper or mixed case depending upon how the query is created. Performing any QofQ converts the TypeName value to all uppercase.)
@James,
I don't think there will be any good way to set this up dynamically, since this is about overcoming the key-casing that ColdFusion uses by default. But, the way I've been using this is that I have one "master" serializer with all the keys that I end up passing to the client. Then, I don't have to set anything up dynamically.
Oh! I see! You've got your JavaScript inside of your ColdFusion page. I would have to embed that as a special function because normally I keep all my JavaScript in a separate .js file.
I didn't realize that you could put <cfscript> at the top of a component. I was using "component {}" instead of <cfscript>component</cfscript>. ColdFusion has gone from being a very simple language to two languages - 1 of which has little to no documentation.
You have a typo in JsonSerializer.cfc: complext
I realize that cfcontent charset=utf-8" is probably in your standard template, but what is its purpose since you also have meta charset="utf-8"?
@Phillip,
I put the CFScript inside of the CFC *purely* for color-coding on GitHub :) Without it, GitHub treats its like a plain text file. I know it's not good form; but, in production code, I don't do that.
As for the CFContent/charset, I try to make it a habit to put that in since I'm not always returning an HTML page. For example, if I were returning JSON, there is no opportunity for a meta tag. So, I just try to do it everywhere so that I don't forget.
@Ben,
Regarding query objects and auto-case rules. If you use getMetaData() on a query object (i.e. getMetaData(queryVar)), then you will get an array of column information.
The data should contain:
* Name - case sensitive--either exactly how you typed it or if "*" then the case it is in the database.
* TypeName - The data type of the column.
Now, I haven't tested this with anything other than MSSQL, but it's worth investigating. You could potential have a method you can use to map from a query object:
serializer = new lib.JsonSerializer()
.fromQuery( queryVar )
;
@Ben - great stuff. It's a very different approach to serialisation, but I can see its benefits! Neater than my current method of deep-looping through a data structure and forcing all the variable names to lower case before serialising...
One question: arrays. If I have an array:
MYLUCKYNUMBERS = [3,7,13]
If I specified:
.asInteger("myLuckyNumbers")
...would that output each of those values as an integer? And if it was an array of arrays, would that work the same?
@Dan,
Ah, very good point. I forgot about the query meta data. It's been a long time since I played around with it. I'll take another look, thanks!
@Seb,
Currently, the numbers would probably be converted to strings. But, I love that as a suggestion - the data type can definitely be mapped from the array to the array items. I'll try to get that done!
@Seb,
I've updated the project so that the "closest key" now gets passed-down through the serialization process. This allows the serializer to apply parent "type" to arrays of simple and/or nested array values.
Thanks for the suggestion!
This is really good. Any chance we will be seeing this on github?
@James,
Blam!!! https://github.com/bennadel/JsonSerializer.cfc
Ben, your Git copy of the code still uses <cfscript> around the component. You should remove that.
I wrote a blog post talking about your code here: http://www.raymondcamden.com/index.cfm/2013/9/4/Implementing-custom-JSON-serialization-for-your-CFCs
@Raymond,
Yeah, I use the CFScript just for the Gist color coding :( I know it's super lame...
@Ben,
Ok, but to be clear, you get my point of why you need to remove it from Git, right?
@Ray,
Oooh, I see what you're saying. Yeah, I could definitely remove it from there. I think I tend to just put them in by-default when I'm doing anything with GitHub - force of habit.
I want to point out something that may trip up folks - it did me. I wanted to use this CFC to help a user who was using cfgrid. He was having some issues with the data in the grid and I thought this would fix it up pretty quickly. However - the CFC transforms a query into an array of structs. That is cool - and sensible - and heck - thats how I like my CF query type data to be on the client-side anyway. But this broke the cfgrid as it expected the query in the form that CF normally makes it: An object with two keys, columns and data, both arrays (*).
So to use Ben's code I first recreated the query into a struct with 2 arrays that matched how CF would have encoded the query and then passed it to the CFC.
I'm not calling this a bug, just something to watch out for.
* As weird as CF's query serialization may be, it actually does create a smaller JSON packet, which is cool. I wish it supported both styles though.
@Ray,
When I was building the Serializer, I had considered trying to pass-in a constructor flag for both Query serialization and Date serialization.
The flag for Date _would_ have been to convert it to UTC milliseconds; and, the flag for Query _would_ have used the native serialization approach. In the interest of time, however, I just went with hard-coded choices.
I'll add an Issue to the repo to add the query-flag option; if it works better with other native ColdFusion features, than it's probably worth adding.
I was trying to use the same technique quickly to fix a simple array of objects being returned and then passed through window.JSON.parse(), and it didn't like the chr(2) prepended to strings. chr(12) it seems to be happy with though.
@Tom,
Hmm, that's interesting. It sounds like maybe the char(2) was being passed back to the JSON.parse() call. I say that only because I know that JSON can't be parsed if many of the "control characters" are present. Funky chicken!
Ben,
Thanks for this awesome stuff... as it is very helpful for using ColdFusion with other case-sensitive language like JavaScript.
I am using ColdFusion with ExtJS in an Application.To implement the case-sensitive things there, this worked perfectly.
I found it working awesome, when I was using it in loading a Grid with pagination of 50 records at a time, with some time around 0.5sec and even less.
But now I am using this one to load around 2400 records at a time (for a tree structure loading as there we can't use pagination). It's taking around 3.5-4 secs to do the operation,which causing the entire tree structure to load around 6-7 secs overall.
So Plz help me out.
Thanks.
Getting of topic, but you can populate a tree on demand by only loading the root nodes to start with.
@Tom,
That's cool....But there are case in which one root node containing more than 2000 leaf nodes.
There occurs the main problem with tree loading with this much of data.
@Ben,
Any Suggestion from your side??
I am still waiting for a solution.
Thanks
Hi,
much rewrite to make this work in CF8? Must I change all the functions etc.?
This seems great and addresses the issue we are having with serializing a large query object on a preview page that we want to pass to the action page in a form post (rather than using a session var or putting into a temp db table) and then deserialize on the action side once user confirms the action.
My issue is that when i work locally on my dev box using this function i receive '//' in front of each key pair. For example,
[{//"AAPROBLEM"://"",//"avg_price":"3.52",//"CHILDCOMPANYID":440415,//"curr_traited_store_item_comb_":"1",//"item_desc_1"://"AMY'S TOFU SCRMB",//"PRODUCTID":18707,//"store_nbr":"1",//"TLNAME"://"Walmart Supercenter - Rogers (1)",//"upc":"0004227200054"},{//"AAPROBLEM"://"",//"avg_price":"3.531429",//"CHILDCOMPANYID":440419,//"curr_traited_store_item_comb_":"1",//"item_desc_1"://"AMY'S TOFU SCRMB",//"PRODUCTID":18707,//"store_nbr":"4",//"TLNAME"://"Walmart Supercenter - Siloam Springs (4)",//"upc":"0004227200054"},{//"AAPROBLEM"://"",//"avg_price":"2.37",//"CHILDCOMPANYID":440925,//"
When we use the native CF SerializeJSON function we receive the // in front of the first key value only but this is not normal and i have no idea why.... any clues?
Thanks again for a great function!
Matt
@Matthew,
check your cold fusion administrator settings where you have JSON prefix setting. I think that is adding your // in front of your JSON strings.
Something I dOnT geT aBout THis?
Declaring your structure like so would preserve any case sensitivity you want, so why is that an issue.
tricia = {
"fIRStNAMe" = "Tricia"
};
Wonderful!! Your JsonSerializer.cfc saved the day.
Had issues with serializeJSON of a dynamically built structure including key values with an array of more structures.
Notably I'm using CF 11 which had advancements to serializeJSON; however still had issues with missing double quotes around string values (unless an all numerical string representation had a leading zero or space).