deserializeJson() Silently Fails On High-Precision Numbers In Lucee CFML 5.3.4.77
Yesterday, Nicholas Mackey - from our Inspect team - stumbled upon an interesting behavior in which a JSON (JavaScript Object Notation) payload coming out of the Craft plug-in was including a Float value with something like 20-digits of precision. I have no idea how that value was being calculated on the Craft side; but, the interesting part of this is that when Lucee CFML 5.3.4.77 went to deserialize this JSON payload, it silently failed on said high-precision value, leaving it as a String, not a Number.
We weren't sure where in the data-processing pipeline the transformation was failing. So, I set up a small test for the first step in the pipeline: deserializing the JSON payload.
<cfscript>
```
<cfsavecontent variable="jsonPayload">
{
"a": 1.1,
"b": 1.01,
"c": 1.001,
"d": 1.0001,
"e": 1.00001,
"f": 1.000001,
"g": 1.0000001,
"h": 1.00000001,
"i": 1.000000001,
"j": 1.0000000001,
"k": 1.00000000001,
"l": 1.000000000001,
"m": 1.0000000000001,
"n": 1.00000000000001,
"o": 1.000000000000001,
"p": 1.0000000000000001,
"q": 1.00000000000000001,
"r": 1.000000000000000001,
"s": 1.0000000000000000001,
"t": 1.00000000000000000001,
"u": 1.000000000000000000001,
"v": 1.0000000000000000000001,
"w": 1.00000000000000000000001,
"x": 1.000000000000000000000001,
"y": 1.0000000000000000000000001,
"z": 1.00000000000000000000000001
}
</cfsavecontent>
```
dump( deserializeJson( jsonPayload ) );
</cfscript>
Here, you can see that I have a stringified version of a Struct-literal (via CFSaveContent). The values in the struct are floats with increasing precision. And, when we run this ColdFusion code in Lucee CFML 5.3.4.77, we get the following output:
As you can see, half the values get parsed into numbers; half the values get parsed into strings.
ASIDE: If you add more digits to the "whole number" portion of the value, Lucee will accept fewer decimal places. This has to do with the underlying Java data type that is being used, which can accurately represent only a certain amount of data. I believe this relates to a 32-bit value container; but, honestly, I don't really understand the low-level mechanics of data-types and precision.
It would be easy to call this a "bug", since numbers are getting parsed as strings. However, I do not believe this is a bug; because, what's the alternative? Unless Lucee were going to parse the value into a long, which is not the number-type is typically uses for numbers, the only other outcome would be throwing an error. And, if it threw an error, we wouldn't be able to see the data at all. At least with the current implementation, you get to see the data. And, from there, you can "transform it" programmatically to something that can fit into the Lucee CFML data-type scheme.
For example, we can take the stringified numbers and convert them into less-precise numbers. As Bryan Stanly pointed out, we can just round() the values to cast them to a lower-precision value:
<cfscript>
```
<cfsavecontent variable="jsonPayload">
{
"a": 1.1,
"b": 1.01,
"c": 1.001,
"d": 1.0001,
"e": 1.00001,
"f": 1.000001,
"g": 1.0000001,
"h": 1.00000001,
"i": 1.000000001,
"j": 1.0000000001,
"k": 1.00000000001,
"l": 1.000000000001,
"m": 1.0000000000001,
"n": 1.00000000000001,
"o": 1.000000000000001,
"p": 1.0000000000000001,
"q": 1.00000000000000001,
"r": 1.000000000000000001,
"s": 1.0000000000000000001,
"t": 1.00000000000000000001,
"u": 1.000000000000000000001,
"v": 1.0000000000000000000001,
"w": 1.00000000000000000000001,
"x": 1.000000000000000000000001,
"y": 1.0000000000000000000000001,
"z": 1.00000000000000000000000001
}
</cfsavecontent>
```
// Since we know some of the numeric values may come back with a higher-precision
// that Lucee can handle, we can cast all the values to numbers with lower-precision.
rounededData = deserializeJson( jsonPayload ).map(
( key, value ) => {
return( round( value, 5 ) );
}
);
dump( rounededData );
</cfscript>
As you can see, we're passing each value to round(), which in our case, will implicitly cast the stringified numbers to numbers. And, when we run this ColdFusion code, we get the following output:
As you can see, all of the data-types in this case are numeric. Of course, the stringified values with 20-digits of decimal precision are simply rounded to whole-numbers.
Unfortunately, since this data gets persisted in our system using MongoDB - which is schemaless - we probably have Documents that contain a mixture of value-types. So, in order to normalize the data for the client, we'll likely have to do the rounding on both the input and the output so that we can fix incoming data, but also gracefully handle existing data.
Want to use code from this post? Check out the license.
Reader Comments
Thanks for sharing this Ben, but OMG what awful behaviour!
Here's a hack for recursive correction when your data is in a struct, which is basically all the time when you use JSON:
It will fail if legitimate strings have numeric format, so things like "2.0" will be coerced into being numbers.
@Rupert,
Very nice. And, totally agree that 99.99% of the time, JSON is Structs :D At least at the top-level. In fact, every API that I've ever created that returns anything other than a Struct at the top level, it always comes back to bite me (in terms of flexibility and ease-of-change). Looking at your code, I think you can even replace:
... with:
... as the
for-inconstruct should just be able to iterate over the keys in a Struct now without having to explicitly get the key-list.Ah, yes you are right that's neater.