Playing With Lists And Blocking Pop Operations In Redis And Lucee 5.2.9.40
I have a really embarrassing confession to make: until just recently, I thought that the Redis List data-type worked like the ColdFusion List data-type. Which is to say, I assumed that it was just an abstraction over simple, delimited string values. This wasn't based on anything that I read - it was just a really, really poor assumption that my brain made. As anyone who uses Redis Lists would tell you, Lists in Redis are really more like (bi-directionally linked) Arrays, where each index is a completely isolated string value. Once I realized how very wrong I was about Redis Lists, I wanted to put together a little learning experiment, where I Push and Pop JSON (JavaScript Object Notation) values on to and off of a Redis List using a Blocking Read operation (BLPOP) in Lucee 5.2.9.40.
ASIDE: I'm using Lucee 5.2.9.40, not the latest Lucee, simply because I have a running instance for work that already has a Redis server. I didn't want to have set up a new Redis instance just for this experiment.
To experiment with the Redis List data-type, I've created a demo with two ColdFusion pages: a Write Message page and Read Message page:
The Write Message page is an HTML Form that allows me to submit a Message value to the Lucee server. This value is composed in a ColdFusion Struct, which is then serialized, as JSON, and pushed onto the tail (Right) of the List.
The Read Message page uses the the blocking read operation -
BLPOP(Blocking Left Pop) - to sit and wait for list items to be available on the head (Left) of the list. As items become available, they are deserialized and rendered to the page. The Read Message page will continue blocking, reading, and dumping values for about 30-seconds before it gives up.
For this experiment, I'm using the JavaLoader and JavaLoaderFactory projects to load the Redis JAR file, along with its compile dependencies. This JavaLoader is initialized on ColdFusion Application start-up. Then, with each page request, I expose a .withRedis() method on the request scope that can be used to access an instance of the Jedis Connection object.
Here's my Application.cfc ColdFusion framework file:
component
output = false
hint = "I define the application settings and event handlers."
{
// Configure the application runtime.
this.name = "RedisListWithJsonExploration";
this.applicationTimeout = createTimeSpan( 0, 1, 0, 0 );
this.sessionManagement = false;
// Setup the mappings for our path evaluation.
this.webrootDir = getDirectoryFromPath( getCurrentTemplatePath() );
this.mappings = {
"/": this.webrootDir,
"/javaloader": "#this.webrootDir#vendor/JavaLoader/javaloader/",
"/JavaLoaderFactory": "#this.webrootDir#vendor/JavaLoaderFactory/",
"/jedis": "#this.webrootDir#vendor/jedis-2.9.3/"
};
// ---
// EVENT METHODS.
// ---
/**
* I get called once when the application is being initialized.
*/
public boolean function onApplicationStart() {
var javaLoaderFactory = new JavaLoaderFactory.JavaLoaderFactory();
application.javaLoaderForJedis = javaLoaderFactory.getJavaLoader([
expandPath( "/jedis/commons-pool2-2.4.3.jar" ),
expandPath( "/jedis/jedis-2.9.3.jar" ),
expandPath( "/jedis/slf4j-api-1.7.22.jar" )
]);
var jedisConfig = application.javaLoaderForJedis
.create( "redis.clients.jedis.JedisPoolConfig" )
.init()
;
application.jedisPool = application.javaLoaderForJedis
.create( "redis.clients.jedis.JedisPool" )
.init( jedisConfig, "127.0.0.1" )
;
return( true );
}
/**
* I get called once at the start of each request.
*/
public boolean function onRequestStart() {
request.withRedis = this.withRedis;
return( true );
}
// ---
// PUBLIC METHODS.
// ---
/**
* I invoke the given Callback with an instance of a Redis connection from the Jedis
* connection pool. The value returned by the Callback is passed-back up to the
* calling context. This removes the need to manage the connection in the calling
* context.
*
* @callback I am a Function that is invoked with an instance of a Redis connection.
*/
public any function withRedis( required function callback ) {
try {
var redis = application.jedisPool.getResource();
return( callback( redis ) );
} finally {
redis?.close();
}
}
}
As you can see, the .withRedis() method accepts a Callback function; then, it gets a Jedis / Redis connection from the connection pool and invokes the Callback, passing-in the connection object. This allows the calling context to consume the Redis connection without having to worry about the complexities of interacting directly with the connection pool.
And now that we have a way to access Redis, let's get on with the experiment. First, let's look at the Write Message page. Again, this is just a simple ColdFusion page that uses a post-back model to read a Message in from the user. This message is then wrapped in a ColdFusion Struct, serialized, and pushed onto the tail of the Redis List:
<cfscript>
// If the form has been submitted, push a Message onto the List.
// --
// NOTE: Each list item is a String that represents a serialized object.
if ( form.keyExists( "message" ) && form.message.len() ) {
listItem = {
"id": createUUID().lcase(),
"message": form.message,
"createdAt": getTickCount()
};
request.withRedis(
( redis ) => {
redis.rpush( "list:messages", [ serializeJson( listItem ) ] );
}
);
}
</cfscript>
<cfoutput>
<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8" />
<title>
Post a Message
</title>
<link rel="stylesheet" type="text/css" href="./styles.css" />
</head>
<body>
<h1>
Post a Message
</h1>
<form method="post" action="#cgi.script_name#">
<strong>Message:</strong><br />
<input type="text" name="message" autofocus size="30" />
<button type="submit">
Post
</button>
</form>
</body>
</html>
</cfoutput>
As you can see, I'm using the RPUSH operation to push the serialized JSON payload onto the Redis List at key, list:messages. If this is the first time a list item is being pushed, Redis will automatically allocate a new List data-type at the given key. Once the list item is pushed, the ColdFusion page re-renders the form, allowing the user to continue pushing new messages.
On the Read side of the experiment, I created a ColdFusion page that sits and waits for messages using the blocking read operation, BLPOP. As messages become available, I'm dumping them to the output buffer and then telling the Lucee Server to flush (cfflush) the content to the browser. This allows the messages to show up in "realtime" (for about 30-seconds).
<cfoutput>
<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8" />
<title>
Read a Message
</title>
</head>
<body>
<h1>
Read a Message
</h1>
<cfscript>
// For the purposes of the demo, we're going to block and listen for about
// 30-seconds until the page has to be refreshed (for more messages).
startedAt = getTickCount();
cutoffAt = ( startedAt + ( 30 * 1000 ) );
while ( getTickCount() < cutoffAt ) {
// Since the BLPOP operation is going to BLOCK the page request, let's
// flush data to the Browser incrementally so that we can consume and
// display the messages in "realtime" as they are pushed onto the list.
cfflush( interval = 1 );
results = request.withRedis(
( redis ) => {
// NOTE: If the operation doesn't receive any message in 5
// seconds, it will stop blocking and return NULL.
return( redis.blpop( 5, [ "list:messages" ] ) );
}
);
// The result of the BLPOP operation is 2-tuple where the first index is
// the name of the key (since we can block on multiple keys at the same
// time); and, the second index is the value of the list item.
if ( ! isNull( results ) ) {
dump( deserializeJson( results[ 2 ] ) );
echo( "<script>scrollTo( 0, 999999 );</script>" );
echo( "<br />" );
}
}
</cfscript>
<a href="#cgi.script_name#">Refresh listener</a>
</body>
</html>
</cfoutput>
The Read page is a bit more complicated than the Write page; but, the logic is still fairly simple. We're looping until the current time passed the cut-off time (about 30-seconds in the future). Then, for each loop iteration, we're blocking and waiting (up to 5-seconds) for a Redis List item to become available. If the blocking operation, BLPOP, times-out, it returns null. If it receives a list item within those 5-seconds, the result is a 2-tuple containing the Redis key and the List Item value.
So, essentially what we have is a page that pushes values onto a Redis list; and then, a page that reads items from that list and renders them in real time. Basically, we've created a super simple Message Queue.
Now, if we run these two ColdFusion pages side-by-side, we get the following output:
As you can see, as I write / push messages on to the Redis List from one ColdFusion page, they are immediately read / popped off of the Redis List by the other ColdFusion page. As the values are read, each list item JSON payload is deserialized as a ColdFusion Struct and then dumped to the page response output buffer.
There's something kind of magical about Redis. Every time I interact with it, I get a warm feeling in my tum-tum. I believe that Redis holds a tremendous amount of power; and, that I'm only just scratching the surface in terms of the ways in which I can use it in my Lucee / ColdFusion applications. Of course, the first step in better consumption is actually understanding how the Redis data-types work. My mental model for Redis Lists was laughably wrong; but, now I'm moving in the direction.
Want to use code from this post? Check out the license.
Reader Comments
Ben. This is very interesting. Is this similar to an XHR request? But I can see the browser refresh button flickering?
And, secondly is Redis like MongoDB? A key value storage mechanism using JSON?
I really like your Blogs on Redis. I keep wanting to try it out, but then the time starts to fly again...
@Charles,
In this post, you can think about my technique (for the demo) as being akin to "Long Polling". I've never actually used long-polling before; but, my understanding is that the browser would make a request (via something like AJAX) to the server and the server would hold the request open. Then, it would start flushing data to the client. On the client-side, the code would be buffering the flushed data; and, would be looking for certain delimiters (like a newline character), at which point it would take the buffered data and treat it like a cohesive value.
It's not an easy pattern to explain. It's kind of like Morris Code.
So, in my demo, the "Read a Message" pages are just making a single request. Only, the Lucee server isn't sending all the data right-away. It's using a combination of
sleep()andblpop()to "hold the request open". And, as more data becomes available, thecfflush()command flushes that data from the server down the browser on the connection that's being held-open.Basically, the "Read a Message" page is really just a very slow loading page, that takes like 30-seconds to load :D
This makes sense for a demo; but, in a "real" app, you'd probably just use an AJAX request or something like a WebSocket.
Regarding Redis, it's really cool and really fast. I don't think I use it to even a fraction of its potential - we just use it for session management / caching. But, it seems like it can do so much more, including Publish/Subscribe and interesting event-based stuff. But, as you say, the time just flies!
@Charles,
Oh, but the "Post a Message" page is doing a full page-POST. Which is why you see the refresh button refreshing. The "Post a Message" is like:
Browser => FORM Post => Lucee CFML => Redis
Then, the "Read a Message" is just reading out of Redis. So, to be clear, the demo has two different pages in it.
Cheers for the explanation.
I have just done the interactive tutorial on the Redis website, and I really like its simplicity.
If the application scope is refreshed, does the data, in the Redis cache, persist, or does it get wiped, as well?
I am just trying to figure out what the advantages are of using a Redis object over an Application scope Struct? I would like to start using it, but I need to know why I should use it?
@Charles,
Great question - Redis is entirely external to the ColdFusion application. Think of Redis just like you would MySQL or MongoDB. So, if ColdFusion restarts, it just needs to reconnect to Redis and it will get all the same data. So, if you have multiple instances of ColdFusion running, for example, they can all connect to the same Redis and you can use Redis as the session store so that your user will have the same session regardless of which ColdFusion instance they hit (assuming they are behind some load-balancer that is distributing requests across the ColdFusion instances).
The biggest difference between Redis and other DBs i that, by default, it's just in-memory data; so, if Redis crashes or restarts, the data is wiped. However, I believe you can configure Redis to write to disk just like other DBs. But, that's going a bit beyond my rather shallow understanding of Redis.
That's really helpful information.
I will tell my boss about this, because he is looking for an alternative session storage mechanism and this sounds like a good option. I presume you can store complex objects by converting to JSON, before storing in Redis and then deserialising after reading from Redis?
I guess I could hold the Redis DB on it's own server that sits behind the application server cluster. So, this will serve as a single source of truth and prevent sticky sessions?
@Charles,
Redis has some complex structures. But, it's basically strings, numbers, lists, and hashes. And, there's no "nesting" of data-types. For example, you can't have a "List of Hashes" - you can only have a List of Strings. Similarly, you can't have a "Hash of Hashes", you can only have a Hash of simple values.
That said, the Hash structure essentially maps to a flat Struct in ColdFusion. So, if you perform:
It return the
some-keyvalue as a ColdFusion Struct (assuming you're using Jedis with ColdFusion). This is how we manage sessions - just a Hash in Redis. And we use a key-convention like:sessions:{ user_id }:{ session_token }And, of course, the
user_idandsession_tokenvalues are just taken from the data passed from the client to the server (via a JWT or something of the sort).Ben. Thanks for this info.
So something like this:
So, presumably your Redis store is on its own server and then your many application server instances feed off it, thus avoiding sticky sessions.
Client --- Application 1 -- Redis
\ /
Application 2 -----
This is why pipelines are important? It enables you to queue up multiple Redis commands on the Redis store server?
Oh dear. My diagram didn't work. I think MD got confused!
@Charles,
Yes, exactly - the Redis database (which is actually managed by ElasticCache in AWS) is shared among the Lucee applications. Our load-balancers use some sort of weighted-round-robin for traffic, so we don't have "sticky sessions". Instead, any of the Lucee servers needs to be able to serve any of the sessions. So, on each request, it contacts the Redis server based on the ID/Token and loads the Hash into memory.
This, in and of itself, doesn't really relate to Pipelines. We only need a Pipeline when we want to send multiple commands in a row. So, your
HMSETis really just one command, that atomically sets the key-value pairs you provided. You would need a Pipeline if, for instance, you wanted to sent multipleHMSETcommands in a row.That said, there's also the concept of
MULTI- this is like Pipeline - but, it performs the operations atomically. So, for example, when a user starts a new session, we actually have to perform two operations:Both of these things need to happen atomically, or we would lose track of the user's set of sessions (think different browsers, or Mobile + Desktop login). So, we do something like:
.multi().hmset( ... )- creates the new session hash..sadd( ... )- add the session token to the user's session..exec()- commit the MULTI and return the results.In this case, the
hmset()andsadd()get sent to Redis and executed as an atomic action. The results of each command is then returns as ordered items in an Array.So, Pipeline and Multi are both used to send "multiple commands". But Pipeline is geared towards performance (having zero guarantees about atomoticity) and Multi is designed for all-or-nothing atomic actions.
Thanks Ben.
This information, about multi & pipeline definitions, is very useful.
It is great that Redis can carry out atomic operations.
Having said this, it is possible to set up a custom session database in Lucee:
https://docs.lucee.org/guides/Various/FAQs/technical-FAQs/database-session.html
But, I think the performance advantages of Redis, is probably the way to go.
And, I guess, if I want to store a complex Struct in Redis, I can just add it as JSON & then deserialize. So, it has all the advantages of native SESSION storage, but with a lot of added extras.
Plus, it is great that the Java language can be used to manipulate Redis. And even better than CFML can then extend this.
I am just compiling a list of pros & cons to present to my boss.
@Charles,
To be honest, I don't know very much about plugging custom Session or Cache features into Lucee. I know that it is possible; but, I have no experience there.
Re: Structs, a "flat" / 1-level struct is just a Hash in Redis. So, you don't even need to serialize it as JSON - it will just work. That said, if you need to get nested structs / objects, then there is no native support for that. As such, you either have to use JSON; or, you have to start using keys a bit more creatively (but I have no experience with the latter idea - just things I've read on various blogs).
@All,
As a related follow-up, it occurred to me that I might be able to use Redis' blocking list operations to power long-polling as means to provide a fall-back in cases where a WebSocket-based workflow might fail:
www.bennadel.com/blog/3921-using-redis-blocking-list-operations-to-power-long-polling-in-lucee-cfml-5-3-7-47.htm
I've only used long-polling once or twice; so take this with a grain of salt. It was just a fun thought-experiment.