Building A Simple Redis Key Scanner Using Lucee CFML 5.2.8.50 And Jedis
The other day at work, I noticed that our production Redis instance was holding steady, on balance, at about 7GB of memory usage and 14M keys. Anything "holding steady" in Redis makes me suspicious, especially over the course of several months, as I would expect the Redis resource consumption to ebb-and-flow with the natural rhythm of user engagement. As such, I wanted to start digging into the Redis data to see if anything looked wonky. To aid in this discovery process (and to have some fun), I created a super light-weight Redis Key Scanner using Lucee CFML 5.2.8.50 and Jedis, which I can run locally using CommandBox.
View this code in my Redis Key Scanner project on GitHub.
Because I don't know what I'm looking for, inspecting the Redis instance requires a brute-force approach. My suspicion is that many Redis keys don't have a specified TTL (Time To Live); which means, they will persist in Redis indefinitely. But, in order to figure out what those keys are (so that I can then figure out where in the application they are being generated), I need to traverse the Redis key space and start spot-checking value payloads.
To this end, my Redis Key Scanner uses the SCAN
command to retrieve 100 keys at a time. This collection of keys is then listed on the screen along with pagination that allows me to move forward to the next cursor offset.
For each key listed, I can click-through to a detail page that includes the following data:
- Key
- Type - String, List, Hash, Set, ZSet, etc.
- TTL - Time To Live
- Value - The payload stored in Redis.
To render the Redis Key Scanner, I'm using an old-school frameset
. Even though the frameset
is a deprecated HTML element, it seemed like exactly the right tool for job. In fact, getting this whole project up and running in about two hours was just a reminder of how powerful ColdFusion is, especially when you want to get things up and running fast.
Interacting with the Redis Key Scanner looks like this:
As you can see, it's super simple. I'm just iterating over the Redis key-space and then inspecting individual keys. The bulk of the logic is handled by a single ColdFusion component, Scanner.cfc
. This component gets configured to look at a Redis host; and then, provides two methods, .scan()
and .inspect()
. The .scan()
method provides the data for the "list" page; and, the .inspect()
method provides the data for the "detail" page:
component
output = false
hint = "I provide a simple API for iterating and viewing keys in a given Redis database."
{
/**
* I initialize the Scanner with the given JavaLoader.
*
* @javaLoaderForJedis I am the JavaLoader for the Jedis library.
*/
public any function init( required any javaLoaderForJedis ) {
variables.loader = javaLoaderForJedis;
// These properties will only be made available when the Scanner is configured to
// inspect a given Redis host.
variables.host = "";
variables.pool = "";
}
// ---
// PUBLIC METHODS.
// ---
/**
* I configure the Scanner to inspect the given Redis host.
*
* @newHost I am the Redis host to connect to.
*/
public void function configure( required string newHost ) {
// If there is an existing connection pool, close it.
if ( isConfigured() ) {
variables.pool.close();
variables.pool = "";
variables.host = "";
}
var config = loader
.create( "redis.clients.jedis.JedisPoolConfig" )
.init()
;
var pool = loader
.create( "redis.clients.jedis.JedisPool" )
.init( config, newHost )
;
variables.pool = pool;
variables.host = newHost;
}
/**
* I get the data and meta-data stored at the given key.
*
* @key I am the Redis key being inspected.
*/
public struct function inspect( required string key ) {
assertIsConfigured();
var results = {
key: key,
type: "none",
ttl: "none",
value: ""
};
results.type = withRedis(
( redis ) => {
return( redis.type( key ) );
}
);
// If the key doesn't exist, there's no point in trying to access the rest of
// key meta-data.
if ( results.type == "none" ) {
return( results );
}
results.ttl = formatTTL(
withRedis(
( redis ) => {
return( redis.ttl( key ) );
}
)
);
results.value = getValueByType( key, results.type );
return( results );
}
/**
* I determine if the Scanner has been configured for a Redis host.
*/
public boolean function isConfigured() {
return( ! isSimpleValue( pool ) );
}
/**
* I scan over the Redis keys, using the given cursor and pattern.
*
* NOTE: The pattern is applied to the keys AFTER they have been scanned. As such,
* it's possible to use a pattern that returns zero results prior to the end of a
* full iteration of the Redis database.
*
* @scanCursor I am the cursor performing the iteration.
* @scanPattern I am the post-scan filter to apply to the result-set.
* @scanCount I am the number of keys to scan in one operation.
*/
public struct function scan(
required numeric scanCursor,
required string scanPattern,
numeric scanCount = 100
) {
assertIsConfigured();
var scanParams = loader
.create( "redis.clients.jedis.ScanParams" )
.init()
.match( scanPattern )
.count( scanCount )
;
var results = withRedis(
( redis ) => {
return( redis.scan( scanCursor, scanParams ) );
}
);
return({
cursor: results.getCursor(),
keys: results.getResult()
});
}
// ---
// PRIVATE METHODS.
// ---
/**
* I assert that the Scanner is configured; and, throw an error if not.
*/
private void function assertIsConfigured() {
if ( ! isConfigured() ) {
throwNotConfiguredError();
}
}
/**
* I format the given TTL value to make it more human-readable.
*
* @ttl I am the TTL in seconds being formatted.
*/
private string function formatTTL( required numeric ttl ) {
if ( ttl < 0 ) {
return( "none" );
}
if ( ttl < 60 ) {
return( ttl & " seconds" );
}
var ttlInMinutes = ( ttl / 60 );
if ( ttlInMinutes < 60 ) {
return( numberFormat( ttlInMinutes, "0.0" ) & " minutes" );
}
var ttlInHours = ( ttlInMinutes / 60 );
if ( ttlInHours < 24 ) {
return( numberFormat( ttlInMinutes, "0.0" ) & " hours" );
}
var ttlInDays = ( ttlInHours / 24 );
if ( ttlInDays < 28 ) {
return( numberFormat( ttlInDays, "0.0" ) & " days" );
}
var ttlInWeeks = ( ttlInDays / 7 );
return( numberFormat( ttlInWeeks, "0.0" ) & " weeks" );
}
/**
* I get the Redis key value for a key of the given type.
*
* @key I am the key being read.
* @type I am the data-type stored at the given key.
*/
private any function getValueByType(
required string key,
required string type
) {
var value = withRedis(
( redis ) => {
switch ( type ) {
case "hash":
return( redis.hgetAll( key ) );
break;
case "list":
return( redis.lrange( key, 0, -1 ) );
break;
case "set":
return( redis.smembers( key ) );
break;
case "string":
return( redis.get( key ) );
break;
case "zset":
return( redis.zrange( key, 0, -1 ) );
break;
default:
return( "Redis type [#type#] not supported by Scanner." );
break;
}
}
);
return( isNull( value ) ? "" : value );
}
/**
* I throw a Not Configured error.
*/
private void function throwNotConfiguredError() {
throw(
type = "RedisScannerNotConfigured",
message = "Redis Scanner not yet configured.",
detail = "Before the Redis Scanner can be used, it must be configured using the .configure() method."
);
}
/**
* I invoke the given Callback with an instance of a Redis connection from the Jedis
* connection pool. The value returned by the Callback is passed-back up to the
* calling context. This removes the need to manage the connection in the calling
* context.
*
* @callback I am a Function that is invoked with an instance of a Redis connection.
*/
private any function withRedis( required function callback ) {
try {
var redis = pool.getResource();
return( callback( redis ) );
} finally {
redis?.close();
}
}
}
The rest of the code in this ColdFusion application consists of a few no-frills CFML pages that work using simple request-response workflows. No AJAX; barely any JavaScript; no compile step; just raw CFML horsepower! It's not even worth sharing the code here - you can view it in the GitHub repo if you want.
I am sure there are plenty of professional GUI (Graphical User Interface) applications for Redis. But, what's the fun in using one of those when I can spend an hour or two building one for myself. Plus, when I build it, I can tailor the UI (User Interface) for my own needs. If nothing else, just another opportunity to practice coding; and, a nice vacation from all the thick-client, single-page application development that I'm usually doing.
Want to use code from this post? Check out the license.
Reader Comments
Ben. This is interesting stuff!
The part I get a bit lost at:
Is this a call to the java library or a call to the CFC:
Method
I had a look at:
http://xetorthio.github.io/jedis/
But, I couldn't see:
And I presume the cursor, acts like pagination for Redis. Are you collecting 100 keys at a time?
@Charles,
Yes, that is all correct. I have a ColdFusion Component
Scanner.cfc
, which has a.scan()
method. But, turns around and calls the.scan()
method on the Java Driver for Redis (Jedis). And then, the actual scan is powered by a Cursor which takes the current Cursor value and the number of keys (100 in this case).The Cursors in Redis are interesting. They maintain no state; so, you can have as many Cursors as you want to at any one time. And, you can change the key-count at any time during the iteration to get a larger collection of keys. The benefit of the Cursor is that you can safely iterate over the database without locking up access. Trying to do something like
KEYS *
would be a huge mistake as it would (among other issues) lock access to all other requests while the keys were being gathered.SCAN
on the other hand, can safely and efficiently look at a small slice at a time without a negative impact.Wow. This cursor feature is amazing.
I'm really excited about trying some of this Redis stuff out. I am going to create a POC session storage application, to try and persuade my boss that this would be a good alternative to Coldfusion's native session scope.
@All,
I updated the Redis Key Scanner project to include some RegEx filtering for both include and exclude functionality:
www.bennadel.com/blog/3708-using-regex-to-filter-keys-with-redis-key-scanner-in-lucee-cfml-5-2-8-50-and-jedis.htm
By default, the
SCAN
operation only has aMATCH
which uses glob-style includes. In my update, I am pulling the key-set back into memory (just the keys in the given cursor iteration); then, I am using.filter()
Lucee CFML methods to perform the include / exclude on the ColdFusion side.It will pull back more keys than using
MATCH
; but, with a much-improved user experience (UX).@Charles,
Yeah, Redis is just a lot of fun to play with for some reason. At least, for me. I'm actually working on a fun little algorithm to scan over the Redis database and set a
TTL
on any key that is persistent (ie, never going to expire). That's why I created this Redis Scanner -- to get a sense of what was going on. I should have more to post about that tomorrow.Looking forward to the next installment.
Learning so much about Redis from these blogs!
@All,
As a follow-up, I created this Redis Key Scanner to help me better understand the contents of the Redis database such I could create a subsequent task that scans the key-space and adds TTL values:
www.bennadel.com/blog/3712-adding-a-ttl-to-all-persistent-keys-in-redis-using-launchdarkly-feature-flags-and-lucee-cfml-5-2-9-40.htm
This code was a lot of fun to write! And, what's really cool is that I am use LaunchDarkly feature flags to both enable/disable the clean-up task as well as ramp up the aggressiveness of the task's
SCAN
operation.