Turning Off Session Management for Web Spiders

Michael Dinowitz came to my office to do some code review since our server was having some trouble, slowing down and what not. We ended up having to restart the server several times to kill the slowly growning memory usage of the ColdFusion service. One suggestion that Dinowitz made was to turn off session management for web spiders so that they do not get any server memroy usage. Since spiders do not use cookie, their sessions never hold. And so, for every single page the spider hits, a new session is created. And with that new session goes every bit of user-related memeory that you have created in the application.

As a solution to this problem, he did something that I have never seen before. He put logic in the Application.cfc around the application definition. It never even really occurred to me to put ColdFusion code before the CFAPPLICATION tag or the application definition in the CFC.

Here is my adaptation of his solution. I use the User Agent to set the Application session management. Additionally, browsers that I determine are "spiders," I set HasSessionScope variable in the request. Then, at any time during the request processing, I can determine if I need to alter my code to allow for non-session based usage.

// Define the application. To stop unnescessary memory usage, we are going to give
// web crawler no session management. This way, they don't have to worry about cookie
// acceptance and object persistance (except for APPLICATION scope).
if ((NOT Len(CGI.http_user_agent)) OR REFindNoCase("Slurp|Googlebot|BecomeBot|msnbot|Mediapartners-Google|ZyBorg|RufusBot|EMonitor|researchbot|IP2MapBot|GigaBot|Jeeves|Exabot|SBIder|findlinks|YahooSeeker|MMCrawler|MJ12bot|OutfoxBot|jBrowser|ZiggsBot|Java|PMAFind|Blogbeat|TurnitinBot|ConveraCrawler|Ocelli|ColdFusion", CGI.http_user_agent)){
 
	// This application definition is for robots that do NOT need sessions.
	THIS.Name = "KinkySolutions v.1 {dev}";
	THIS.SessionManagement = false;
	THIS.SetClientCookies = false;
	THIS.ClientManagement = false;
	THIS.SetDomainCookies = false;
	// Set the flag for session use.
	REQUEST.HasSessionScope = false;

} else {

	// This application is for the standard user.
	THIS.Name = "KinkySolutions v.1 {dev}";
	THIS.SessionManagement = true;
	THIS.SetClientCookies = true;
	THIS.SessionTimeout = CreateTimeSpan(0, 0, 20, 0);
	THIS.LoginStorage = "SESSION";
 
	// Set the flag for session use.
	REQUEST.HasSessionScope = true;

}

Want to use code from this post? Check out the license.

Short link: https://bennadel.com/39