Trying To Get The Most Trustworthy IP Address For A User In ColdFusion
On a recent Penetration Test (PenTest), one of our systems was flagged for not properly validating the X-Forwarded-For HTTP header, which is a recording of the various IP addresses along the network path being made by an inbound request. To be honest, I've never really thought deeply about IP addresses from a security standpoint before; but, having this show up on a PenTest sent me down a bit of a rabbit hole. And, I thought it might be worth talking a bit about why IP addresses pertain to security in ColdFusion.
When I first started programming in ColdFusion, everything lived on a single server. And, in those days, using the cgi.remote_addr to access the user's IP address was simple and safe - this CGI value was not a value that could be easily spoofed (as I understood it). As such, using the cgi.remote_addr for things like audit logging and rate limiting in ColdFusion was straightforward.
But, when we moved our ColdFusion server behind a load-balancer for the first time, things started breaking. This is because the load-balancer sat in between the user and the ColdFusion server and changed the value being stored in cgi.remote_addr: it was no longer the user's IP address, it was the load-balancer's IP address.
You can imagine that this immediately tripped all rate-limiting circuit breakers since all inbound request traffic appeared to be coming from just a handful of (internal) IP addresses.
To fix that issue, we started using the X-Forwarded-For HTTP header. This HTTP header is a comma-delimited list of IP addresses, starting with the user's IP address and followed by each subsequent reverse proxy's IP address.
And, I haven't given IP addresses too much thought since then. Until this Penetration Test. And when I started to read-up on using and validating X-Forwarded-For headers, I came across this amazingly in-depth article on "real" IP addresses by Adam Pritchard (March 2022). In that article, Pritchard states that the X-Forwarded-For header value should be considered a user-provided value and therefore cannot be trusted.
The cautions in his article have since been included in the Mozilla Developer Network (MDN) docs on X-Forwarded-For:
This header, by design, exposes privacy-sensitive information, such as the IP address of the client. Therefore the user's privacy must be kept in mind when deploying this header.
The
X-Forwarded-Forheader is untrustworthy when no trusted reverse proxy (e.g., a load balancer) is between the client and server. If the client and all proxies are benign and well-behaved, then the list of IP addresses in the header has the meaning described in the Directives section. But if there's a risk the client or any proxy is malicious or misconfigured, then it's possible any part (or the entirety) of the header may have been spoofed (and may not be a list or contain IP addresses at all).If any trusted reverse proxies are between the client and server, the final
X-Forwarded-ForIP addresses (one for each trusted proxy) are trustworthy, as they were added by trusted proxies. (That's true as long as the server is only accessible through those proxies and not also directly).Any security-related use of
X-Forwarded-For(such as for rate limiting or IP-based access control) must only use IP addresses added by a trusted proxy. Using untrustworthy values can result in rate-limiter avoidance, access-control bypass, memory exhaustion, or other negative security or availability consequences.Conversely, leftmost (untrusted) values must only be used where there will be no negative impact from the possibility of using spoofed values.
For someone like me who doesn't spend time thinking about how networks and proxies work, this language is a bit hard to parse. But, my understanding is that the X-Forwarded-For header is inherently untrustworthy; and, that we should only trust the value being provided by the first trusted proxy in the network hops.
Thankfully, Pritchard goes on in his article to talk about the different types of proxies that are commonly in use; and, how we - as ColdFusion developers - can lean on the various technical choices that said proxies are making in order write more secure code.
At work, our entire system sits behind Cloudflare. And, as Pritchard states in his article, Cloudflare injects an HTTP header value on inbound requests that we can consider "trusted":
Let's start with some good news.
Cloudflare adds the
CF-Connecting-IPheader to all requests that pass through it; it addsTrue-Client-IPas a synonym for Enterprise users who require backwards compatibility. The value for these headers is a single IP address. The fullest description of these headers that I could find makes it sound like they are just using the leftmost XFF (X-Forwarded-For) IP, but the example was sufficiently incomplete that I tried it out myself. Happily, it looks like they're actually using the rightmost-ish.
After reading this, I went to look up the Cloudflare docs on HTTP headers, and found the same concept:
CF-Connecting-IPprovides the client IP address connecting to Cloudflare to the origin web server. This header will only be sent on the traffic from Cloudflare's edge to your origin web server.
What I believe this all means is that the CF-Connecting-IP HTTP header value is essentially the cgi.remote_addr as seen from the CDN's perspective. Which means, it should be the non-spoofable IP address of whatever machine is connecting to the CDN.
Taking this information, I updated our ColdFusion (Lucee CFML) logic to give priority to the CF-Connecting-IP HTTP header if it is present. And, since we have to deal with Staging and Local development environments that don't sit behind Cloudflare, falling back to older means of access then user's IP address:
NOTE: In the following code, I include the use of the Java Commons IP Math library to actually try and parse the inbound IP address. I included that for funzies using Lucee's ability to load JAR files on the fly; but, in production, I just use the "light-weight validation".
<cfscript>
echo( "Hello from IP: #getRequestIpAddress()#" );
// ------------------------------------------------------------------------------- //
// ------------------------------------------------------------------------------- //
/**
* I get the most appropriate IP address for the current, incoming HTTP request.
*/
public string function getRequestIpAddress() {
// In Production, we are sitting behind the Cloudflare CDN. Cloudflare injects the
// header, "CF-Connecting-IP", which is (according to their docs) "the client IP
// address connecting to Cloudflare". Since Cloudflare is a "trusted proxy", this
// value represents the most trustworthy IP address in the request chain. If this
// header is available, it should take precedence.
var ipAddress = getHeaderValueSafely( "CF-Connecting-IP" );
// In non-production environments, we are NOT behind the Cloudflare CDN; but, we
// are behind several proxies (load balancers, nginx, etc). If the Cloudflare
// header isn't available, fallback to the "X-Forwarded-For" proxy-injected value.
// --
// CAUTION: This might be a USER-PROVIDED VALUE and should be consumed with much
// caution. A malicious actor might provide this header in a crafted request; and,
// most proxies are designed to just accept-and-augment any existing header.
if ( ! ipAddress.len() ) {
ipAddress = getHeaderValueSafely( "X-Forwarded-For" )
.listFirst()
.trim()
;
}
// If no header-based IP address is available, fallback to using the remote IP
// address.
// --
// CAUTION: Depending on your server configuration, this value may be an automatic
// reflection of one of the header-based IP values. For example, Tomcat can be
// configured to use the "X-Forwarded-For" header to populate CGI.remote_addr. As
// such, this value may also be considered a USER-PROVIDED value.
if ( ! ipAddress.len() ) {
ipAddress = cgi.remote_addr;
}
// Since we may be dealing with a user-provided value, we need to apply some
// validation, making sure the provided IP address looks like an IP address. If
// any of the validation fails, we'll return this fallback IP.
var fallbackIp = "0.0.0.0";
// LIGHT-WEIGHT VALIDATION: The longest valid IPv6 address should be no longer
// than 45-characters (from what I've read).
if ( ipAddress.len() > 45 ) {
return( fallbackIp );
}
// LIGHT-WEIGHT VALIDATION: Make sure that the IP-address (IPv4 or IPv6) contains
// only the expected characters. This doesn't validate formatting - only that all
// of the characters are in the expected ASCII range.
if ( ipAddress.reFindNoCase( "[^0-9a-f.:]" ) ) {
return( fallbackIp );
}
// HEAVY-WEIGHT VALIDATION: We could parse the IP address and make sure that it is
// semantically valid. However, since the IP address is already not to be trusted,
// I am not sure that the overhead of parsing it holds much value-add.
if ( ! isValidIpFormat( ipAddress ) ) {
return( fallbackIp );
}
return( ipAddress.lcase() );
}
/**
* I get the HTTP header value with the given name; or, an empty string if none exists.
*/
public string function getHeaderValueSafely( required string name ) {
var headers = getHttpRequestData( false ).headers;
if ( ! headers.keyExists( name ) ) {
return( "" );
}
var value = headers[ name ];
// While the vast majority of headers are simple string values, the specification
// allows for multiple headers (with the same name) to be provided in an HTTP
// request. ColdFusion combines those like-named headers as an index-based (ie,
// Array-Like) Struct. This UDF makes an arbitrary decision to collapse the header
// value into a list in such cases. Workflows that are expecting complex headers
// in the request need to use a non-generic solution.
if ( ! isSimpleValue( value ) ) {
value = headerValueStructToList( value );
}
return( value.trim() );
}
/**
* I convert the given complex HTTP header value to a collapsed, comma-delimited list.
*/
public string function headerValueStructToList( required struct value ) {
var items = [];
var size = value.size();
for ( var i = 1 ; i <= size ; i++ ) {
items.append( value[ i ] );
}
return( items.toList( ", " ) );
}
/**
* I use the Commons IP Math library to parse the given IP address (thereby validating
* that it is in the proper format).
*
* Commons IP Math: https://github.com/jgonian/commons-ip-math
*/
public boolean function isValidIpFormat( required string input ) {
var jarPaths = [ "./commons-ip-math-1.32.jar" ];
var IpClass = input.find( ":" )
? createObject( "java", "com.github.jgonian.ipmath.Ipv6", jarPaths )
: createObject( "java", "com.github.jgonian.ipmath.Ipv4", jarPaths )
;
// There is no validation method on the IP classes. As such, we simply have to try
// and parse the input value and see if an error is thrown.
try {
IpClass.parse( input );
return( true );
} catch ( any error ) {
return( false );
}
}
</cfscript>
As you can see in the getRequestIpAddress() method, I am giving precedence to the Cloudflare header first; then falling back to the other IP address values if the Cloudflare one doesn't exist.
In Prichard's article he warns against having generic fallbacks:
A default list of places to look for the client IP makes no sense
Where you should be looking for the "real" client IP is very specific to your network architecture and use case. A default configuration encourages blind, naive use and will result in incorrect and potentially dangerous behavior more often than not.
If you're using Cloudflare you want
CF-Connecting-IP. If you're usingngx_http_realip_module, you wantX-Real-IP. If you're behind AWS ALB you want the rightmost-ishX-Forwarded-ForIP. If you're directly connected to the internet, you wantRemoteAddr(or equivalent). And so on.There's never a time when you're okay with just falling back across a big list of header values that have nothing to do with your network architecture. That's going to bite you.
But, my fallbacks aren't generic. They are specifically looking for and giving highest precedence to the CDN that we use at work.
To test this, I tried making a local HTTP request to this above ColdFusion script using various combinations of header:
<cfscript>
// Try with both HTTP headers.
http
result = "apiResponse"
method = "get"
url = "http://127.0.0.1:57833/x-forward-for/target.cfm"
{
httpParam
type = "header"
name = "CF-Connecting-IP"
value = "101.101.101.101"
;
httpParam
type = "header"
name = "X-Forwarded-For"
value = "202.202.202.202"
;
}
echo( apiResponse.fileContent & "<br />" );
// ------------------------------------------------------------------------------- //
// ------------------------------------------------------------------------------- //
// Try with one fallback header.
http
result = "apiResponse"
method = "get"
url = "http://127.0.0.1:57833/x-forward-for/target.cfm"
{
httpParam
type = "header"
name = "X-Forwarded-For"
value = "202.202.202.202"
;
}
echo( apiResponse.fileContent & "<br />" );
// ------------------------------------------------------------------------------- //
// ------------------------------------------------------------------------------- //
// Try with no fallback headers.
http
result = "apiResponse"
method = "get"
url = "http://127.0.0.1:57833/x-forward-for/target.cfm"
;
echo( apiResponse.fileContent & "<br />" );
</cfscript>
And, when we run this ColdFusion code, we get the following output:
As you can see, when the CF-Connecting-IP is present, it takes precedence and is used as the "trusted" IP address for the inbound request. But, when that header is no present (which it won't be in non-production environments), we fallback to the X-Forwarded-For header and the cgi.remote_addr value.
In the world of web development, security is a never-ending journey. And, now that I'm using Cloudflare's CF-Connecting-IP as the source of truth for inbound IP addresses in production, I feel like I took a baby-step forward in securing our ColdFusion application.
More IP Address Learnings With Tomcat And cgi.remote_addr
As I mentioned earlier in this article, when we first moved our ColdFusion application behind a load-balancer years ago, the value of cgi.remote_addr was showing up as the internal IP of the load-balancer. Which is why we switched over to using X-Forwarded-For. I had no reason to believe this behavior had ever changed.
But, that was back when we were running ColdFusion 8 on a Windows server behind IIS. And, apparently, on our current stack, things have changed significantly with regard to cgi.remote_addr. Because, when I was testing my new IP address logic, I was shocked to see that cgi.remote_addr - despite being behind several reverse proxies - was holding my actual IP address.
I did some Googling and found this Lucee Dev Forum post on Docker. Apparently, in Tomcat, you can configure the server to use the X-Forwarded-For header to drive the cgi.remote_addr value:
<Valve
className="org.apache.catalina.valves.RemoteIpValve"
remoteIpHeader="X-Forwarded-For"
requestAttributesEnabled="true"
/>
Now, I don't know anything about low-level server stuff. So, I don't even know where this "Valve" is. But, given everything that I've read recently about IP addresses and security, this setting seems ... not great. Meaning, it's essentially taking the user-provided / untrusted X-Forwarded-For HTTP header and jamming it in what I have historically thought of as a trusted value, cgi.remote_addr. That's gotta be a bad idea, right?
I'll be discussing this setting with our security team.
Want to use code from this post? Check out the license.
Reader Comments
Welcome down the rabbit hole, Ben. :-)
FWIW, I did a blog post on this in 2019, addressing cf, Lucee, cloudflare, and the Tomcat valve--including where to find it and how to put it in place.
I also shared how that tomcat remoteip valve has still more to it, related to whether and how to trust the proxy reporting the ip header. I discussed there it's available internalproxies attribute, which some found they had to tweak (which was not trivial to do).
I also shared where to find more on the valve, which then was the Tomcat 8.5 docs (as that's what cf supported then). I can see there's now there much more detail there, including more examples, and discusssing also a trustedproxies attribute, which may be new since then. I don't recall. Of course, the tomcat 9 version of the docs also discuss the same things.
Here's my post, for those wanting to go further down the rabbit hole:
https://coldfusion.adobe.com/2019/04/get-cf-know-users-real-ip-address-behind-proxy-load-balancer-caching-solution-etc/
I point there also to a cloudflare technote on the matter.
Thanks as always for sharing your explorations.
@Charlie,
Oh awesome! I'll be sure to take a look during lunch. This is one of those situations where being a ColdFusion developer isn't just about the CFML - it seems more and more, we're having to know about the system architecture and the network. Exciting, but also daunting. And, with security, ColdFusion makes some things really easy (think
CFQueryParam,CFParam, etc); but, there's also so many things that can still go wrong so easily (having nothing to do with the CFML language itself).Baby steps!
Some interesting comments from James Moberg on Twitter:
I'll be discussing this with my engineers. Networking is not something that is immediately obvious to me.
@Charlie,
Great write-up, I wish I had seen it earlier :D It does explain why the
cgi.remote_addrbehavior changed. We started on ColdFusion 8, and it sounds, from your article, like Tomcat was only introduce in CF10, which is probably when theX-Forwarded-Forheader started getting pushed into thecgi.remote_addrfield.I really wish I know more about what happens in ColdFusion under the covers. I should see if there's a Udemy course or something on Tomcat. Or Java servlets. Or whatever the more generic term would be. I want to say "J2E", but I don't even know what that means 🤪
At work, we've been discussing this, and some engineers are concerned that a malicious actor could bypass the Cloudflare CDN and then inject their own spoofed
CF-Connecting-IPheader value. I am not sure how possible that it, since it would require a malicious actor to know our internal IP addresses, which I am not sure if something they can get at? But, I'm not a security expert by any means.One idea that we've been exploring is having Cloudflare inject a "secret key" header that the ColdFusion server could then use as an indication that the other header can be trusted. Interestingly, over on the Nginx site, they talk about a similar approach. Apparently, part of the more modern
Forwardedheader (less theX-prefix) uses a similar approach:So, it seems like including some sort of a secret might be a good idea.
Those headers, even cloudflare can be spoofed like you said- that is why validating the origin of the request matters.
If it has a cloudflare ip then you check for that header… if not, it's a spoof and move on.
The issue is how do you know what ips are in your list. James said he had a list… but in this cloud world I wonder if we know that as easily.
Should be able somehow as env vars
Definitely an interesting blog, thanks.
@Gavin,
Cloudflare has a definitive list of IPs. But, I have no idea how often it changes; and, if it does change and we have old values, it could be pretty bad because our app - depending on how it is authored - could suddenly stop trusting the IP and start viewing all requests from that IP as a the same client. Which would play havoc with things like rate-limiting.
I'm growing more partial to having Cloudflare just inject a secret key in the inbound request headers. This way, we control the key and the check for it, and the IPs can change.
I suppose the next thing is what do you do if the IP is wrong/faked, sometimes it may not matter. Other times you might want to cfexit and stop because its a clear sign of a malicious actor.
Its difficult to use IP to determine anything if you have a whole load of users coming from the same ip address. School students all have the ip address and hit your site at the same time is that 30 requests from one person or 1 requests from 30 different students.
At your work that could be a load of users in the same office. With one person "misbehaving".
@Adam,
Schools and Businesses being behind 1 IP address is definitely something we've had to deal with. This is especially true in our application when someone includes a link to the app in an internal company newsletter (or something to that affect), and then there's a "stampeding herd" of users all coming from the same place. This is often the reason that we have to be more relaxed with our IP-based rate-limiting than we might otherwise be.
Post A Comment — ❤️ I'd Love To Hear From You! ❤️
Post a Comment →