Ask Ben: Cross-Site AJAX Requests Using jQuery And ColdFusion

By Ben Nadel

Published 2009-02-15 in Ask Ben, ColdFusion, JavaScript / DHTML — Comments (18)

Using JQuery is it possible to use .load() or .ajax() to bring content from another site into your locally running page (Screen scraping)? I am basically trying to display the weather from the URL: http://weather.yahooapis.com/forecastrss?p=11385 into my div. Is this even possible with JQuery?

jQuery is the most awesome Javascript library ever created; but, as with all Javascript code, it exists in the security sandbox of the browser. Unfortunately (or fortunately, depending on how you look at it), the security of the browser prevents the client from making cross-site AJAX requests - that is, making an AJAX request to a page located under a different domain. If you try to make such a request, you will find that Javascript throws the following error:

Access to restricted URI denied

There are a number of ways that I have read about to allow Javascript to make cross-site AJAX requests; but, these all seem a bit hacky and are not always cross-browser compatible (not tested personally, but from what I have read - I may be ignorant on this matter). Personally, the method that I like the best is to create a proxy page under your own domain that acts as a server-side intermediary in your cross-domain AJAX. The purpose of this page is that it performs your AJAX request using server-side technologies that are not bound by the security of the client:

Cross-Site AJAX Requests Using A Server-Side Proxy Page To Get Around Client Security Model.

As you can see from the diagram, your client code makes the AJAX request to this proxy page rather than the target page. Naturally, we have to tell the proxy page where to make its request so, in addition to any url or form values, we have to pass along the proxyURL (target URL):

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html>
<head>
	<title>Cross-Site jQuery AJAX Demo With Proxy</title>
	<script type="text/javascript" src="jquery-1.3.1.pack.js"></script>
	<script type="text/javascript">

		// Run when DOM is ready.
		$(
			function(){
				var jForm = $( "form" );

				// Hook up form submit.
				jForm.submit(
					function( objEvent ){
						// Get weather from Yahoo.
						GetWeather( jForm );

						// Prevent default.
						return( false );
					}
					);
			}
			);


		// This method actually performs the cross-site AJAX
		// using a proxy ColdFusion page.
		function GetWeather( jForm ){
			// Hook up form submit to pull down weather from
			// Yahoo API using proxy AJAX request.
			$.ajax(
				{
					url: "ajax_proxy.cfm",
					type: "get",
					dataType: "xml",
					cache: false,

					// As part of the data, we have to pass in the
					// the target url for our server-side AJAX request.
					data: {
						proxyURL: "http://weather.yahooapis.com/forecastrss",
						p: jForm.find( ":text" ).val()
						},

					// Alert when content has been loaded.
					success: function( xmlData ){
						// Get the content from the response XML.
						var strData = $( xmlData )
							.find( "description" )
							.text()
						;

						// Load content into DOM.
						$( "#content" ).html( strData );
					}
				}
				);
		}

	</script>
</head>
<body>

	<h1>
		Cross-Site jQuery AJAX Demo With Proxy
	</h1>

	<form>

		<p>
			Zip Code:<br />
			<input type="text" name="zip" />
			<input type="submit" value="Get Weather" />
		</p>

	</form>

	<div id="content">
		<!-- Here, we will store the AJAX response. -->
	</div>

</body>
</html>

As you can see in the demo, when the DOM loads, we are binding the form submit to fire the Javascript method, GetWeather(). This method then initiates an AJAX request to our proxy page, "ajax_proxy.cfm". Because this proxy page is on the same domain as the current page request, there are no security concerns. Notice, though, that in the data for our request, we are passing along the actual target URL (proxyURL), the Yahoo Weather API url: http://weather.yahooapis.com/forecastrss.

When the form is submitted, our proxy page makes the request and returns the resultant XML response to the client which uses jQuery to grab the content and inject it into the client's DOM.

Ok, so now that we see how this proxy page is being used, let's take a look at the code. This proxy page code can get more complicated than the following demo if you need to deal with cookies and authentication; but, I have found this ColdFusion code to be quite dependable for most generic situations:

<!---
	Check to see if the page request is a POST or a GET.
	Based on this, we can figure out our target URL.
--->
<cfif (CGI.request_method EQ "get")>

	<!--- Get URL-based target url. --->
	<cfset strTargetURL = URL.ProxyURL />

	<!--- Delete target URL. --->
	<cfset StructDelete( URL, "ProxyURL" ) />

<cfelse>

	<!--- Get FORM-based target url. --->
	<cfset strTargetURL = FORM.ProxyURL />

		<!--- Delete target URL. --->
	<cfset StructDelete( FORM, "ProxyURL" ) />

</cfif>


<!---
	Remove any AJAX anit-caching that was used by jQuery. This
	is a random number meant to help ensure that GET URLs are
	not cached.
--->
<cfset StructDelete( URL, "_" ) />



<!---
	Make the proxy HTTP request using. When we do this, try to
	pass along all of the CGI information that was made by the
	original AJAX request.
--->
<cfhttp
	result="objRequest"
	url="#UrlDecode( strTargetURL )#"
	method="#CGI.request_method#"
	useragent="#CGI.http_user_agent#"
	timeout="15">

	<!--- Add the referer tht was passed-in. --->
	<cfhttpparam
		type="header"
		name="referer"
		value="#CGI.http_referer#"
		/>

	<!--- Pass along any URL values. --->
	<cfloop
		item="strKey"
		collection="#URL#">

		<cfhttpparam
			type="url"
			name="#LCase( strKey )#"
			value="#URL[ strKey ]#"
			/>

	</cfloop>

	<!--- Pass along any FORM values. --->
	<cfloop
		item="strKey"
		collection="#FORM#">

		<cfhttpparam
			type="formfield"
			name="#LCase( strKey )#"
			value="#FORM[ strKey ]#"
			/>

	</cfloop>

</cfhttp>


<!---
<!--- Debug most current request. --->
<cfset objDebug = {
	CGI = Duplicate( CGI ),
	URL = Duplicate( URL ),
	FORM = Duplicate( FORM ),
	Request = Duplicate( objRequest )
	} />

<!--- Output debug to file. --->
<cfdump
	var="#objDebug#"
	output="#ExpandPath( './ajax_prox_debug.htm' )#"
	format="HTML"
	/>
--->


<!---
	Get the content as a byte array (by converting it to binary,
	we can echo back the appropriate length as well as use it in
	the binary response stream.
--->
<cfset binResponse = ToBinary(
	ToBase64( objRequest.FileContent )
	) />


<!--- Echo back the response code. --->
<cfheader
	statuscode="#Val( objRequest.StatusCode )#"
	statustext="#ListRest( objRequest.StatusCode, ' ' )#"
	/>

<!--- Echo back response legnth. --->
<cfheader
	name="content-length"
	value="#ArrayLen( binResponse )#"
	/>

<!--- Echo back all response heaers. --->
<cfloop
	item="strKey"
	collection="#objRequest.ResponseHeader#">

	<!--- Check to see if this header is a simple value. --->
	<cfif IsSimpleValue( objRequest.ResponseHeader[ strKey ] )>

		<!--- Echo back header value. --->
		<cfheader
			name="#strKey#"
			value="#objRequest.ResponseHeader[ strKey ]#"
			/>

	</cfif>

</cfloop>


<!---
	Echo back content with the appropriate mime type. By using
	the Variable attribute, we will make sure that the content
	stream is reset and ONLY the given response will be returned.
--->
<cfcontent
	type="#objRequest.MimeType#"
	variable="#binResponse#"
	/>

Note: I have left in some debugging code, but commented it out. The downsite of making a proxy-page AJAX request is that debugging can be a real pain. The debug code allows you to see the post and request made by the proxy page.

Once the ColdFusion CFHTTP request comes back, we echo the status code, response headers, and file content in its response to the client. This way, the client receives almost the same exact response as if it were calling the target page directly. The whole thing works quite nicely.

I hope this helps in some way.

Want to use code from this post? Check out the license.

Short link: https://bennadel.com/1498

Reader Comments

Rey Bango Feb 15, 2009 at 9:01 PM

16 Comments

Great post Ben! Note that you can make cross-site calls to APIs that return JSON if you use the $.getJSON() method and the API allows for a callback method to be specified.

Sebastiaan Feb 16, 2009 at 6:06 AM

61 Comments

Hi Ben, thanx 4 the insightful article, think I'm gonna start using it somewhere quite soon ;-) But how to get this dynamically, say on page load and in set intervals? How can the function then be called? And would these regular pings to the remote API not be seen as a spambot or something alike?

Ben Nadel Feb 16, 2009 at 8:47 AM

15,848 Comments

@Rey,

That sounds pretty cool. Will this work on *any* JSON-based API? Or, does the API have to know about this callback concept?

@Sebastiaan,

You could access this the same way you access any API. So, you could do this with the page loads or on a setInterval() or setTimeout() call. As far as a spam flagging, each API is going to have its own behavior that dictates how often you can call its API before you get in trouble.

Rey Bango Feb 16, 2009 at 10:40 AM

16 Comments

@ben: The API has to allow for a callback

Ben Nadel Feb 16, 2009 at 3:02 PM

15,848 Comments

@Rey,

OK cool, that makes sense.

Bucky Schwarz Mar 16, 2009 at 6:29 PM

2 Comments

@Ben,

Ray Camden actually recently wrote an article about the JSON format requiring a callback function = it's known as JSONP:

http://www.insideria.com/2009/03/what-in-the-heck-is-jsonp-and.html

Otherwise, thanks for this proxy solution.

Ben Nadel Mar 16, 2009 at 6:57 PM

15,848 Comments

@Bucky,

I saw that article,it's good stuff. The only down side to JSONP (JSON with Padding) is that it relies on the third party to implement such a convention. Of course, as that becomes more common place, it won't be such an issue.

ruleta en mesas Apr 16, 2009 at 6:48 AM

1 Comments

One of the things I've learned when it comes to ColdFusion development is that it almost always makes sense to go for simplicity. While not always possible, it doesn't hurt to stop and ask myself if the route I'm going down to solve some problem could be simplified in some way. Let me share the problem, and then I'll share the complex solutions I tried until I got to a much simpler fix.

Ben Nadel Apr 16, 2009 at 7:52 AM

15,848 Comments

@Ruleta,

I'm always looking for a good, simple solution.

Nav May 24, 2009 at 9:21 AM

3 Comments

Thanks for this Ben, it really helped me out.

Nav Jun 23, 2009 at 9:22 PM

3 Comments

In the situation of an error like - page not found or an error code in the xml (<statusCode>-3001</statusCode>, how would you alert the user ? Any help would be apprecieated.

Thanks.

Ben Nadel Jun 24, 2009 at 8:26 AM

15,848 Comments

@Nav,

That depends on you AJAX framework. For me, for instance, I always like to return an API response with the following format:

Success: boolean,
Errors: array,
Data: any

This way, if anything goes wrong on the server, I can simply return a non-Success message with the given error. The client code (on the browser) then just needs to know to check the Success flag in the response.

Nav Jun 24, 2009 at 3:10 PM

3 Comments

@Ben,

Thanks for your help again.

Ben Nadel Jun 24, 2009 at 3:24 PM

15,848 Comments

@Nav,

No problem.

Jake Mauer Jul 7, 2010 at 12:28 AM

1 Comments

Hey Ben!

I'm attempting to use your code in order to load an RSS feed from our main site onto our mobile site on a different domain.

It works great on many feeds but gets tripped up with the following XML/RSS url: "http://www.idfive.com/tasks/feed/?feedID=843AB539-C139-9E73-E2172D809FA78B99"

After dumping the httpRequest error I get this:

error in: crossdomain.cfm?_=1278476437853&proxyURL=http%3A%2F%2Fwww.idfive.com%2Ftasks%2Ffeed%2F%3FfeedID%3D843AB539-C139-9E73-E2172D809FA78B99 \nerror:\nTypeError: a is null

I'm assuming it's because the request URL is funky. I appreciate any advice you have. Thanks!

Ben Nadel Jul 18, 2010 at 12:26 PM

15,848 Comments

@Jake,

Hmm, I'm not sure. I would try CFDump'ing out the FORM/URL values before you perform the CFHTTP request to see if there is anything in them that you can see, such as a Url-encoded value that shouldn't be encoded.