Skip to main content
Ben Nadel at Scotch On The Rock (SOTR) 2010 (London) with: Andy Allan
Ben Nadel at Scotch On The Rock (SOTR) 2010 (London) with: Andy Allan

Ask Ben: Grabbing World Of Warcraft Data With ColdFusion And CFHTTP

By
Published in , — Comments (17)

You've been doing a lot with XML lately, so I thought maybe you could solve a problem I'm having. I'm a World of Warcraft junky and want to pull data from Blizzard's "armory" web site. It's all XML, so should be easy, but when I do

<cfhttp url="http://armory.worldofwarcraft.com/....." method="get" result="armoryXML">

It returns the full HTML, post style sheet transformation. Yet go to that page and view source and you see the lovely XML. I've seen PHP code that hits the data on this site by doing the PHP equivalent of a cfhttp.

I tried your ColdFusion CFHttp method and did, indeed, get the transformed page content. Then, I went directly to the page in my browser and viewed the source. The source, just as you said it would be, was an XML document with an attached XSLT processing instruction. This was confusing since ColdFusion's CFHttp tag works just like any browser request. So, what's different between the CFHttp request and the browser's request that would render different content?

There is nothing about this that is obvious at all, but if you have worked with CFHttp for a long time, you might know that CFHttp causes problems because the user agent it broadcasts is that of the ColdFusion server's HTTP agent. So, when you go to a site with your FireFox or Internet Explorer browser, the browser generally broadcasts its user agent as a Mozilla or MSIE compatible browser but ColdFusion, on the other hand, announces itself as the "ColdFusion" user agent.

Knowing this, I did a little experiment, making your ColdFusion CFHttp call in two ways: one with no explicit user agent and one with the FireFox user agent:

<!---
	Store the user agent that I am using with my browser
	(you're damn right I use FireFox!). I am breaking this
	data into two lines for display purposes ONLY.
--->
<cfset strUserAgent = (
	"Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; " &
	"rv:1.8.1.3) Gecko/20070309 Firefox/2.0.0.3"
	) />


<!---
	Get the target URL that we are grabbing. This page
	provides an XML document with an XSL transformation
	processing instruction. We want to grab the XML without
	the data being transformed into HTML. I am breaking this
	data into two lines for display purposes ONLY.
--->
<cfset strURL = (
	"http://armory.worldofwarcraft.com/character-sheet.xml" &
	"?r=Bloodhoof&n=Castlereagh"
	) />


<!---
	Grab the target page without sending across any user agent
	information. When we do this, ColdFusion automatically puts
	in "ColdFusion" as the browser's user agent value.
--->
<cfhttp
	url="#strURL#"
	method="GET"
	result="objHTTP"
	/>


<!---
	Now, let's grab the same page, but this time, instead of
	letting ColdFusion send over a default user agent, we are
	going to explicitly define what user agent the HTTP reuqest
	should announce.
--->
<cfhttp
	url="#strURL#"
	method="GET"
	result="objHTTPWithUA"
	useragent="#strUserAgent#"
	/>



<!---
	Let's output the leading characters of the request without
	the user agent value.
--->
<p>
	<strong>Request With ColdFusion User Agent</strong>:
</p>

<p>
	#HtmlEditFormat(
		Left( objHTTP.FileContent, 500 )
		)#
</p>


<!---
	Let's output the leading characters of the request in which
	we explicitly defined the FireFox user agent.
--->
<p>
	<strong>Request With FireFox User Agent</strong>:
</p>

<p>
	#HtmlEditFormat(
		Left( objHTTPWithUA.FileContent, 500 )
		)#
</p>

Running the above code, we get the following screen output:

Request With ColdFusion User Agent:

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> <?xml-stylesheet type="text/xsl" href="/layout/character-sheet.xsl"> <html> <head> <title>The Armory</title> <script src="/shared/global/third-party/detection.js" type="text/javascript"></script><script src="js/cookies.js" type="text/javascript"></script><script src="js/armory.js" type="text/javascript"></script><script src="js/functions.js" type="text/javascript"></script><s

Request With FireFox User Agent:

<?xml version="1.0" encoding="UTF-8"?> <?xml-stylesheet type="text/xsl" href="/layout/character-sheet.xsl"?> <page globalSearch="1" lang="en_us" requestUrl="/character-sheet.xml"> <characterInfo> <character battleGroup="Ruin" charUrl="r=Bloodhoof&amp;n=Castlereagh" class="Hunter" classId="3" faction="Alliance" factionId="0" gender="Male" genderId="0" guildName="Veni Vidi Vici" guildUrl="r=Bloodhoof&amp;n=Veni+Vidi+Vici&amp;p=1" lastModified="May 18, 2007" level="70" name="Castlereagh" rac

As you can see, the first request in which we do not define a user agent (therefore letting "ColdFusion" be announced) sends us back the fully transformed HTML page complete with HTML, HEAD, and TITLE tags. But, in the second ColdFusion CFHttp request, where we announce the request as coming from a FireFox compatible browser, the true XML document is returned complete with XSLT, DocType, PAGE, and CHARACTERINFO XML nodes.

Now, just because I figured out what was going wrong, that does not mean that I can offer any insight into why this is happening. I assume this is just some functionality somewhere that is trying to be "clever" by servering up the content that it thinks the requester wants to see. And, for some reason, it assumes that FireFox wants the XML, but the ColdFusion request wants the transformed HTML page. Who knows... maybe someone reading this can offer more insight in that respect.

Want to use code from this post? Check out the license.

Short link: https://bennadel.com/713

Reader Comments

44 Comments

All you have to do is pout World of Warcraft in the title, and you've got my attention! ;)

I'm going to play with this tomorrow if I have time, I wonder what kind of cool stuff you can do...

1 Comments

I have only recently started blogging but my first couple of entries were on this very topic. Hopefully your other readers will find it useful. I should finish with an entry on how to breakout raiders by role using the armory data later this weekend:

http://blog.shooksweb.com/index.cfm/2007/4/20/Pulling-World-of-Warcraft-Armory-data-with-Coldfusion
http://blog.shooksweb.com/index.cfm/2007/4/25/World-of-Warcraft-Armory-Data
http://blog.shooksweb.com/index.cfm/2007/4/26/Burning-Crusade-Progress-Page
http://blog.shooksweb.com/index.cfm/2007/5/5/Burning-Crusade-Progress-Page-Part-2

20 Comments

@Matt

Very nice! Wish I knew about your work months ago! I searched all over in February for other folks doing armory scraping with CF, only found ASP and PHP users. I had first wanted to pull armory data because I too was hosting my guild's web site. Then one evening the guild leader decided to go to a paid site, saying mine didn't have enough features. Yet he never requested I add anything that the other site had that he viewed as a must-have. I gave the guild the bird and switched servers. Now I just want the armory data to see what sorts of interesting things I can come up with and to practice some with XML manipulation.

If there's room for another developer on your guild's site, drop me a line.

1 Comments

I actually thought you were playing World of Warcraft as well. However, I am going to take you up on the offer of posting this as a comment to this topic. I assume readers of this topic are WoW players, so this offer is more targeted at them:

We recently released a World of Warcraft Paladin Guide:
http://www.killerguides.com/guides/wow/guide/world-of-warcraft/paladin

Right now we are seeking a few reviewers who can provide us with some additional feedback. If you run a community site or blog we can also provide you with some copies as contest prizes (please use the contact form on the site to get in touch with me).

1 Comments

I'm guessing the reason for this is that Firefox/IE are known to support all the different technologies used by the armory (xsl, ajax etc.), whereas any less-advanced browser might not, so to make sure the content is readable, they're sending a simplified version, which might work on something like a mobile phone! :-)

20 Comments

lol, armory on your cell phone! "I wish I knew how I did in PVP last night, but I'm stuck in this all day meeting, I can't get on the armory to check! ... Oh wait, I can!"

1 Comments

Thank you, thank you, thank you!

This annoying problem had me stumped for hours.

In my case, I was using stanard Java classes to make the requests.

1 Comments

I think this entire thing might be a load of junk personally. Sites have been taking info for a long time now. Its not like something new has entered the filed. Just look at the mmorpg sites or the other database sites out there. Thats 100% what they are doing to help them get all the info from the game and advertise there things.

1 Comments

Thanks for clearing this up! I've been writing a C# application to pull achievement information from the armory, and for the life of me couldn't figure out why my app was returning the entire page, and not just the XML bits.

Thanks again!

3 Comments

Hi Ben,

Any idea of how to use CFHTTP method = "get" to populate the contents of a jQuery modal dialog?

We have a third party URL that we use for online support chat and it's presently in an ugly pop-up window.

Is this possible?

Mike

Oh my chickens, this post is old!
I believe in love. I believe in compassion. I believe in human rights. I believe that we can afford to give more of these gifts to the world around us because it costs us nothing to be decent and kind and understanding. And, I want you to know that when you land on this site, you are accepted for who you are, no matter how you identify, what truths you live, or whatever kind of goofy shit makes you feel alive! Rock on with your bad self!
— Ben Nadel