Does Order Of Twitter API Parameters Make A Difference?
I've been having a hell of a time trying to clean up KinkyTwits - my ColdFusion powered Twitter client - for release. One big problem that I've been having is that the API seems to suddenly stop working. It will be chugging along fine and then all of a sudden, I get nothing but the following errors:
Twitter / Over capacity - Too many tweets! Please wait a moment and try again.
This error will keep coming back for hours. Then, completely randomly, it will work for a moment, and then stop again for hours.
Now, I know it can't be that the API is over capacity because when this starts to happen, I'll jump on TweetDeck and it's fine. I've been banging my head against the wall for many hours on this problem and I think I might have finally gotten to the bottom of the issue. I don't see this mentioned anywhere in the API documentation, but it seems as if the order of the Parameters passed in the GET request makes a difference. To see what I'm saying, take a look at this test code:
<!---
Make the first API request putting the SINCE_ID paramter
followed by the COUNT parameter.
--->
<cfhttp
method="get"
url="http://www.twitter.com/statuses/friends_timeline.json"
useragent="KinkyTwitsTwitterBot"
username="bennadel"
password="123xyz"
result="objTwitterRequest">
<cfhttpparam
type="url"
name="since_id"
value="1158700580"
/>
<cfhttpparam
type="url"
name="count"
value="200"
/>
</cfhttp>
<!--- Output response. --->
<cfdump
var="#ToString( objTwitterRequest.FileContent )#"
label="Request With SINCE_ID First."
/>
<!---
Make the second API request putting the COUNT paramter
followed by the SINCE_ID parameter (the rest of the code
is exactly the same as the first API request).
--->
<cfhttp
method="get"
url="http://www.twitter.com/statuses/friends_timeline.json"
useragent="KinkyTwitsTwitterBot"
username="bennadel"
password="123xyz"
result="objTwitterRequest">
<cfhttpparam
type="url"
name="count"
value="200"
/>
<cfhttpparam
type="url"
name="since_id"
value="1158700580"
/>
</cfhttp>
<!--- Output response. --->
<cfdump
var="#ToString( objTwitterRequest.FileContent )#"
label="Request With COUNT First."
/>
Notice that in the first request, I am sending the SINCE_ID followed by the COUNT parameter. This is the same order as it is listed in the API documentation (although no mention of order specification could be found). In the second request, I am then just switching the order of the parameters. Now, when I run this, very consistently, the first request returns JSON-formatted statuses and the second request returns the "Too many tweets! Please wait a moment and try again" error.
If you are thinking that this is happening because I am following one API request with another immediately after it, don't worry - I thought of that. If I run the code above, with the CFHttp tags reversed, I get the same, consistent outcome but in the reverse order (first request fails, second one succeeds).
Could it be that the order of the URL parameters in the Twitter API requests actually makes a difference? Is that possible? I can't see how that could be true in any sort of name-keyed system. Has anyone else had any experience with this?
So, why does this work sometimes for a while and then start failing all of a sudden? I assume it's because I am passing my Parameters collection through to my Twitter API execution method as a ColdFusion struct. I am then looping over that struct and creating CFHttpParam tags:
<!--- Make URL parameters for GET method. --->
<cfloop
item="LOCAL.Key"
collection="#ARGUMENTS.Parameters#">
<cfhttpparam
type="url"
name="#LCase( LOCAL.Key )#"
value="#ARGUMENTS.Parameters[ LOCAL.Key ]#"
/>
</cfloop>
Because ColdFusion has no inherent order for structs, the order of the CFHttpParams is generated randomly. Of course, from CFDump experience, we know that this order is not totally random - it does get stuck in a rut (or perhaps comes out alphabetical). Either way, if we need ordering to be specific, we simply cannot rely on ColdFusion structs. Looks like I'm gonna have to go back and start passing my parameters through using an array of name-value pairs.
Want to use code from this post? Check out the license.
Reader Comments
Could it be the API Limit being reached you can only make 100 requets in one hour? Not sure but that could be the reason?
@BigMadKev,
I thought that, but if I reverse the order of the CFHTTP tests above, they fail in reverse order. If it was an API request limit, they would both have to fail on repeated requests. But, the once with "since_id" first always works, regardless of test placement.
@BigMadKev,
Plus and API limit issue would return with a 400 status code - these are returning with a 200 OK status code.
In that case then it's time to blame the technology behind twitter ;)
As it's not the developers fault is it :)
Same as MySpace.com was the Technologies fault not the devleopers.
:p hehe
@BigMadKev,
I just switched from using Structs to using Arrays for my Parameters collection and its working :) Crazy stuff!
The only thing I can see is that they're using index-based arguments, rather than the actual argument key.
ie: count = arguments[ 1 ].
Then again, if that's whay they're doing the order would impact the actual values assigned.
Ben,
I am using the twitter api cfc from Ria Forge and I just looked at the internals and here is what I found-
<cfhttp
url="http://twitter.com/#arguments.resource#.#returnFormat#?#queryString#"
method="#arguments.method#"
username="#variables.username#"
password="#variables.password#"
useragent="ColdFusion/8.0">
<cfif arguments.method eq "post">
<cfloop collection="#arguments.params#" item="key">
<!--- Deal with nulls so passing the arguments scope works properly. --->
<cfif structKeyExists(arguments.params,key)>
<cfhttpparam name="#lCase(key)#" value="#arguments.params[key]#" type="formfield">
</cfif>
</cfloop>
</cfif>
</cfhttp>
That appears to be doing about the same thing that you are doing. I haven't had any problems whatsoever.
@Brandon,
I have no idea :) Like I said, mine was working for a long time and then stopped. Maybe ColdFusion started sorting the params differently? Now that I have switched to an array, its all good. Depending on what params you have, CF might just be ordering them correctly.
I'm throwing a hail mary here, but could it be that they're using some sort of hash for the requests you are sending as a key to throttle you and the order of the parameters results in a different hash?
@Todd,
I am not sure what you mean.
Check this:
<cfset a = createObject("java", "java.util.HashMap") />
<cfset a.put("keyOne", "valOne") />
<cfset a.put("keyTwo", "valTwo") />
<cfset b = createObject("java", "java.util.HashMap") />
<cfset a.put("keyTwo", "valTwo") />
<cfset a.put("keyOne", "valOne") />
<cfset c = createObject("java", "java.util.HashMap") />
<cfset c.put("keyOne", "valOne") />
<cfset c.put("keyTwo", "valTwo") />
<cfdump var="#a#">
<cfdump var="#b#">
A = B?<br />
<cfdump var="#a.equals(b)#">
<br />
A = C?<br />
<cfdump var="#a.equals(c)#">
DAMN! I got all excited thinking that was correct (that A != B but A=C), until I saw my copy paste error. Here is the correct code:
<cfset a = createObject("java", "java.util.HashMap") />
<cfset a.put("keyOne", "valOne") />
<cfset a.put("keyTwo", "valTwo") />
<cfset b = createObject("java", "java.util.HashMap") />
<cfset b.put("keyTwo", "valTwo") />
<cfset b.put("keyOne", "valOne") />
<cfset c = createObject("java", "java.util.HashMap") />
<cfset c.put("keyOne", "valOne") />
<cfset c.put("keyTwo", "valTwo") />
<cfdump var="#a.hashCode()#"><br />
<cfdump var="#b.hashCode()#"><br />
<cfdump var="#c.hashCode()#"><br />
A = B?<br />
<cfdump var="#a.equals(b)#">
<br />
A = C?<br />
<cfdump var="#a.equals(c)#">
A == B == C
So, at least in Java the order of the keys in a map does not result in a different hashcode for a hashmap. But do you see where I was going with it?
@Todd,
I am not sure I follow?? We are not passing the hash map to Twitter. The Struct is merely used as a collection over which we iterate to write out the CFHttpParam tags. So, collection-aside, the values are passed individually.
Right, but, my skewed thinking was that somehow on Twitter's side of the API they are taking the arguments that you are passing in and creating a hash of that argument structure as a key to identify your requests. My thought was that they use that key to throttle you if you send too many requests within a specified time period. If the order in which you passed the args (or, the order in which they received the args) resulted in a different hash code then theoretically your requests would appear to be from a different source if you passed them in a different order.
Crazy theory - just disregard me ;)
@Todd,
Thanks for getting creative with your thinking! :)
Ben ...
Have you increased your limits with twitter?
http://twitter.com/help/request_whitelisting