Using CFPOP And CFMAIL For Bidirectional SMS Text Messaging In ColdFusion
Yesterday, I looked into ColdFusion's CFPOP tag for the first time. The CFPOP tag is used to pull down and delete email messages from a POP-enabled mail server. As I stated before, I had to set up a free POP3 email account at LavaBit.com so that I could have non-SSL pop access compatible with my version of ColdFusion (ColdFusion 7). Once I had my account up and running (a two minute task), I did a little bird's eye view of the CFPop functionality.
The second I pulled down my first email message from the server, my instantaneous thought was, This can be used to receive SMS text messages. I have done a good deal of testing with the SMS text message email gateways including sending text messages as well as sending picture messages using ColdFusion's CFMail tag. This is good stuff, but CFMail creates only a single direction (outgoing) SMS text message. Using CFPop to pull messages off of the email server, I might now have the ability to receive incoming SMS text messages, thereby, completing the SMS circle of life.
As always, with new ideas, I like to rock a little "proof of concept" action. This proof of concept application uses a scheduled task to regularly pull down messages from the mail server using CFPop. It narrows those messages to only those messages coming from a cellular email gateway and then it loops over them, evaluates each message, and potentially returns an SMS text message using the CFMail tag.
To start off the application, I took my Application.cfm from the previous post and converted it into an Application.cfc (which I should have done in the first place). The Application.cfc doesn't do anything special; it really just defines the application and then initializes each page request. For testing purposes, I am storing the POP server config information in the REQUEST scope so that I don't have to re-initialize the application if things need tweaking. Since this is just a proof of concept, caching was not a concern.
<cfcomponent
output="true"
hint="Defines the CFPop application.">
<!--- Set up the application. --->
<cfset THIS.Name = "CFPopTesting {#Hash( GetCurrentTemplatePath() )#}" />
<cfset THIS.ApplicationTimeout = CreateTimeSpan( 0, 1, 0, 0 ) />
<cfset THIS.SessionManagement = false />
<cfset THIS.SetClientCookies = false />
<!--- Set page request settings. --->
<cfsetting
requesttimeout="20"
showdebugoutput="false"
/>
<cffunction
name="OnRequestStart"
access="public"
returntype="boolean"
output="false"
hint="Fires when prior to page processing.">
<!--- Define arguments. --->
<cfargument
name="TargetPage"
type="string"
required="true"
/>
<!---
Create the POP server login information. Lavabit's
UNSECURED POP3 port is 110.
--->
<cfset REQUEST.Pop = StructNew() />
<cfset REQUEST.Pop.From = "bennadel@lavabit.com" />
<cfset REQUEST.Pop.Server = "lavabit.com" />
<cfset REQUEST.Pop.Username = "bennadel" />
<cfset REQUEST.Pop.Password = "XXXXXXXXXXXX" />
<cfset REQUEST.Pop.Port = "110" />
<!--- Return out. --->
<cfreturn true />
</cffunction>
<cffunction
name="OnRequest"
access="public"
returntype="void"
output="true"
hint="Fires after pre page processing is complete.">
<!--- Define arguments. --->
<cfargument
name="TargetPage"
type="string"
required="true"
/>
<!--- Include the requested page. --->
<cfinclude template="#ARGUMENTS.TargetPage#" />
<!--- Return out. --->
<cfreturn />
</cffunction>
<cffunction
name="OnError"
access="public"
returntype="void"
output="true"
hint="Fires when an exception occures that is not caught by a try/catch.">
<!--- Define arguments. --->
<cfargument
name="Exception"
type="any"
required="true"
/>
<cfargument
name="EventName"
type="string"
required="false"
default=""
/>
<h1>
An Error Occurred.
</h1>
<!--- Return out. --->
<cfreturn />
</cffunction>
</cfcomponent>
Once the Application.cfc was in place, I set up a ColdFusion template designed to run as a scheduled task. This is currently the only other file in this application. This template, cfpop_task.cfm, grabs the SMS text messages off of the pop server and evaluates them. For the moment, there are only two valid actions to take:
If the SMS text message begins with "Echo", then I simply send back the succeeding message to the cell phone using CFMail. This is just to make sure the service is up and running.
If the SMS text message contains the difficult question, How Sexy?, then I simply return back the most obvious answer, "Too Sexy!".
Once those messages are processed, they are deleted from the mail server.
<!---
Get all the mail headers. We are going this because we
only care about the headers that come from appropriate
FROM addresses.
--->
<cfpop
action="getheaderonly"
name="qHeader"
server="#REQUEST.Pop.Server#"
port="#REQUEST.Pop.Port#"
username="#REQUEST.Pop.Username#"
password="#REQUEST.Pop.Password#"
/>
<!---
From the headres, filter the header query so that it
only contains data from the email gateways of a cellular
caller. This will be 10 digits followed by an "@" sign.
--->
<cfquery name="qHeader" dbtype="query">
SELECT
uid,
[from],
[date]
FROM
qHeader
WHERE
[from] LIKE '#RepeatString( "[0123456789]", 10 )#@%'
ORDER BY
messagenumber ASC
</cfquery>
<!---
Check to see if any headers were found to have come
from cellular-email gateway senders.
--->
<cfif qHeader.RecordCount>
<!---
Now that we have messages that are coming from
cellular email gateways only, let's go back to the
POP server and grab the content of those email using
the more comprehensive CFPOP action, getall. We want
to make sure ONLY to get the headers we have filtered
down to (use UID field to make this happen).
--->
<cfpop
action="getall"
name="qMessage"
uid="#ValueList( qHeader.uid )#"
server="#REQUEST.Pop.Server#"
port="#REQUEST.Pop.Port#"
username="#REQUEST.Pop.Username#"
password="#REQUEST.Pop.Password#"
/>
<!--- Loop over the cellular SMS text messages. --->
<cfloop query="qMessage">
<!---
Get the message from the SMS text message. Since
this is coming a phone, it should be an unformatted
TEXT message. Therefore, it should be save to grab
the BODY without haveing to use the TextBODY field.
--->
<cfset strMessage = qMessage.body />
<!--- Let's set up the default return message. --->
<cfset strReturnMessage = "" />
<!---
Now, we have to figure out what action to perform.
To do this, we are going to try to run regular
expression patterns against the message.
--->
<cfif REFindNoCase( "^echo .+", strMessage )>
<!---
This is just a test case to see if the system
is working. Here, all we want to do is echo
back to the user what they sent via their SMS
text message.
--->
<cfset strReturnMessage = REReplaceNoCase(
strMessage,
"^echo ",
"",
"one"
) />
<cfelseif REFindNoCase( "^how sexy\??.*", strMessage )>
<!---
In answer to the eternal question, give the
most appropariate answer.
--->
<cfset strReturnMessage = "Too Sexy!" />
</cfif>
<!---
ASSERT: At this point, we have run all of
our expressions against the SMS text message
request and have stored any return messages
into the variable strReturnMessage.
--->
<!--- Check to see if we have a return message. --->
<cfif Len( strReturnMessage )>
<!---
Send an SMS text message to the cell phone using
the cellular email gateways.
--->
<cfmail
to="#qMessage.replyto#"
from="#REQUEST.Pop.From#"
subject="">
<!--- Send our message as text content. --->
<cfmailpart
type="text/plain"
>#strReturnMessage#</cfmailpart>
</cfmail>
</cfif>
</cfloop>
<!---
Now that we have finished our message loop, let's
remove those messages from the POP email server.
Again, we ONLY want to remove the SMS text messages
that came from cellular email gateways, so make sure
that we limit the delete command to the UIDs found
in our messages query.
--->
<cfpop
action="delete"
uid="#ValueList( qHeader.uid )#"
server="#REQUEST.Pop.Server#"
port="#REQUEST.Pop.Port#"
username="#REQUEST.Pop.Username#"
password="#REQUEST.Pop.Password#"
/>
</cfif>
<!---
Set results page content type and clear the output
buffer to get rid of extra whitespace.
--->
<cfcontent
type="text/html"
reset="true"
/>
<cfoutput>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html>
<head>
<title>CFPop Task Page</title>
</head>
<body>
<p>
#qHeader.RecordCount# Message(s) processed.
</p>
</body>
</html>
</cfoutput>
Notice that once we grab the mail headers off of the server, we are then performing a ColdFusion query of queries to filter the query down to messages that came from cellular email gateways. While this is not a fool-proof methodology, I think limiting the recordset to FROM addresses that start with 10 digits followed by an @ sign is fairly accurate, at least for our proof of concept.
After we get are SMS text message headers, we then re-query the POP server for the full message of each of headers found above. Since we are assuming all of these emails are unformatted SMS text messages, I think it is OK to use the Body field which should contain plain text. These content bodies are then evaluated and a return message is defined.
Once the responses are all sent, we then delete the email messages from the POP server using the UIDs values in our message query. This will prevent the same text message from being processed multiple times.
Since this is just a "proof of concept", I don't really have a schedule task running (so please don't text my email address). Instead, I would simply refresh this page as a schedule task would do to invoke its actions. Also, if this were a real world scenario, I would definitely want to implement some locking around this whole process to single thread it such that multiple calls to this page don't end up processing the same messages before they could be deleted.
This is really cool. While it's not quite as cool as using an SMS short code (which costs hundreds to thousands of dollars a month for rental), I now have the power to create free bidirectional SMS text messaging in ColdFusion. And this is just scraping the surface; I feel like there's a whole LOT of potential here.
Want to use code from this post? Check out the license.
Reader Comments
This is one way to accomplish two-way SMS messaging without a shortcode, but I've heard stories about carriers blocking gateway access if your sending/receiving too much traffic. So I'm not sure I'd build any real sms apps with this approach.
Anybody out there using carrier email gateway's for sending txt messages to customers at a high volume?
@Dave,
I have also heard this. I attended BarCamp a few weekends ago in NYC ( http://barcamp.org/MobileCampNYC2 ) and this point was raised by one of the presenters. He basically said that this can be done until they take a look at the traffic and then they just blocking the service.
This seems REALLY lame to me. I would like to see some explanation as to why this happens. Is there an overhead to the email gateway that I am not aware of? It still counts as a text messaging charge for the sending phone (right?)??
Don't even get me started. Very lame. Your right, the carriers will charge the recipient to receive the message according to their data plan, so I'm not sure what the issue is.
I've come to the conclusion that in order to build any real system that relies heavily on SMS, you need a short code.
Hello Guys,
I've done a pretty fair amount with high volume sms type applications in the past, I used to sell it as a service as part of my advertising business, but we've since dropped it as we simply didnt have enough demand from clients to keep it on.
I've had this conversation a few times with different people and there is always a differing view on it. From my experiance I have never used POP as the method for anything other than small numbers. I've always purchased an account from a service aggregator and they will recieve requests from a bunch of different avenues, from POP, through to HTTP and special SNPP protocols that are supported by the CF Event gateways.
Most aggregators I have spoken too in the path reccomend HTTP as the simplest and most reliable way of sending/recieving large volumes of messages and this was always the appoach that I took when dealing with it.
I dont really get the cfmail version, it doesnt seem to offer any benefits over somthing like http, the code is more verbous and I would imagine a fair amount slower to process than http requests. I think this method was generaly developed for people looking to send sms campaigns from a desktop based client, like outlook, not for embedding in code.
If you are likely to be sending any real amount of data, other than the basic error alerts and things I would always reccoment speaking to an aggregator about it, they'll be able to help you build something which is scalable, and they'll likely put you on to http.
Rob
@Rob,
I am not sure what you mean by the HTTP method? I remember at BarCamp one of the presenters was talking about send and receiving SMS messages through XML data packets. Are you referring to the method these are posted / received? You can use CFHttp to "post" XML documents. I think maybe this is what you are talking about?
Sorry, I don't really know all that much about this stuff. But, I have to say I am curious. From what you are saying, it looks like you don't need an aggregation service to use this HTTP method? Do you know anywhere I can find documentation on this?
Thanks. And this is all for fun and learning; I am not planning to launch an SMS campaign any time soon.
Hey Ben,
When I say HTTP I mean simple GET and POST commands, from memory my aggregator could accept iether or, giving me the choice. So I could simply have a form which posted to thier action page, or use cfhttp to make the call, like this:
<cfhttp url="http://myaggregator.com/service">
<cfhttpparam type="formfield" name="Username" value="MyAccountUsername" />
<cfhttpparam type="formfield" name="Password" value="MyAccountPassword" />
<cfhttpparam type="formfield" name="RecipientList" value="32432423423, 32432432432, 234233432, 546546544" />
<cfhttpparam type="formfield" name="Message" value="This is my SMS message content to send." />
</cfhttp>
Obviously the method changes form aggregator to aggregator, but they're basicly supplying you with a REST web service which you can make your calls too, I'm sure you'll find others out there which supply SOAP based ones which would be even better :-D then we can call them like objects. You simply roll your request like I've done above and it makes a http request to thier sevice and provides a response, instantly, again somthing which I imagine would be less effective as a POP request.
Its a nice neat way of controlling these things, I'm a mssive fan of webservices and use them a great deal within my applications so perhaps thats why I favour this method, I just prefer this API style interaction over using POP, somthing about it seems more reliable and less cumbersom.
Rob
@Rob,
I see what you are doing. I didn't realize that you mean using this within the context of an Aggregator; I thought you were talking about some SMS hack that was easier.
This answers the question as to why use CFMail over a CFHttp call - CFMail does not require an aggregator, CFHttp does. Of course, not having an aggregator is a dangerous thing, but that is a whole other discussion.
@ Ben
Glad that helped explain the method a little better for you, sorry I wasnt able to let you in on any little 'hacks' involved in SMS. :-D
I would definatly advise using an aggregator, I'm not sure how things lie legaly over in the US, but over here in the UK we have pretty strict rules on the distribution of SMS, and just as importantly, the aquirement of the recipients contact details. By using an aggregator they will always audit and check on these things to make sure you are not accidentaly 'abusing' the system, it keeps you much safer, the financial penalties for over steping the line can sting a little bit (Just ask the BBC) ;-)
An aggregator also has lots of other benefits attached to it, especialy if you're looking into short codes (there are several different types of short code available) and also reverse billing for subscription type services. When dealing with reverse billing the payment tarrifs are all different dependant upon the recipients service prodvider, this can make life quite complicated and by using an aggregator they will manage that for you, keeping life simple for you.
Considering the relative inexpense of using an aggregator I would definatly advise looking into one if you're sending more than a few messages to your staff and developers.
Rob
@Rob,
There is no doubt in my mind that using an aggregator is a requirement for any serious piece of SMS work. For a real project, yes, it is relatively inexpensive. Of course, for someone like myself who is just out there to test new technologies and maybe find some new solutions, there is no way that I would pay hundreds for that privledge.
Of course, it should be not too complicated to wrap all the SMS features into some sort of interface component where you have things like:
SMS.SetMessage( to, from, subject, message )
SMS.GetMessages()
Where the internals could have a "development" mode that uses the Email gateways and a "production" mode that uses the SMS aggregator services. Actually, not so much a different mode, but probably two different CFCs that have the same interface.
Then, you could develop without paying and then move to production with the ease. Of course, what I remember from BarCamp was that there is a really lengthy review process of your campaign. I think it is fairly strict over here too. There are a lot of highly restrictive practices on what language can be used and the protocol for interacting with users. This has nothing to do with how things are built :(
Ben old Chap,
I'm not sure which aggregators you have been looking at but I know the ones I've worked with in the past have never charged any form of setup fee, you simply pay for messages sent, at perhaps somthing like 8 pence each, at least for simple 2 way messaging, this isnt the case with short codes, if you want short codes you have to pay a few hundered as a premium on a monthly basis, as there are only so many short codes to go around I guess.
I've luckily not been subject to a lengthly review process myself, but I do know that the aggregators keep a close eye on what is being distributed, and they tend to tighten things down if you're working with reverse billing, as they should. There is often the complicated rigmerole of having terms and conditions on all your advertising of the service and what not. :-D
This stuff is good fun to play around with, I have my application logging text me any 'critical' level exception alerts thrown to my application as part of my logging model, its nice to know whats going on in those kinds of cases, I really find it a great benefit and a quick instant addition to the basic email alerts.
Thanks for the great blog posts again mate,
Rob
@Robert,
Maybe I am getting confused between Short Code usage and aggregators. Can you use an aggregator to receive SMS messages that aren't going to short codes? I don't want to take all your time on this. I should probably just go to an aggregator site to look up some info :)
Two things come to mind:
1.) cfx_pop3pro for CFPOP replacement, allowing SSL
2.) The cell carrier blocking issue -
Ive actually been experiencing this recently, from a different perspective with my iPhone. Since an iPhone cant/doesnt send pictures via SMS technology, you are forced to send through your pop email setup on the phone. Thus, when sending to a friends carrier phone (ie verizon) they start to block your messages and they bounce when you have a few too many drinks on a saturday night and start drunk picture messaging.
No worries about my time at all Ben, god only knows you've given me plenty of good advice in the past, I enjoy being able to return the favour.
Lets talk a little bit about an aggregator, an aggregator is basicly a 3rd party service provider, like a middle man between you and the proper network service provider.
If you want to send lots and lots of messages, you can just go straight to a mobile service provider (like AT&T or somthing? I dont know the
US providers names, but the dudes who provide your cell phone) and use thier message sending servers using similar kinds of protocols, but, they generaly require you to send about a squillion gajillion messages a week to keep the account alive so its not in the least bit plausable for you to achieve it.
So, the idea is that you, and lots of other companies can join all your message sending requirements (or 'aggregate' them) through a 3rd party company. This is the job of an aggragator, they sit in the middle and provide you with a sort of proxied access to the service provider, whether you're sending from a standard number, or a short code, it doesnt matter.
Short codes are a completely seperate thing, they are like a facia over your normal number, a bit like a freephone number for your standard phone, you simply pay for the facia.
There are two main types of short codes available, the expensive type (thousands of pounds a month) which is an exclusive 5 digit short code, like 88666 which is ALL yours, and you keep it to yourself. Alternativly you have cheap(er) short codes, which run from hundereds of $ a month, which you share with lots of other people, but you have a 'key' word which identifies your number, so all texts sent to your short number 88666 and must begin with the word 'kinky' like 'kinky Ben is a pretty cool bloke' or somthing like that.
Reverse billing is slightly different, people have to send a message to you before they are added to your contacts list and effectivly 'subscribed' to your service, this text will come to a short code and will cost them a premium price, of which the sending service provider takes about 80% the aggregator another 10% and you get left with the creamy 10% which is left, so say a text costs your consumer $1, your consumers service provider will earn 80c the aggregator will earn another 10c and you will earn 10c yourself (the amount the service provider takes is like bloody daylight robbery).
You can usualy setup an account with an aggregator and use those lovely pop or http calls free of charge, you usual have to buy 100 text credits or somthing like that to get you started. In addition to this, you can also mask your number with a name if you want it to look more professional, so when I revieve messages to my phone, from my free aggregator account, it doesnt come from +44028376376343 it says it has come from 'Think Blue'. Provided you dont want to do reverse billing you wont really need a short code at all, you can achieve two way messaging and a professional image with the free accounts.
Hope that helps mate, if you have any questions give me a shout.
Rob
@Rob,
Thanks for the explanation. I have to let it set in a bit (I will read a few more times), but I see that I clearly had some of these concepts jumbled in my head. It seems like if you don't need a Short Code then things become much easier and less expensive. This is great to know. Thanks again for the time you put into that.
You're more than welcome mate, If you have any more questions then let me know.
Rob