ColdFusion CFPOP - My First Look
I have never done anything mail related with ColdFusion other than sending mail using ColdFusion's CFMail tag. After years of this, I think it's finally time to start making this a two way street - healthy relationships are all about give AND take. I want to start looking into ColdFusion's CFPOP tag to see how I can get mail off a server and thereby, open up an automated bidirectional line of communication using the pop mail server as the go-between.
Before I could even start looking into ColdFusion CFPop, I had to get a mail server that could handle non-SSL POP3 support. I can't use my gMail accounts because apparently they use SSL which ColdFusion cannot yet handle (as of CF8). So, I did a quick Google search for "Free Pop mail" and came across the site Lavabit. This site seems perfect for testing. Their FREE, basic account offers:
- Storage: 128 MB
- Advertising: No
- Virus Protection: Yes
- Statistical Spam Filter: No
- Incoming Message Limit: 1,024 messages/day
- Outgoing Message Limit: 256 messages/day
- Message Size Limit: 32 MB
This should be more than enough to provide the perfect ColdFusion CFPop testing environment. I would say the setup took about 2 minutes... and did I mention that the basic account is FREE :)
Once I got the mail server setup done, I had to start building my ColdFusion testing environment. I am going to keep this ultra simple while I get my bearings on how all the CFPop stuff works. I set up a simple ColdFusion Application.cfm template:
<!--- Set page settings. --->
<cfsetting
showdebugoutput="false"
requesttimeout="20"
/>
<!---
Create the POP server login information. Lavabit's
UNSECURED POP3 port is 110.
--->
<cfset REQUEST.Pop = StructNew() />
<cfset REQUEST.Pop.Server = "lavabit.com" />
<cfset REQUEST.Pop.Username = "bennadel" />
<cfset REQUEST.Pop.Password = "XXXXXXXX" />
<cfset REQUEST.Pop.Port = "110" />
Now, all of my testing code can use the centralized REQUEST.Pop configuration information.
Ok, so let's start taking a look at how ColdFusion's CFPop tag works. The CFPop tag has three actions:
- GetHeaderOnly
- GetAll
- Delete
GetHeaderOnly and GetAll both receive information from the POP mail server. The Delete action deletes a given message (or list of messages) from the mail server. The reason there are two "get" actions is for performance optimization. The GetHeaderOnly action gets information about the messages but leaves out the attachments and content information that can take more processing and transfer time:
<!--- Gather the email headers. --->
<cfpop
action="getheaderonly"
name="qHeader"
<!--- Server configuration. --->
server="#REQUEST.Pop.Server#"
port="#REQUEST.Pop.Port#"
username="#REQUEST.Pop.Username#"
password="#REQUEST.Pop.Password#"
/>
The email header information that comes back from the server is stored in a standard ColdFusion query object with the following columns:
- To
- CC
- From
- ReplyTo
- Subject
- Date
- Header
- MessageID
- MessageNumber
- UID
The To, CC, From, ReplyTo, Subject, and Date fields are pretty self-explanatory. The Header field contains the email header information that seems to have a lot of routing and "behind the scenes" information. The MessageID is some sort of unique identifier that is parsed out of the Header data. The MessageNumber seems to be the sequential "order ID" of the message as it sits in the mail box. The UID seems to be a unique identifier for that message that can be used to filter all of the CFPop actions (GetHeaderOnly, GetAll, and Delete); apparently it is the header's X-UID field, but this field is not present in the Header data.
Note that I am saying "seems to be" on several of these values above because really, I am not 100% where these values come from. This is the first time I am looking into this CFPop stuff, so please don't take this with any sort of authority.
When you use the GetAll action:
<!--- Gather the entire email messages. --->
<cfpop
action="getall"
name="qMessage"
<!--- Server configuration. --->
server="#REQUEST.Pop.Server#"
port="#REQUEST.Pop.Port#"
username="#REQUEST.Pop.Username#"
password="#REQUEST.Pop.Password#"
/>
... ColdFusion retrieves all of the above columns plus the following additional content and attachment columns:
- Body
- TextBody
- HTMLBody
- AttachmentFiles
- Attachments
- CIDs
The TextBody and the HTMLBody fields contain data from the different parts of the message if different Content Types were supplied (think CFMailPart [type = text/plain or text/html]). Needless to say, Text/Plain goes into the TextBody field and the Text/HTML goes into the HTMLBody field. The Body field always contains the first content part found in the email message (in top-down order). The attachments is a tab delimited list of attached file names and the AttachmentFiles is a tab delimited list of file path corresponding to the file names (the files are downloaded to the ColdFusion server as part of the CFPop action when AttachmentPath is supplied in the CFPop tag). CIDs is a structure that contains the key-value pair data for the embedded images in an HTML email. I have not seen this in action yet, but from the looks of it, the key is the name of the image file and the value is the CID that it corresponds to.
Once we get the email messages from the POP server, we can delete them using the Delete action:
<!--- Delete the given email message. --->
<cfpop
action="delete"
uid="82379091"
<!--- Server configuration. --->
server="#REQUEST.Pop.Server#"
port="#REQUEST.Pop.Port#"
username="#REQUEST.Pop.Username#"
password="#REQUEST.Pop.Password#"
/>
As you can see here, I am filtering the delete action using the UID attribute. This UID attribute can contain a single value or a comma delimited list of values that correspond to the UID fields in the GetAll and GetHeaderOnly CFPop actions. If you do not supply some sort of filtering UID (exclude the UID attribute in the CFPop tag), then all of the emails on the server will be deleted.
Ok, so that's like a birds eye view of the ColdFusion CFPop tag. Please don't look at this as a tutorial; this was mostly a gathering of notes for myself. There is a lot that can be done here, but I think the premise is fairly simple. Now, I just need to come up with a really cool idea that will help me learn how to fully utilize this functionality. My first instinct is to integrate this with some SMS text messaging through the cellular email gateways... but that's another post, and another adventure.
Links for my future study:
Want to use code from this post? Check out the license.
Reader Comments
The fun part comes when you are trying to render the html body. for instance, Outlook likes to throw a lot of Outlook specific tags in its message body. CID's are sent as attachments. The fun part comes in when you are trying to render the HTMLBody. You get to go in and parse out the CID tags and link it back to the attachment.
I have a sick sense of fun. ;)
Nicely done. CFPOP can be a real pain some days - depending on the server you are dealing with. Having an email account you know works with CFPOP is greatly helpful in debugging other email providers.
@Shane,
Good challenges are fun indeed. Nothing sick or twisted about it. I am sure I will build up to doing something cool like that.
@WTL,
I would imagine so. Luckily, this LavaBit.com place seems pretty good. From the testing I have done, it seems snappy enough and has not shown any problems with my action. I haven't messed around with any attachments yet, but so far, so good.
Ben - as always great explanation. I'd read your blog even if you were writing about CFPOOP and instead of CFPOP.
@Bruce,
Ha ha ha :) Thanks a lot. I hope I can do some cool stuff with CFPOP. It seems like there is all this potential here that I can run wild in.
Hi Ben,
you can integrate with gmail probably using IMAP...
read here
http://webtrenches.com/post.cfm/connecting-to-gmail-using-coldfusion
I really look forward to your postings on SMS integration. I've been meaning to set aside some time to research that myself, so if you beat me to it you will save me some time :).
Hey Ben, I was just researching CFPOP and SSL tonight when I came across your post. It turns out there is a way to support CFPOP over SSL: just go into the JVM properties and set it to use SSL over the specified port. I learned of this technique by looking at the code in a UDF at
http://cf_sslpop.riaforge.org/
I've found nothing yet to indicate whether CFPOP can be made to work with TLS (although you can make it work with SSL over port 995, although I have no idea whether that's technically using TLS or not).
@Tom,
This is interesting. Do you have any idea if it changes the JVM settings for just my page request? Or does that SSL setting stick across pages. I just hesitate to mess with settings that may or may not affect other applications running on the same box.
@Anuj,
It looks like far more code goes into using IMAP over POP. I will look into IMAP stuff after I get a little more comfortable with the POP world. This is my first foray into the world of "receiving mail" so I want to take it slow.
@Ben,
Regarding whether changing the javaService's pop3 properties affects other applications or other sessions, the answer is that I think it does, but not in a bad way. After changing the javaService properties to enable SSL in one session (for a connection to a Yahoo! mailbox over port 995), I dumped the same properties in another session (for a connection to my own QMail server over port 110). Both sessions showed the same, new settings with SSL enabled. But cfpop continued to work correctly for both sessions, even though my own server doesn't support SSL.
So let's put it this way-- I think the changes to the java properties do affect other sessions and applications, but my guess is that they only enable the use of SSL. The changes don't seem to make all cfpop calls require SSL.
That being said, I am definitely a noob when it comes to the underlying Java platform. Do share a little uneasiness about this technique, and I would love to hear the opinions of someone who's more of an expert. I'll contact the UDF's author to see if he can shed any light on the discussion.
@Tom,
Good point as to the minimal impact. Like you, my only uneasiness is that I don't know much about the underlying Java platform (only some of its classes). I suppose enabling SSL can never really harm other projects.
You can also use Stunnel. I've been successfully using it for a long time now and was hoping to quit using it in CF8 (with new CFMail SSL/TLS support) but nope... still is needed as CFPOP was left behind.
More here:
http://www.harelmalka.com/?p=64
and here:
http://www.stunnel.org/
@Harel,
Thanks for the tip.
This is a great introduction to the use of CFPOP. Recently I was asked to write an app that would go through about 8,000 emails, filter the bouncebacks, and retrieve all the bouncing email addresses that came back from a recent newsletter we sent out. I immediately thought of CFPOP even though I had never used it before. The second link from my Google search took me here.
Initially, I couldn't get CFPOP to work because of the SSL requirement our mail server has. Luckily, I found a workaround for that on livedocs.adobe.com. Once I got it to connect, all I had to do was loop through each email and filter out the addresses with some regular expressions. This tag made it so much easier than having to go through each message manually!
@Jose,
That's awesome my man! Glad to know that I could help out a bit. I've also been getting back into CFPOP and found a great link with SSL and GMail via POP:
http://techfeed.net/blog/index.cfm/2008/3/28/No-SSL-With-ColdFusions-CFPOP
Anyway, good luck with your endeavors!
Nice cfpop intro. I have just started using it and have a problem that no one so far has responded to on other sites so I thought I would see if you have any suggestions. My uids all have commas in them, so it basically makes them useless as far as I can tell. What should the uids look like? Running CF8. Here are a few examples of the uids I am getting:
1242171504.5738.mail16,S=7799
1256158656.31258.mail41,S=4187
1256160932.4879.mail4,S=5539
Since the uid attribute is a comma-separated list or single value, none of these uids work.
Thanks for any suggestions.
@Cheryl,
I think it's probably ok that the UUID has a comma in it. UUIDs are not numeric values - they usually contain alpha-numeric characters and dashes. As long as the mail client can use it to look up an item, it should be fine.
Are you getting an error when trying to use them?
Thanks for responding Ben. The problem is that the cfpop tag uid attribute is looking for a single value or a comma separated list of ids. Since the uid has a comma in it, the tag things it is handling multiple ids when really it is just one. So, the cfpop query does not find the messages by uid because the ids that cfpop is using are incorrect as it is breaking the single id into two ids because of the comma. I have not found a way to specify a different list delimiter. I also tried using pieces of the uid to see if that would work, but the message are not found. I found one other person reporting the problem (they posted a message to the CF8 documentation page for cfpop as did I), but no one else has commented. I have also opened a thread in the Adobe CF forum and posted messages to my local user group, but no one has responded with a solution. For now, while it is not ideal, I moved forward using the message id, but just seems that there should be some solution to this problem. The mail account is hosted on Network Solutions, and the site is currently running CF7 (not CF8).
Let me know if you have any suggestions. Your site is great by the way:)
Thank you very much,
Cheryl
@Cheryl,
Ahhh, very interesting. I didn't even think of that. I can't say that I have any advice to offer up - I'm sorry. I haven't played with this too much and I certainly never had to deal with that problem.
Ben - just thought I would give you an update (or in case anyone else is looking for a fix to this problem). Here is a reply back I received from Adobe from their bug reporter project:
"UIDs are generated by mail server. Probably there might be some setting in mailserver to generate in a particular format."
So, back to Network Solutions I go though I doubt I will get far. I worked around the problem by using the message ID, but Adobe should realize that the UID does not always work and include something in the docs.
Anyway, thought you might be interested.
@Cheryl,
Ok, that's good to know (that it might be a setting with the mail provider). I am eager to hear what you figure out with your hosting provider.
Hi Ben
Just a quick question
I have a mail system that sends mail out to lists using cfmail.
I am using cfpop to identify hard / soft bounces.
But what I need to do is identify which list the failed address belonged to so that i can delete from the list..
How can i identify that?
Thanks in advance.
Delon
@Delon
Check this UDF out.
http://cflib.org/udf/getEmails
My suggestion would be to run a query loop through whatever is returned from CFPOP. Use this UDF to extra email addresses from the body. Most of the times bounced emails are placed there. You can also do some filtering by subjects that are common to bounces such as "Delivery Notification," "Failed" etc.
@Jose Galdamez,
Im already using that UDF.
Problem is...
THe email subject is rarely the subject which was used in the mail which the system sent.
It is usually a bounce mail subject such as "Out of office".
But subsequently to that... i started playing around with the MailerID inside the CFMAIL tag.
It seems to be doing the trick.
Now JI just need to figure a way to filter that mailer ID out of the mail header.
@Delon
Good to hear you're making some progress. Maybe if you post the sample text that shows the Mailer ID one of us can write the RegEx you would need to extract the number. Good luck.
@Jose Galdamez
Hi Jose
The mailerID isn't showing up in the bounced mail.
Here is an example header returned from a bounced mail...
X-MSK: CML=1.002000
Received: from mail-08.jhb.wbs.co.za ([196.2.97.5]) by fusebox.co.za with MailEnable ESMTP; Mon, 01 Mar 2010 10:28:56 +0000
Message-Id:
Received: from localhost by mail-08.jhb.wbs.co.za;
01 Mar 2010 12:28:30 +0200
Date: 01 Mar 2010 12:28:30 +0200
To: delon@fusebox.co.za
From: "Mail Delivery System"
Subject: Delivery Status Notification (Failure)
MIME-Version: 1.0
Content-Type: multipart/report; report-type=delivery-status; boundary="54xgg.4WBgrsvEE.jxm5U.6+rT4iu"
Precedence: bulk
Return-Path: <>
Is there somewhere besides this forum that we can discuss this, as I am sure it will start to bug some people with all the messages?
Best regards
Delon
delon@fusebox.co.za
@Delon
Sure. Feel free to email me. galdamez@mac.com
I had to a similar project with 1,000+ bounce backs. Using a combination of CFPOP, some looping, and RegEx I was able to compile a list of about 95% of the bouncing email addresses. Afterwards, I ran a similar loop to remove the bounce backs from the server. Hopefully, some of the logic I used can be applied to your situation.
Cheers.
Hi Jose.
Thanks a mil.
Sent you a mail.
Best regards
Delon
@Delon, @Jose,
Been following the thread (didn't have anything much to contribute - not enough experience with mail receiving). Did you guys get to the bottom of this?
@Ben
Thanks for the follow up!
The root of the problem had to do with being able to trace bounced emails to specific records in a DB table. Let's say you run an email campaign and you get 1,000 bounce backs. For each bounce back, you would a) not want the sender to receive the bounce back and b) be able to flag that record in the DB as bouncing.
As you are probably aware, bounce backs are only as good as what's inside of them. Even then it's a bit of a pain to parse through the message to find the email address that's actually bouncing. Some servers don't even give you that much info.
I think when you are dealing with sophisticated bounce back tracking, you have to do something fancy along the lines of:
1) use CFMAIL's failto attribute to have all bounce backs go to an account like bounces@foo.com
2) generated random strings as you are looping through your broadcast e.g. something like 83i39ac948x903848c
3) append that string to your failto address as an alias so you now set failto = bounces-83i39ac948x903848c@foo.com
4) maintain a log of each random string to its associated email address
5) assuming the alias works, you can then comb through that account using CFPOP and associate the bounce with the outgoing message. The failto makes sure the sender doesn't ever seen the bounce back.
I'm unaware of how to make sure email aliases actually work though. That's a mail server setup I've never had to deal with.
@Jose Galdamez,
Hi Ben and Jose
1st of all.. big thanks to Jose for his Skype chat a few weeks back. Your time was much appreciated.
I have come up with a rather unelegant solution to my problem and would like Ben to review and advise.
Ben, i would like to mail you the cf file to check out and advise.
But basically what i did was similar to what Jose mentions...
1. I send out mailers to database lists of 30 000 plus
2. I have setup a catchall.
3. Each mail list has a unique failto address (eg: #listid#-#templateid#@abc.com
4. I run a daily scheduled task which uses cfpop to capture all failed addresses.
5. I read through body of the email for all email addresses and bounce code or bounce message.
6. I store the bounced mail in the db. Db fields are
a. EmailAddress
b. ListId (captured from the bounced mail address)
c. Hard / Soft / Unknown bounce (according to error code or message)
d. Date
I then do a loop through the email list table and delete all hard bounce email addresses.
I can also do reports then.
...The code works, but it needs to be refined.
Can i post my code tou you to take read through and advise?
Thanks for a great forum.
It really assists more people than you can imagine!
Best regards
Delon
Apologies...
The field name in the db for C. is "BounceCode"
It stores the code / message which is returned in the email.
Sorry for the confusion.
@Jose, @Delon,
Very cool approach. I have never dealt with bounce-back emails so this is all a learning experience for me.
To be completely honest, I never even knew what the "FailTo" attribute was for. I knew it had to do with delivery failure; but, I had never tested it and was not aware that it could be used to track bounce-backs.
Nicely done @Jose!
@Delon,
The approach sounds pretty solid to me. I am not sure I can offer any better advice at this time. In fact, I'd like to play around with the concept a bit myself.
@Jose, @Delon,
I just started trying to play with FailTo and I cannot for the life of me get it to work. I've tried using GMail as SMTP as well as SmarterMail as SMTP. In no case does a "failed send" every get to the FailTo email address.
Is there some sort of crucial logic that I am missing here in terms of what can be used for FailTo... or perhaps what triggers a failto?
@Ben
I've yet to test it myself, but I'm under the impression is only for bounce backs. Is that what you were trying it out for? I don't come back into work until Friday, but I can toy around with it more then. I would probably do this: send an email from account A to account B with a failto of account C. B has to be non-existent or a bad email address. If the "delivery failed" bounce back goes to account C then the test worked. Otherwise, one would have to run the test again, send to an existing address and disect the mail headers to see what went wrong.
@Jose Galdamez,@Ben Nadel
Hi fellas
@Jose... faito worked perfectly for me.
WHat i did was setup a catchall@abc.com
I made the failto address = #sendid#-#listid#@abc.com
All failed emails are sent to that catchall acocunt. The failto address makes it easy to identify which listid and sendid the address belongs to, aiding the mail reporting.
I then insert the bounce code or error message found into the db which allows me to identify hard / soft / unknown bounces :)
Been a massive learning curve for me!
But alas, it works wonders.
Let me know if you would like to share my code.
Drop me a mail and I'll gladly send the file.
Best regards
Delon
@Delon,
I appreciate that. I think part of what I'm dealing with is a "Gmail" issue. From what I have been reading all morning, it looks like GMail simply ignores any Return-Path headers (what failto sets) and uses the From address.
I'll do some more digging; it seems that when I use a non-Gmail SMTP server, this works much better.
@Ben,
Can we read mail by mail from the POP server as,I want to have attachments of each mail in different folders.
Iam using CFpop in this way.
<cfpop server="xyz"
username="xyz"
password="xyz"
attachmentpath="xyz"
action="getAll"
port="xyz"
name="qryGetMails"
generateUniqueFilenames = "Yes">
This is getting all the attchments from mails into the same folder.
@Deepak,
I'm not in front of a server right now, so I can't test; but, I believe that once you get the headers, you can loop over them and hit the the POP server to download one message at a time (using messageID) and store the attachments to a unique location using the attachmentPath attribute. But again, I am not in front of a server right now, so that it just guess work.
Call me crazy (or stupid, I guess), but I don't think "from" is that intuitive. I'd really like to be able to get the email of the person who sent the email. When I use the query object, I just get their name.
When I use CFPOP to pull out .ics files, they're not there! Usually sending a meeting invitation attaches an .ics file. Do you have an idea as to what is happening with them?
Thanks!