Including Custom SMTP Headers For Debugging Using CFMailParam In Lucee CFML 5.3.6.61
Recently at InVision, I've been digging into the new Message Streams / Broadcast Streams feature that Postmark is currently rolling-out. This feature allows Postmark to serve both Transactional emails as well as list-serve emails; and, might help me cater to a few needs of InVision's Enterprise customers. According to the Postmark documentation, you can choose the desired "message stream" by including the custom SMTP / Mail header, X-PM-Message-Stream. This got me thinking about custom email headers - a feature of the ColdFusion CFMail tag that I've never used before. It got me wondering if I could use the CFMailParam tag to inject debugging information right in my outbound emails in Lucee CFML 5.3.6.61.
Just as with an outbound HTTP request, ColdFusion has the ability to add custom headers to an outbound SMTP message. Except, instead of using CFHttpParam[type=header], we use the CFMailParam tag with a name / value pairing. I use HTTP Headers to inject helpful debugging information all the time (as do many web application); so, why not use SMTP Headers to inject the same kind of helpful debugging information.
In an InVision context, it would often be helpful to embed relevant IDs. When we include links in our emails, those links also contain IDs (and are securely signed to prevent tampering); but, when investigating email-delivery issues, it's not very user-friendly to have to look at IDs embedded within the very messy and very noisy HTML message content. Instead, it would be great to pull those IDs out of the content and duplicate them as custom SMTP headers.
CAUTION: Like HTTP headers, SMTP headers can be inspected by the client. As such, you should never include "secure data" in a custom SMTP header that you do not want the client to see.
To explore this idea, I've mocked up an email that would be sent when one InVision user leaves a comment on a prototype for another InVision user. In addition to the comment body, we're going to include contextual IDs for the relevant users, the conversation, and the comment in the message header:
<cfscript>
// For the purposes of this demo, let's imagine that one user is posting a COMMENT.
// And, that such an action triggers an email that gets sent to another user on the
// same team. Here's the mock data for this workflow:
// The user leaving the comment.
authenticatedUser = {
id: 1,
name: "Ben Nadel",
email: "ben@bennadel.com"
};
// The user who is meant to receive the comment notification.
targetUser = {
id: 4,
name: "Sally Smith",
email: "ben+sally@bennadel.com"
};
// The comment data itself.
comment = {
commentThreadID: 1001,
commentID: 2001,
content: "Hello Sally, how's it going?"
};
// ------------------------------------------------------------------------------- //
// ------------------------------------------------------------------------------- //
mail
to = targetUser.email
from = authenticatedUser.email
subject = "#authenticatedUser.name# has left a comment"
type = "html"
// Don't spool the email, I want to see immediately if it fails.
async = false
// NOTE: Normally this would be defined as an Application setting or in the Lucee
// server admin; but, for the sake of simplicity, I am providing these inline.
server = "smtp.postmarkapp.com"
port = "2525"
username = request.postmarkApiToken
password = request.postmarkApiToken
{
// In addition to the standard email content, we're also going to include some
// reference data so that our Support / Engineering team can more easily see the
// context in which this email was sent, should they need to debug anything.
// --
// CAUTION: This data is sent OPENLY with the email. DO NOT INCLUDE SECURE DATA
// in these SMTP HEADERS! The recipient can see this data in their email client.
// In this case, I am only including IDs, which have no inherent value other than
// for debugging.
mailparam
name = "X-InVision-From-User"
value = authenticatedUser.id
;
mailparam
name = "X-InVision-To-User"
value = targetUser.id
;
mailparam
name = "X-InVision-CommentThread-Id"
value = comment.commentThreadID
;
mailparam
name = "X-InVision-Comment-Id"
value = comment.commentID
;
```
<cfoutput>
<h1>
#encodeForHtml( authenticatedUser.name )# has left a comment
</h1>
<p>
#encodeForHtml( comment.content )#
</p>
</cfoutput>
```
}
</cfscript>
AWESOME SAUCE: Note the use of Tag Islands to embed HTML markup in my
CFScripttags - just a freaking awesome feature of Lucee CFML.
As you can see, just as with custom HTTP Headers, I'm using the de facto X- prefix to denote a header that is considered e**Xperimental / eX**tension. And, I'm further prefixing it with InVision to denote relevance to our application. Then, it's just a matter of including the IDs that are contextual to the commenting workflow.
ASIDE: It appears that the
X-prefix pattern has been deprecated by the Internet Engineering Task Force (IETF). However, the headers that I am including are not meant to convey behavior of the outbound message. As such, I do not see any issue with including theX-prefix for my purposes.
Now, if we run this ColdFusion code and look at the message record that shows up in Postmark, we can see our debugging SMTP headers:
As you can see, the custom SMTP headers that we added with the CFMailParam tag are easily visible within the raw source code of the email in the Postmark console. And, again, this could be super helpful when trying to debug email delivery problems; or, bugs in the way in which the message content has been generated by our application.
Custom HTTP headers have already proved to be super valuable for debugging purposes within a ColdFusion application. And now that I see how easy it is to add custom SMTP headers with the CFMail / CFMailParam tags in Lucee CFML 5.3.6.61, I think that the same type of debugging information can be embedded within my outbound emails. I'm looking forward to experimenting with this further.
Want to use code from this post? Check out the license.
Reader Comments
@All,
One thing I forgot to define here was the term SMTP. It stands for Simple Mail Transfer Protocol - and it's the standard by which email messages are sent around on the inter-webs. Think of it like HTTP is to a web page as SMTP is to an email.
Ben. Isn't it a bit dangerous to be exposing numeric IDs like that. If you can see the ID, then the recipient can see the ID, as well. And, anyone who has access to his/her e-mail client.
Wouldn't it be better to use a GUUID?
But, it is great that we can send debugging info, in a custom mail header. I must say, I haven't used this feature yet?
@Charles,
"Danger" here is a matter of perspective. There is nothing inherently dangerous about exposing IDs - having an ID shouldn't give an end-user information that becomes problematic. If all we're doing is trying to prevent an ID from being "guessable", then we have "security through obscurity", which any security person will tell you is not a winning strategy.
That said, there is a danger in having a predictable ID in the event that there is a security hole in your application architecture. If a malicious user were to detect a vulnerability in the app, then they could more easily take advantage of that vulnerability if the parameters provided to that end-point were easily guessable.
So, to that end, I would say that my Security Team (at InVision) would absolutely agree with you. They advocate that people never expose an auto-incrementing ID value. However, they will tell you that it is for the same reason I have given - that an auto-incrementing ID makes an existing vulnerability worse, even if there is nothing inherently wrong with the ID itself.
In my case, our application is quite old now, and was born and raised on auto-incrementing IDs. As such, we (on the legacy team) have no UUIDs to work with - we only have INTEGER values.
Ben. I tend to have both an auto incremental ID & GUUID in every table. I then use the former when I send IDs covertly and I allow the system to expose the latter to the public or in query strings.
I also use the incremental ID in SQL queries, because someone told me that the look up is quicker. But, this might be nonsense. It would be good to have your perspective on this?
@Charles,
I believe that it is the same strategy that newer teams at the company are doing: having both an internal ID that is "easy" and a UUID that is "hard" on each record. We had considered trying to retrofit the legacy app with that kind of approach; but, it's hundreds of tables with billions of records and only a handful of people on my team. As such, we decided to just soldier-on with the existing architecture.
So, all to say, it sounds like you are on the right track :D
@Ben,
@Ben, you can actually "Level Up" what you're doing here by taking advantage of the features that your email sending provider offers, Postmark for you.
Looks like Postmark has a feature called Custom Metadata that you pass as SMTP headers -- exactly what you're doing here. If you name them with a prefix of X-PM-Metadata-, they'll be recognized as PostMark automatically ... limited to 10 per message.
Benefits
That last point means that if you have something setup to track bounces, opens, clicks, spam reports, etc using the webhook data ..... then that JSON struct will also have your custom metadata.
We don't use Postmark, we use Sendgrid .... which has "unique arguments" that work in a similar way.
With Sendgrid, you actually add an entire JSON struct in the X-SMTP-API header with the unique args and all sorts of other things ... mail merge commands, assigned IP_Pools. We have access to that data in our delivery dashboards so we can monitor delivery by message type and SAAS client.
Looks like AWS SES calls them Message Tags.
If you're not using a Outbound SMTP service and are doing direct delivery, then these options wouldn't be available (unless you build them yourself).
But if you are using an SMTP provider, check their documentation and see if you're missing out on something.
@Aaron,
This is awesome (Insert "We're not worthy" GIF from Wayne's World)! I've been a customer of Postmark for like 8-years now, and I think - much to my chagrin - that I've not kept up with any of the improvements that they've made over that time. This has given me a wake-up call that I really need to go back and see what else they offer. Just looking at one of the links you shared, they have the ability for me to Tag messages for grouping within the dashboards.
Really really awesome stuff!
@Ben,
No doubt, powerful stuff at your fingertips. We're on the tail end of an outbound email sending rearchitecture, so I've been neck deep in it for the past couple of months.
Amazing how you can analyze it all. Happy building -- and maybe I'll see another blog post about your discoveries.
@Ben,
Thanks very much for the assessment;)
I never really know whether I am doing things correctly or not, half the time.
Especially, when I working on solo projects...
@All,
I wanted to do a quick follow-up post using the
X-PM-Metadata-prefix that Aaron turned me onto. It's basically the same exact demo as in this post; but, using different SMTP header names:www.bennadel.com/blog/3906-using-smtp-headers-to-send-custom-metadata-through-postmark-in-lucee-cfml-5-3-6-61.htm
Thanks again Aaron!