Software // Enterprise Applications
10:17 PM
Fred Langa
Fred Langa
Core System Testing: How to Achieve Success
Oct 06, 2016
Property and Casualty Insurers have been investing in modernizing their core systems to provide fl ...Read More>>

Langa Letter: E-Mail--Hideously Unreliable

A recent test by InformationWeek columnist Fred Langa shows that up to 40% of valid E-mails never reach the recipient. Here's what it all means to you.

The Test
The basic test concept was simple: I'd send one plain text, attachment-free E-mail to each volunteer. The content of the E-mail would simulate normal, safe, business or interpersonal correspondence. It would contain no deliberate or obvious spam or virus-filter triggers (e.g., no spamlike components, such as offers to enlarge this or shrink that; no attachments; no viruses; no HTML; no embedded scripts; etc.). The subject line also would be plain and general, neither designed to trigger nor avoid spam filters. To count as a successful delivery, the test E-mail had to be seen and responded to by a human; not by a mailbot, responder, or other auto-reply mechanism. The body of the message contained instructions for the volunteer either to send a new, original E-mail, or to forward the test E-mail to a unique reply mailbox. This was similar to the original sign-up process the volunteers had already used to join the test, and so wasn't a high hurdle. But it did ensure that simple automatic reply mechanisms couldn't skew the test results. Instead, only human-generated responses that arrived at the designated reply mailbox would count as a complete, successful communication.

I took steps to ensure that the test E-mails would be treated completely normally; subject to all the standard E-mail processing and filters that might be in place at the volunteers' ISPs, mail servers, or desktops. That meant that I wouldn't send the E-mails from a "" or other address that the volunteers might already associate with me, or already have in their "whitelists." (Whitelisted addresses are known to be safe; mail from those addresses is preapproved, and bypasses normal filtering.) So, I made arrangements to send the test E-mails from a name and address the volunteers hadn't seen before.

I'd planned the above basic approach when I thought I'd have maybe 500 volunteers. But with more than 10,000 volunteers available, I also was able to add tests for several additional E-mail variables. I did this by splitting the pool of volunteers into large subgroups. Each subgroup got a slightly different test E-mail; or got E-mails that were sent out in slightly different ways. (I'll detail the variables in a moment.) Each subgroup was given its own unique mailbox to reply to, so I could track the response rates separately.

I sent all 10,979 test E-mails from a private E-mail account, using normal desktop E-mail tools, during East Coast business hours on Nov. 17 and Nov. 18, a Monday and a Tuesday. I left the response mailboxes--the mailboxes by which the volunteers could acknowledge that they got the test E-mail--open for a week. I ran no mail or spam filters on the response mailboxes so all acknowledgements would be delivered untouched and intact. Recall that all the volunteers were expecting some kind of test E-mail to arrive, although they didn't know exactly when, where, or how. Recall also that these were motivated, E-mail-savvy test subjects. Given this, the overall response was astonishingly poor: Of the 10,979 test E-mails I sent, I received 6,551 acknowledgements; a gross success rate of just 60%; or more pointedly, a failure rate of 40%.

But not all the subgroups yielded the same performance. To see and understand the differences and to have a basis for analyzing the gross results, we need to go into more detail:

Test Group One
This test group was closest to the small-scale test I'd originally envisioned; intended to simulate an original, unanticipated (i.e. non-whitelisted or preapproved) correspondence from an unknown, but nonhostile personal sender.

This test group comprised 1,500 individually composed, addressed, and mailed messages. (This was by far the most labor-intensive test.) The messages were in plain text with no attachments and sent one at a time. Each message had one recipient--one of the volunteers. (I used the volunteers' addresses in the order in which I'd received them--first volunteer, second volunteer, third volunteer, etc.) The E-mails were sent from a name and E-mail account the volunteers had not seen before: Liam Sugob, or (a temporarily valid personal and E-mail address I set up at my Freetune.Com domain). The E-mails carried a totally generic subject line of "Hello."

2 of 6
Comment  | 
Print  | 
More Insights
Newest First  |  Oldest First  |  Threaded View
Building A Mobile Business Mindset
Building A Mobile Business Mindset
Among 688 respondents, 46% have deployed mobile apps, with an additional 24% planning to in the next year. Soon all apps will look like mobile apps and it's past time for those with no plans to get cracking.
Register for InformationWeek Newsletters
White Papers
Current Issue
Top IT Trends to Watch in Financial Services
IT pros at banks, investment houses, insurance companies, and other financial services organizations are focused on a range of issues, from peer-to-peer lending to cybersecurity to performance, agility, and compliance. It all matters.
Twitter Feed
InformationWeek Radio
Sponsored Live Streaming Video
Everything You've Been Told About Mobility Is Wrong
Attend this video symposium with Sean Wisdom, Global Director of Mobility Solutions, and learn about how you can harness powerful new products to mobilize your business potential.