Interoperability Breakdown: Who's To Blame? IMAP Or E-Mail Vendors?

In the e-mail world, where possible, I used to urge organizations to stick with the Internet-standard Internet Message Access Protocol (IMAP) instead of using the addictive proprietary alternatives from Microsoft and IBM Lotus (found in Outlook/Exchange and Lotus Notes). Now, thanks to Google's GMail service, I realize I may have been mistaken.Once you're addicted to a proprietary technology, you've essentially relinquished control of some part of your IT's budget, security, reliability, and performance to the vendor of that technology. From that point forward, that vendor is in charge of how much you'll be paying for it, the overall security and reliability of your systems, and how fast they work. The option to take your business elsewhere (the option that normally gives you the upper hand over tech sellers) is no longer an option. Whether its file formats, e-mail and calendaring systems or databases involving proprietary stored procedure languages, it doesn't take long to pass that addicted point of no return and once you do, the idea of switching is too painful to consider.

Theoretically, where standards for interoperability exist -- for example IMAP and POP for e-mail clients and servers or MP3 for music players -- there also exists the opportunity to build your IT deployments around them in such a way that solution providers are beholden to you instead of the other way around. Using e-mail as the example (theoretically, again), if your e-mail clients used the SMTP protocol to send email and the IMAP protocol to keep e-mail folders and their contents in synch, then you'd be free (ok, freer) to switch e-mail servers or service providers without too much pain. If your current solution suddenly becomes too expensive to keep, too unreliable, or too threatening to the overall security of your IT infrastructure, you could exercise your options.

But, as is often the case with such standards, you may have to sacrifice certain luxuries that are common to proprietary alternatives. The ability to keep group calendars and e-mail folders in synch at the same time is a capability that's built into the proprietary solutions from IBM and Lotus. SMTP and IMAP may be standard alternatives for sending e-mail and keeping e-mail folders in synch. But as messaging protocols and APIs go, they are calendar-dumb.

Those of us purists who desperately want to avoid being locked into proprietary technologies must find other means (and often calendaring clients that are separate from our e-mail clients) in order to leverage the Internet's e-mail system as a group calendaring system too. It's doable. It's painful. But that's not what I'm here to complain about.

When Google decided last year to turn its Gmail service into one big honkin' IMAP-compliant e-mail server, a lot of us standards-happy people cheered the move. Finally, a major e-mail service provider was going to support IMAP in addition to the standard, but folder-dumb POP protocol for e-mail retrieval. But, in so doing, Google has also flushed IMAP's many weaknesses to the surface. At least I think they are IMAP's weaknesses. I could be wrong. Perhaps it's the current crop of e-mail clients that are dumb?

When the last version of IMAP was ratified in 1996, the idea that you'd tag e-mail with one or more keywords instead of moving them into one folder was unheard of. Gmail has no folders. It has labels (or what most of us call tags) and like with blogs, YouTube, or social bookmarks on http://del.icio.us, users of Gmail (and its enterprise cousin in Google Apps) are free to assign as many user-created tags to an email as they want.

On the Google-end of things (deep in the bowels of the Google-verse), such multi-tagging is no doubt a one-to-many relationship: one email record to which one or more tags are assigned. But on the IMAP-client side where every tag must be resolved into a different folder and where the idea of storing messages in more than one folder at a time is hardly doable through their own native user interfaces (now, Outlook 2007 has "Copy To Folder" in one of its MOVE TO FOLDER dialogs, but not all), the process of IMAP synchronization results in the generation of extra copies of each e-mail; one for each of the keywords it is tagged with (so it can go into the corresponding folders).

Most of the black magic is done on the Google side. But, to the extent that Google must generate multiple copies of a message and send them down the wire for storage in separate folders, the are implications for both storage and networks. When using Mac Mail to access Gmail for example, if through GMail's Web interface, you tag an email with five separate labels, you can see the Mac Mail's status area go into a synchronization frenzy as it resynchs every affected folder including the inbox (in the event that, as a result of tagging an email, it was also untagged for the inbox).

That last parenthetical statement is not meant to be casually tossed aside. As far as an IMAP client is concerned, many of Gmail's reserved labels (the Inbox, Starred items, All Mail, Drafts, Spam, Trash, etc.) are nothing more than other folders (nested under a parent called "Gmail"). I'm guessing that the IMAP protocol includes special accommodations for the Inbox since, regardless of which IMAP client I've tried (Outlook, Thunderbird, Mac Mail, and Mobile Outlook on Windows Mobile 6), the Inbox on the client side and the Inbox on the Gmail side manage to stay in synch with each other. However, the same cannot be said for the Drafts, Trash and Sent Mail folders which are way more problematic.

For example, on Gmail, you don't really delete mail (even though the user interface leads you to believe that that's what you are doing). Given how label-driven Gmail is, what really happens when an e-mail is "deleted" is, in one fell swoop, the "Inbox" label is removed and the "Trash" label is added. This is where the IMAP chaos takes over. As I just said, to an IMAP client, Gmail's Trash-label comes across as a child-folder that's nested under a parent-folder called "Gmail" (see image, above left). The same goes for all of Gmail's other reserved labels (they all appear under the Gmail parent).

Meanwhile, on the client side, where deleted e-mail is filed into a folder that often goes by an entirely different name (eg: Outlook's name for the folder is "Deleted Items"), synchronizing with Gmail over IMAP creates a new label on the Gmail side called "Deleted Items" (see image, right) and to Gmail, it's simply another label. Why is this problematic? Well, if you want all deleted mail to go to the place that Gmail really thinks is for deleted mail, then on the client side, you can't just press the delete key or delete icon in the user interface when you want to delete an e-mail. That's because, like with Gmail's Web interface, the act of deleting is really more a special macro that moves the e-mail into a folder that gets slightly different treatment from the other others. For example, no other folder comes with the option to permanently delete all mail or empty itself.

On the Gmail side, partially composed or unsent mail is tagged with the label "Drafts." On the IMAP client-side, the Gmail:Drafts folder is just a regular folder. Meanwhile, the IMAP client has it's own special folder where unsent mail is automatically deposited.

In the case of deleted mail, the solution, if you're accessing your Gmail with an IMAP-compliant client like Outlook, Thunderbird, or Mac Mail, is to resist the temptation to press the mail client's delete key or icon, and instead MOVE the e-mail into the folder that corresponds to Gmail's Trash label. As mentioned earlier, this is usually a child folder that nested under a parent "branch" called "Gmail."

For Sent mail and Drafts where those folders are populated automatically, there is no solution.

Whose fault is this?

Like I said before, it appears as though IMAP has some special sensitivity to the Inbox. So, the first question is, why not some special sensitivity to other folders that common to most e-mail systems (eg: Drafts, Sent Mail, Trash, Spam, etc.). The Spam one especially since, if the client and server were in agreement as to what folder should store all spam, then, by depositing mail into that folder on the client side, you'd essentially be sending instructions to any anti-spam technology on the server-side that tells it what is and what isn't spam. Right now, nothing like that exists in the "open" world. It's all proprietary.

So, perhaps IMAP needs fixing.

In terms of the email clients, it's bad enough that some of them (like Outlook) can't be configured to store their IMAP folders in any folder but the default folder on your C: drive. But to not be able to tell the client exactly which folders are for Drafts, Sent Mail, Deleted Items, etc. is insane. I was pleasantly surprised when, after sending its first e-mail, my new installation of Outlook 2007 asked me which folder should be used to keep copies of sent mail from that point forward. I pointed it to the same folder (aka "label") that Gmail uses. Unfortunately, I have no way of doing the same for Deleted Mail, Spam, or Drafts.

Is this the fault of the e-mail client providers? One could make the argument that, knowing these weaknesses on the client-side, Google could make it possible for us to reconfigure how Gmail labels Deleted Mail, Drafts, Spam, etc on the service side. This way, we could tell Gmail "Assign the same label that Outlook uses for deleted mail ("Deleted Items") to all deleted mail. Sounds good, right? Wrong. What happens if you introduce a third client into the mix that uses a completely different name for where it keeps deleted mail. That is, after all, one of the big advantages of IMAP: you should be able to keep multiple clients in synch with each other (ie: The GMail Web interface, Outlook, and your mobile phone's e-mail client).

There are some things that Google could do. For example, Gmail also has a label called "All Mail". If you click on it in Gmail's Web user interface, you see all sent and received mail. Looking at my "All Mail" label in Gmail, I even see histories of my AOL chats (how it does this, I have no idea. But it's cool to have). But just like with all of Gmail's other labels, the "All Mail" label results in a separate folder on the client side that contains all the same things that you'd see when you clicked on "All Mail" in the Web interface. That sort of redundant storage of e-mail messages (if that's the way the client handles such redundant tagging or foldering) could swallow a hard drive in no time. One nice feature would be to optionally allow certain labels on the Gmail side to be configured for invisibility to all IMAP clients.

I don't think I need to continue to get my point across. IMAP was designed with a 1996-style high-level of interoperability in mind. But it's 12 years later and, regardless of whose fault it is, that interoperability is no longer the high-level of interop that would be beneficial to all. Particularly since the proprietary solutions are such an inhibitor to choice on the mobile front. Fixing this problem -- whether those fixes involve repairs to the protocols, the servers/services, or the e-mail clients -- is imperative.

Finally, this sort of interoperability breakdown is exactly the sort of "failure to communicate" that I'd like to see us start addressing at the show whose DNA is all about interoperability: Interop. IMAP isn't the only place in the stack where there's friction. You must have some pet peeves of your own. What are they? Let me know and then, I'll help to elevate them and make them a part of the conversation that takes place at this Spring's Interop as well as at future editions.