Understanding and Replacing Microsoft Exchange
First, we observed that Outlook did around 95% of the work in the Microsoft Exchange model. Exchange, as its name implies, transfers data between or among Outlook users. Even so, Exchange controlled the transport protocol and that presented a problem. People use the term MAPI to describe the Exchange protocol for transferring data among Outlook users. I don't see MAPI as being the appropriate term.
When intercepting Outlook messages, we discovered large arrays of binary data instead of text. I recognized the data but didn't realize where I had seen it before.
Outlook runs in two different modes: Corporate Workgroup mode and Internet Mail Only mode. In the Corporate Workgroup mode, Microsoft turns on all of Outlook's highly regarded features. In the Internet Mail Only mode, Microsoft uses a completely different and undocumented application program interface (API) with a limited feature set. Without an Exchange server, Outlook doesn't function in the Corporate Workgroup mode at all and has a limited set of features.
In Outlook's Workgroup mode, or when connected to an Exchange server, its data moves around in binary form. That binary data becomes impossible to recognize by an uninformed observer.
While researching, I found a developers' site on SourceForge.net that was porting Open DCE to Linux. I e-mailed one of the developers who wrote back and mentioned that the Open Group contributed the code.
I went to the Open Group's web site, searched the archives and found an old article mentioning that Microsoft had licensed DCE. We downloaded the Open DCE code and, using the engine, shook hands with Outlook and then Exchange. We knew more about the transport protocol as a result. We also understood the presence of binary data streams.
What we discovered is Microsoft uses the distributed computing environment (DCE) as its transport when using Exchange and Outlook in the Corporate Workgroup mode. Microsoft provides a programming interface on top of DCE, which it calls MAPI. Still, underneath MAPI exists an open standards-based protocol (DCE), which Microsoft bought from the Open Group and modified.
One of the default functions in DCE automatically translates ASCII text into binary objects. Microsoft leaves the binary object undocumented. So most of the MAPI properties programmers tag wind up as binary code they would not recognize. To make matters a little more complex, Microsoft embeds the binary property code in a large array of null binary data, thus hiding it.
We began to understand the transport, but we realized that Outlook sent MIME attachments to other Outlook clients. Those attachments did not transform themselves into binary data. We concluded that Outlook also used encapsulation to pass attachments around, which led us to the TNEF object.
Microsoft Exchange uses a number of programs it calls “service providers” that Linux users might call dæmons. Exchange service providers handle objects, which have state and behavior.
Transport neutral encapsulation format (TNEF) is a method to pass ASCII text, other files and objects, along with binary message data. The binary message data makes up the bulk of each TNEF object. TNEF encapsulates MAPI properties into a binary stream that accompanies a message through transports and gateways. Outlook can decode the encapsulation to retrieve all the properties of the original message. The TNEF object hides in MIME as an attachment.
When we found the properties, which created calendar events, we built a TNEF encoder and soon began sending calendar events to and from Outlook clients with SMTP. We immediately recognized that we could use internet transport protocols and turn on Microsoft's Corporate Workgroup mode without MAPI. We knew we had arrived when we saw Microsoft Knowledge Base Article Q197204, which says that Microsoft does not support our transport protocol in the Workgroup mode.
With our primary goal being server-side calendaring, we needed to create a message store to hold our Outlook client objects. As we used an IMAP server, we needed IMAP support, which Microsoft did not provide in the Workgroup mode. So we had to find a way to add IMAP client support to Outlook. The facilitator for adding functionality to an existing Outlook client involved what most people think of as a plugin.
When Microsoft first released Exchange, Outlook didn't exist. Instead, Microsoft provided a set of Exchange messaging clients for its different Windows operating systems. Microsoft also provided an extensible architecture for those Exchange messaging clients. Client extensions provided developers with a way to change the default behavior of the Exchange client. When Microsoft released Outlook, it continued providing support for the Exchange client extension architecture for compatibility with existing client extension DLLs.
Client extensions allow one to alter the default behavior of the client. Microsoft saw the advantages of an extension as a convenient way to integrate customized features and new behavior directly into the client, instead of having to write a separate MAPI application. We, however, saw extensions as a way to add IMAP client services to Outlook in the Workgroup mode. Using this architecture, we added commands to Outlook menus, custom buttons to the toolbars and the ability to preprocess outgoing and incoming messages with IMAP client services.
Luckily, we had already written client libraries for IMAP when we built our Linux client the previous year. We simply needed to port them to Windows. Our familiarity with the function calls, headers and protocols reduced our overall effort.
Once we built a Microsoft DLL for the client functions, we added it as an Outlook extension. Luckily, it worked the first time we tried it. By choosing the rich text format (RTF) for mail and meeting invitations, our TNEF objects attached themselves to the messages. Because Outlook created the TNEF objects, it exchanged them without any problems.
At this point, we uploaded our messages to our IMAP folders using the Microsoft .pst file as store and swap space. Staying connected to Exchange and using our server for message stores, we noticed compatibility between the two systems. We dragged and dropped objects from the Exchange folders into our IMAP folders. By doing this we discovered that tasks, journal entries, calendar events and so on, all showed up in Outlook as if they arrived from Exchange. The calendar also worked perfectly.
Editorial Advisory Panel
Thank you to our 2014 Editorial Advisors!
- Jeff Parent
- Brad Baillio
- Nick Baronian
- Steve Case
- Chadalavada Kalyana
- Caleb Cullen
- Keir Davis
- Michael Eager
- Nick Faltys
- Dennis Frey
- Philip Jacob
- Jay Kruizenga
- Steve Marquez
- Dave McAllister
- Craig Oda
- Mike Roberts
- Chris Stark
- Patrick Swartz
- David Lynch
- Alicia Gibb
- Thomas Quinlan
- Carson McDonald
- Kristen Shoemaker
- Charnell Luchich
- James Walker
- Victor Gregorio
- Hari Boukis
- Brian Conner
- David Lane