Understanding and Replacing Microsoft Exchange

 in
The hardest-to-replace Microsoft server software is the expensive, frustrating Exchange. Here's how IBM and Bynari sent it packing.
Forget the Transport and Focus on the Data

First, we observed that Outlook did around 95% of the work in the Microsoft Exchange model. Exchange, as its name implies, transfers data between or among Outlook users. Even so, Exchange controlled the transport protocol and that presented a problem. People use the term MAPI to describe the Exchange protocol for transferring data among Outlook users. I don't see MAPI as being the appropriate term.

When intercepting Outlook messages, we discovered large arrays of binary data instead of text. I recognized the data but didn't realize where I had seen it before.

The Different Faces of Outlook

Outlook runs in two different modes: Corporate Workgroup mode and Internet Mail Only mode. In the Corporate Workgroup mode, Microsoft turns on all of Outlook's highly regarded features. In the Internet Mail Only mode, Microsoft uses a completely different and undocumented application program interface (API) with a limited feature set. Without an Exchange server, Outlook doesn't function in the Corporate Workgroup mode at all and has a limited set of features.

In Outlook's Workgroup mode, or when connected to an Exchange server, its data moves around in binary form. That binary data becomes impossible to recognize by an uninformed observer.

While researching, I found a developers' site on SourceForge.net that was porting Open DCE to Linux. I e-mailed one of the developers who wrote back and mentioned that the Open Group contributed the code.

I went to the Open Group's web site, searched the archives and found an old article mentioning that Microsoft had licensed DCE. We downloaded the Open DCE code and, using the engine, shook hands with Outlook and then Exchange. We knew more about the transport protocol as a result. We also understood the presence of binary data streams.

What we discovered is Microsoft uses the distributed computing environment (DCE) as its transport when using Exchange and Outlook in the Corporate Workgroup mode. Microsoft provides a programming interface on top of DCE, which it calls MAPI. Still, underneath MAPI exists an open standards-based protocol (DCE), which Microsoft bought from the Open Group and modified.

One of the default functions in DCE automatically translates ASCII text into binary objects. Microsoft leaves the binary object undocumented. So most of the MAPI properties programmers tag wind up as binary code they would not recognize. To make matters a little more complex, Microsoft embeds the binary property code in a large array of null binary data, thus hiding it.

We began to understand the transport, but we realized that Outlook sent MIME attachments to other Outlook clients. Those attachments did not transform themselves into binary data. We concluded that Outlook also used encapsulation to pass attachments around, which led us to the TNEF object.

TNEF

Microsoft Exchange uses a number of programs it calls “service providers” that Linux users might call dæmons. Exchange service providers handle objects, which have state and behavior.

Transport neutral encapsulation format (TNEF) is a method to pass ASCII text, other files and objects, along with binary message data. The binary message data makes up the bulk of each TNEF object. TNEF encapsulates MAPI properties into a binary stream that accompanies a message through transports and gateways. Outlook can decode the encapsulation to retrieve all the properties of the original message. The TNEF object hides in MIME as an attachment.

When we found the properties, which created calendar events, we built a TNEF encoder and soon began sending calendar events to and from Outlook clients with SMTP. We immediately recognized that we could use internet transport protocols and turn on Microsoft's Corporate Workgroup mode without MAPI. We knew we had arrived when we saw Microsoft Knowledge Base Article Q197204, which says that Microsoft does not support our transport protocol in the Workgroup mode.

Exchange Client Extensions

With our primary goal being server-side calendaring, we needed to create a message store to hold our Outlook client objects. As we used an IMAP server, we needed IMAP support, which Microsoft did not provide in the Workgroup mode. So we had to find a way to add IMAP client support to Outlook. The facilitator for adding functionality to an existing Outlook client involved what most people think of as a plugin.

When Microsoft first released Exchange, Outlook didn't exist. Instead, Microsoft provided a set of Exchange messaging clients for its different Windows operating systems. Microsoft also provided an extensible architecture for those Exchange messaging clients. Client extensions provided developers with a way to change the default behavior of the Exchange client. When Microsoft released Outlook, it continued providing support for the Exchange client extension architecture for compatibility with existing client extension DLLs.

Client extensions allow one to alter the default behavior of the client. Microsoft saw the advantages of an extension as a convenient way to integrate customized features and new behavior directly into the client, instead of having to write a separate MAPI application. We, however, saw extensions as a way to add IMAP client services to Outlook in the Workgroup mode. Using this architecture, we added commands to Outlook menus, custom buttons to the toolbars and the ability to preprocess outgoing and incoming messages with IMAP client services.

Luckily, we had already written client libraries for IMAP when we built our Linux client the previous year. We simply needed to port them to Windows. Our familiarity with the function calls, headers and protocols reduced our overall effort.

Once we built a Microsoft DLL for the client functions, we added it as an Outlook extension. Luckily, it worked the first time we tried it. By choosing the rich text format (RTF) for mail and meeting invitations, our TNEF objects attached themselves to the messages. Because Outlook created the TNEF objects, it exchanged them without any problems.

At this point, we uploaded our messages to our IMAP folders using the Microsoft .pst file as store and swap space. Staying connected to Exchange and using our server for message stores, we noticed compatibility between the two systems. We dragged and dropped objects from the Exchange folders into our IMAP folders. By doing this we discovered that tasks, journal entries, calendar events and so on, all showed up in Outlook as if they arrived from Exchange. The calendar also worked perfectly.

______________________

Comments

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

Software

Keylogger's picture

Key Logger keystroke recording software records keystrokes of every user activity performed on the computer in encrypted log files records all typed emails conversations, chat records, website visited, password and documents lists. It can record keystrokes, Clipboard monitoring, records voice chat conversations, captures screen snapshots visual surveillance, records passwords, records chat conversation, records application activity, Monitor all users’ accounts. Key logger will not appear on the desktop, add/remove Programs, control panel and even hidden in installation path folders. It cans easily records voice chat conversation. Secretly monitor users activity hidden utilities store messenger chat visited website URLs save word excel power point documents files folders.

Exchange is easy to replace

Digital Networking's picture

There isn't anything that exchange can do that open source components can't. I found an article which explains exactly how to replace exchange here http://digitalnetworking.biz/how-tos/how-to-replace-exchange-with-open-s...

good article,thn

greymarkus's picture

These two steps can allow a programmer or a seasoned administrator to create an alternative service provider for Outlook and serve a number of conventional mail clients. Linux mail servers do not discriminate based on the platform one uses. One can use Netscape Mail, Outlook Express, Ximian Evolution, mutt or Pine, to mention a few of the available MUA. celebrex 200 mg

Microsoft Outlook the most stable software????

Wong Seoul's picture

"Microsoft Outlook the most stable software"
I can not disagree more. I am not sure how much email you receive everyday.. My MS Outlook crashes all the time.. I lose my emails all of a sudden.. I do not also use MS Exchange and recommend it..

Thanks

Just a comment

Jairo's picture

Funny how things are... I've been using Outlook for (at least) 10 years now. Without a single "hard crash" as you say, *never* lost any email, *never* had a reason to change. In the (few) times it began to crash from time to time, the reason always was some other software installation that turned it unstable. But it can happen with any other windows software.
In time: except for the extremely high cost, I use (also for several years now) and recommend MS Exchange Server. Having a good (and simple-tasks-based maintenance) you can run it for years with no complaints.
Regards,
Jairo

Zarafa full mapi implementation on Linux

Anonymous's picture

Take a look at the Exchange alternative Zarafa (www.zarafa.com). This product has a native MAPI implementation on your Linux server and even support for advanced Exchange features, like Freebusy times, Global Address Book and full TNEF support.

The best thing of the product is the AJAX based webaccess, which is completely in the Outlook "Look & Feel". See demo.zarafa.com.

Awesome work. Thanks for

Anonymous's picture

Awesome work. Thanks for saving the world. You're definitely earned your success and money.

I am a windows developer and

Anonymous's picture

I am a windows developer and worked primarily on windows. I still support linux and love it. I have just thrown windows out of my home computer (will love to do this at work when time comes). I am more than impressed by this effort. Keep up the good work.

Microsoft is doomed!

Anonymous's picture

I am so happy to see efforts like this. I wish for Microsoft to lose their stronghold in the marketplace as soon as possible because diversity in choice of Operating Systems means better security for everyone. Replacement of Microsoft's product lines, one by one, is a truly wonderful move in this direction.

Thanks.

I must admit to being more im

Outlook Express repair's picture

I must admit to being more impressed for it. I've even recommended it to all my friends.

what a waste of time... So

Anonymous's picture

what a waste of time... So have you kept up with Exchange 2007?

I thought not. This is a pointless excercise... How many programmer for how many
years will it take to keep up with Exchange? It would cost you a couple hundred grand
to keep up and you still couldnt handle it. Programmers arent cheap.

Exchange has competitors but they stink and exchange is cheap.

There's nothing "cheap"

Anonymous's picture

There's nothing "cheap" about Exchange!

Webinar
One Click, Universal Protection: Implementing Centralized Security Policies on Linux Systems

As Linux continues to play an ever increasing role in corporate data centers and institutions, ensuring the integrity and protection of these systems must be a priority. With 60% of the world's websites and an increasing share of organization's mission-critical workloads running on Linux, failing to stop malware and other advanced threats on Linux can increasingly impact an organization's reputation and bottom line.

Learn More

Sponsored by Bit9

Webinar
Linux Backup and Recovery Webinar

Most companies incorporate backup procedures for critical data, which can be restored quickly if a loss occurs. However, fewer companies are prepared for catastrophic system failures, in which they lose all data, the entire operating system, applications, settings, patches and more, reducing their system(s) to “bare metal.” After all, before data can be restored to a system, there must be a system to restore it to.

In this one hour webinar, learn how to enhance your existing backup strategies for better disaster recovery preparedness using Storix System Backup Administrator (SBAdmin), a highly flexible bare-metal recovery solution for UNIX and Linux systems.

Learn More

Sponsored by Storix