Securing Sendmail with TLS

by Jose Nazario

One of the most common uses of network systems is for e-mail-based communications. As such, a great deal of attention has been given to securing e-mail systems. However, most of this focus has been in two areas: security at the message level, using PGP or S/MIME, and confidentiality in the authentication process for mail retrieval, i.e., using APOP or SSL-based IMAP. This still leaves mail server-to-server traversal vulnerable to several attacks.

The introduction of basic cryptography into a mail server-to-server session alleviates most of these risks. Mail transport agents (MTA) and software such as Sendmail or Postfix can be augmented to handle their communications over transaction layer security (TLS) channels. This provides for a number of services, including confidentiality, integrity protection and strong authentication.

This article will introduce the features and use of SMTP over TLS as defined in RFC 2487, as well as illustrate their setup in the Sendmail package. We will cover cryptography basics, though not delve into the mathematics of cryptosystems and simple mail transfer protocol (SMTP) and extended simple mail transfer protocol (ESMTP) basics.

Some Basics of Cryptography

We will outline the basic components of cryptography, which will be useful for understanding the setup and benefits of using TLS in ESMTP transactions. Further information on cryptography is available from the references listed in the Resources section below.

Private key cryptography, or symmetrical cryptography, utilizes a shared secret, the key, for both encryption and decryption of a message. Input data is mathematically processed using the algorithm and the key to produce the ciphertext output, which must be decrypted to be understood by the recipient. Commonly found private key algorithms include DES, Blowfish, AES and IDEA.

Public key algorithms utilize two mathematically related keys to separate the processes of encryption and decryption. By utilizing functions that are easy to perform in one direction, but difficult to do in the reverse, the two keys provide for a high level of security if large enough numbers are used. The relationship between the two keys allows for a variety of appealing functionality, including confidentiality and authentication. Commonly used public key algorithms include RSA, El Gamal and Diffie-Hellman.

Hash algorithms are a set of one-way functions that take a variable length input and, after mathematical processing, produce a fixed length output. The transformations of the data produce a fingerprint of the input, and small changes to the input appear as large scale changes in the output. Popular hash algorithms include SHA-1, MD5 and RIPEMD.

These three components are used by the TLS cryptosystems to provide for a variety of services: integrity checking, authentication and confidentiality. During the establishment of a TLS session, public key cryptography is used to exchange a session key used in a private key algorithm. These public keys can also be used for authentication of the server and, if desired, the client. This is then used to provide session-level encryption and provide confidentiality for the entire session. Lastly, hash algorithms are used for integrity checking to ensure that the data has not been tampered with during transmission. TLS is closely related to SSL, and it is the IETF standards version.

Certificates and Authorities

Having outlined the basics of cryptography, including public key and symmetrical cryptography, let's now take a brief look at what we are talking about when we say "certificates". These are vital to SSL and TLS setup, and it's important to understand their principles.

In a nutshell, a certificate is nothing more than a collection of identifying information about a client or a server. It contains descriptive fields, such as the name of an organization and its location, as well as cryptographic information, such as keys and signatures.

Recall that the private key of an asymmetrical keypair can be used to sign material, that, when decrypted with the public key, establishes the signature. This can be used to offer proof of identity. Public key infrastructures (PKI) use a hierarchy of trust for validation of identities, extended by signing certificates and keys. This is in comparison to the web of trust used in PGP, which has no central authority.

The central authority in a PKI issues what is called a certificate authority, or CA, a definitive certificate containing their information and their public key. This CA can be used to sign other certificates by signing the public key of a requesting body, such as your server, with their private key, as described in the preceding paragraph. Since the CA is recognized by all of the involved parties as authoritative, the trust in identity is therefore transitive. "I trust the CA, and the CA says that is you, so it must be true."

Certificates can also be revoked due to expiration or compromise of their security. To do this, the issuing body issues a certificate revocation list (CRL) identifying the certificate to be invalidated. This also has to be trusted, and strong proof is also provided through trust mechanisms.

Certificates can come in a few formats, though we usually deal with them in PEM format. PEM encoding is nothing more than an ASCII text representation of the binary data in ASN.1 format. The X.509 standard defines the distinguished name (DN) format used in these certificates. This information contains several pieces of identifying information to accompany the cryptographic keys:

  • "common name" (CN) being certified

  • "organization" (O) associated

  • "organizational unit" (OU), such as a department within this organization

  • "city or locality" (L) where this organization is located

  • "state or province" (SP) where this city is located

  • "country" (C) is ISO format (such as US)

The DN is therefore the combination of this information. The PEM encoded certificate contains this information along with the DN of the issuer, the validity period of the certificate, various administration information, such as a serial number of the certificate, and any other needed information, such as Netscape specific tags.

These certificates are then used to establish the identity and trustworthiness of the presenter, such as a server or a client. This is used to authenticate the connecting party and take appropriate action, such as allowing a connection to proceed, allowing for mail relaying or entrance into a network.

A Brief Introduction to SMTP and ESMTP

SMTP is the basic protocol spoken by Internet e-mail servers. ESMTP is a set of extensions to the protocol to allow for enhanced features, and it is a superset of the SMTP protocol. In their basic form, however, they are fairly similar.

At the opening of a session, the greeting determines the session type, SMTP or ESMTP. For an SMTP session, the greeting is HELO (/hostname/), where /(hostname)/ is the name of the client machine connecting. To chose an ESMTP session, enter EHLO (/hostname/), and you are presented with a set of options:

<<< 220 ESMTP Sendmail ready for love at Tue, 15 May 2001
16:10:36 -0400
>>> EHLO mail
<<< Hello jose@localhost [], pleased to
meet you
<<< 250-8BITMIME
<<< 250-SIZE 72000000
<<< 250-DSN
<<< 250-ONEX
<<< 250-XUSR
<<< 250 HELP

Unless various ESMTP options are chosen, the MTA then designates the origin of the mail and the recipient (or list of recipients), then begins the data transmission. After completion of the message body delivery, the session is closed. The angled braces indicate the direction of the transmission, either server to client or client to server.

When the MTA supports TLS security, one additional option is presented in the list from the ESMTP notification:

<<< 250-STARTTLS

This is the server's indication that it is capable of handling TLS sessions. The client then sends the same command, STARTTLS, to the server and the session is negotiated. Secure ESMTP transactions occur over the same channel as normal SMTP connections, port 25/TCP, which means you don't have to open new holes in your firewall. You can control if connecting MTA systems must, can or must not use STARTTLS in their connections, even down to the specific mail server level.

Secure ESMTP connections utilize TLS almost immediately. After the greeting, when the server presents STARTTLS as an option, the client initiates the TLS setup by issuing the command STARTTLS.

TLS In Sendmail

We will illustrate how to set up a TLS capable SMTP server using Sendmail 8.12 beta. Since version 8.11, Sendmail has been capable of natively using TLS security when it was compiled. This was, in fact, the major difference between versions 8.10 and 8.11. When compiled to support TLS transactions, it is often referred to as "Sendmail Secure Switch". It is also commercially available in this configuration (see Resources).

We will be using Sendmail 8.12beta17, the current version as of this writing. Note that if you would like to use 8.11.5 (the current release version), you will also need to obtain and install the sfio library, and even then version sfio1999. For either version, 8.11 or 8.12, we will use the OpenSSL library for our cryptographic routines.

Compiling Sendmail to use TLS security is pretty straight forward. First, make sure that you have OpenSSL installed. The current version is 0.9.6a, which has a number of security and performance fixes. We will assume the default installation location, /usr/local/ssl. You should be comfortable with building, configuring and installing Sendmail normally.

The following are additions to the file sendmail-8.12.0.Beta17/devtools/OS/Linux:

define(`confSTDIO_TYPE', `portable')
APPENDDEF(`conf_sendmail_LIBS', `-lssl -lcrypto')

These directives are processed by the Build script to define parameters used in building the Sendmail package. While the Sendmail binary is being built, you should see the -DSTARTTLS directive sent to the compiler.

We also need to edit the configuration file for the build process. The Makefile calls the script sendmail-8.12.0.Beta17/devtools/bin/Build, a master configuration and compilation script. We need to teach it where to look for our cryptographic header files and libraries. We will edit these values, which appear near the top:


At this point you can build the Sendmail package as you would normally. In the top level directory type make. Install it as you would normally.

Configuring the Setup

Sendmail is controlled by a set of configuration files. While they are complex, they are easily set up using the M4 macro scripts, the accompanying .mc files. The following additions to your .mc file (usually found in sendmail-8.12.0.Beta17/cf/cf/ should get you stated using STARTTLS in the configuration file:

define(`CERT_DIR', `MAIL_SETTINGS_DIR`'certs')dnl
define(`confCACERT_PATH', `CERT_DIR')dnl
define(`confCACERT', `CERT_DIR/CAcert.pem')dnl
define(`confSERVER_CERT', `CERT_DIR/MYcert.pem')dnl
define(`confSERVER_KEY', `CERT_DIR/MYkey.pem')dnl
define(`confCLIENT_CERT', `CERT_DIR/cert.pem')dnl
define(`confCLIENT_KEY', `CERT_DIR/MYkey.pem')dnl

We will be using the /etc/mail/certs directory to store our certificates, and the server will use the same certificates both as a client and as a server (when sending and receiving mail, respectively, to another MTA).

Having made these additions, you can run make in the directory that contains the configuration files, sendmail-8.12.0.Beta17/cf/cf, to build the new .cf files. Be sure to install this new configuration file.

Key and Certificate Generation

Generating certificates and keys is readily done with OpenSSL accessory programs. We will modify the script a bit in two places to prevent our certificates from being DES-encrypted. This new script, which we will call, has the following differences (in "patch" format):

# diff -ur /usr/bin/ /usr/bin/
--- /usr/bin/      Fri Sep  1 19:51:57 2000
+++ /usr/bin/     Sat May 26 11:31:18 2001
@@ -57,12 +57,12 @@
            exit 0;
        } elsif (/^-newcert$/) {
            # create a certificate
-           system ("$REQ -new -x509 -keyout newreq.pem -out newreq.pem $DAYS");
+           system ("$REQ -new -x509 -nodes -keyout newreq.pem -out newreq.pem $DAYS");
            print "Certificate (and private key) is in newreq.pem\n"
        } elsif (/^-newreq$/) {
            # create a certificate request
-           system ("$REQ -new -keyout newreq.pem -out newreq.pem $DAYS");
+           system ("$REQ -new -nodes -keyout newreq.pem -out newreq.pem $DAYS");
            print "Request (and private key) is in newreq.pem\n";
        } elsif (/^-newca$/) {

The -nodes option to the certificate generation is the change, otherwise Sendmail will complain it cannot load the (encrypted) key at startup:

May 19 05:42:59 linux sendmail[12018]: starting daemon (8.12.0.Beta17):
May 19 05:42:59 linux sendmail[12018]: STARTTLS=server, error:
SSL_CTX_use_PrivateKey_file(/etc/mail/certs/MYkey.pem) failed

Now we will begin the generation of our own certificate authority (CA), generate a client certificate and use this CA to sign it.

$ -newca

Answer the questions presented to you, such as your organization, your location and your name. Choose a decent passphrase for the CA certificate so no one can abuse your CA and sign any certificate. Copy the CA file, which is in demoCA/cacert.pem, to /etc/mail/certs/CAcert.pem, as we have it configured in the above Sendmail configuration file.

Now we generate the server's private key, which we will want signed by this CA:

$ -newreq

This will generate a private key and certificate request that we want our CA to sign (ourselves really). This is a more sane and flexible method to use when compared to self-signed certificates. We can recycle the CA for other certificates and not have to install multiple CAs, only one for each group. The private key will be in the file newreq.pem. Copy this file to /etc/mail/certs/MYkey.pem, based on the naming scheme we used above in our .mc configuration.

Then we use our CA certificate to sign this request:

$ -sign

This will generate the third component of our server's configuration, the signed public certificate. This will be in the file newcert.pem, which we copy to /etc/mail/certs/MYcert.pem.

Sendmail is, understandably, quite picky about using tight permissions on those files, so remove group and world readability for them:

chmod -R go-rwx /etc/mail/certs

The private keys and the CA file amount to the keys to our cryptographic kingdom, and we don't want just anyone being able to see them.

Testing Your Installation

To test the installation, start up Sendmail in dæmon mode (*sendmail -bd*) and begin a manual ESMTP session to the server:

$ telnet localhost 25
Connected to
Escape character is '^]'.
220 ESMTP Sendmail 8.11.1/8.11.1; Sat, 19 May 2001
08:04:04 -0400
EHLO localhost Hello IDENT:jose@[], pleased to meet you
250 HELP

While you could, at this time, issue the command *STARTTLS*, and the server would respond with a reply like 220 2.0.0 Ready to start TLS, you probably could not negotiate a secure connection by hand.

A simple test to perform is to point an SSL-compatible e-mail client, such as the Netscape Communicator mail client, at this new Sendmail server. Chose to attempt a secure connection if possible and send a mail message through this server. After accepting the new site certificate, a log line like the following should appear in your Sendmail logs:

May 19 23:09:24 linux sendmail[914]: STARTTLS=server,
relay=IDENT:jose@[], version=TLSv1/SSLv3, verify=NO, cipher=RC4-MD5, bits=128/128

This tells you that the server is capable of negotiating a TLS session with a client, indicating a successful installation.

Uses for TLS-Equipped Sendmail

The most obvious use of a cryptographically enabled Sendmail installation is for confidentiality of the electronic mail transaction and the integrity checking provided by the cipher suite. Everything between the two mail servers is encrypted, including the sender and recipient addresses. TLS also allows for authentication of either or both systems in the transaction.

One excellent use of public key cryptography is for strong authentication. We can use this authentication to selectively relay clients, including other mail servers and mobile clients like laptops. However, there have been some problems getting the Outlook mail client working using certificate-based authentication. The Netscape mail client in the communicator package, however, can do successful STARTTLS negotiation. Note that your clients will have to generate certificates and have them signed (for trust validation) by a CA you also trust, if you configure your server to do client certificate checking.

The use of the access map (usually held in /etc/mail/access), which is normally used to determine connections and relaying, can also be extended to give server level control for the use of TLS. Two new entries are available for TLS options: VERIFY, which contains the status of the level of verification (held in the macro {verify}; and ENCR, the strength of the encryption in the macro {cipher_bits}. VERIFY can also accept the argument for {cipher_bits}. Here are a few example entries that illustrate these features, and the role based granularity as well:

Force strong (112 bit) encryption for communications for this server:             ENCR:112

For a TLS client, force string verification depths to at least 80 bits:

TLS_Clt:desktop.example:net     VERIFY:80

Much more complicated access maps are possible, and error conditions (such as permanent or temporary, PERM+ or TEMP+) can be set on the basis of various criteria. This allows you fine-grained control over the types of connections you can allow. Documentation is emerging for these features, and the newsgroup comp.mail.sendmail is also a great place to get questions answered.

Note that it is unwise to force all SMTP clients to use TLS, as it is not yet widespread. The RFC document notes that publicly referenced SMTP servers, such as the MX servers for a domain, must not refuse non-TLS connections. However, restricted access SMTP servers, such as those for a corporate intranet, can use TLS as an access control mechanism.

Limitations of Sendmail with TLS

While all of this work may seem to be a total security solution for e-mail, it cannot totally secure your e-mail system.

One often forgotten limitation of using TLS on your mail servers is the payload of the mail message and the resulting security there. Many virus and worm files are now distributed using electronic mail. While the mail maybe encrypted and the servers authenticated, the payload will still be malicious. Using a good content protection program on the desktop will be of great value even with TLS at the MTA level.

Because Sendmail with TLS only can authenticate at the server level, true end-to-end authentication of the mail message cannot be performed with only the use of Sendmail Secure Switch. By including the use of S/MIME or PGP e-mail and trustworthy key hierarchies, full confidentiality and integrity can be accomplished from end-to-end of the mail message path.

Furthermore, if a mail message traverses more than the starting and ending servers, there is no way to control interactions between the additional mail servers, which may use non-secure connections. This introduces a point of vulnerability in the chain.

Additionally, SMTP over TLS is not yet widely implemented. The standard, in fact, doesn't force it, leaving it only as an option, though specific sites can configure their servers to force it for specific clients. As such, it is difficult to see the widespread use of SMTP using TLS, despite the fact that the standard is over two years old.

Lastly, interopability problems can creep in. On UNIX systems, the major mail dæmons, including Sendmail, QMail and Postfix, all work quite well with TLS enabled, but various commercial MTAs seem to have problems integrating with Sendmail. Exchange had some problems which Microsoft was working to patch; which may be available by now, as versions have changed since this was first noted. As such, some testing should be performed if you are going to use TLS in a heterogeneous environment.


RFC 2487, "SMTP Service Extension for Secure SMTP over TLS"

RFC 1869, "SMTP Service Extensions"

RFC 2246, "The TLS Protocol Version 1.0"

See IETF for RFC documents.

Claus' STARTTLS docs


Sendmail, source and documentation

Sendmail Secure Switch, commercial version

SSL and TLS Essentials by Stephen Thomas

Public Key Crypto Basics

Jose Nazario is a biochemistry graduate student nearing the completion of his PhD. Side projects include Linux and other UNIX variants, software and security related matters, and hobbies outside of the office include fly fishing and photography.

Load Disqus comments