Anatomy of Postfix

Developed with security and speed in mind, Postfix has become a popular alternative to Sendmail. The Book of Postfix published by No Starch Press is a complete guide to Postfix whether used by the home user, as a mailrelay or virus scanning gateway, or as a company mailserver. Practical examples show how to deal with daily challenges like protecting mail users from SPAM and viruses, managing multi
Maps

Maps are files and databases that Postfix uses to look up information. Maps have many different purposes, but they all have one thing in common--a left-hand side (LHS, or key) and a right-hand side (RHS, or value).

Here are a few examples of keys and values:

KeyValue
postmaster:john
postmaster@example.comjohn
192.168.254.12REJECT
spammer@example.comREJECT
/^Subject: your account {25}[a-z]{8}/REJECT Mimail Virus Detected

To use a map, you specify a key and get the associated value as a result.

Note: The keys and values here come from various files and would not make sense in one file. The preceding list is just an illustration to show that all map entries take the same basic form.

Map Types

Postfix can use many different kinds of maps. The formats available depend on the way Postfix was compiled on your particular system. To find out what formats your Postfix supports, run postconf -m on the command line. You should get a list of map types:

# postconf -m
btree
cdb
cidr
environ
hash
ldap
mysql
nis
pcre
proxy
regexp
sdbm
static
tcp
unix

Indexed Maps (hash, btree, dbm, and So On)

Indexed maps are binary databases built from regular text files with commands such as newaliases, postalias, and postmap. The binary maps have an indexed format so that Postfix can quickly retrieve the value associated with a key. As a further performance improvement, the Postfix daemons open these maps when starting up, and they do not re-read them unless they notice a change in the map files in the filesystem. To reload a map, a daemon exits and a new one is started by the master daemon.

Note:

If you have indexed maps that change frequently, the daemons using these maps will restart just as often. Under a heavy load, this can lead to performance problems.

The most common indexed maps are built from the aliases, virtual, transport, relocated, and sasl_passwd text files. You can identify a map file because its name is the original file with a suffix that also tells you the index format. For example, an aliases map file built with the postalias command is named aliases.db.

Note:

When you create a file in order to build an indexed map from it you don't have to put keys in a specific order. The conversion tools and programs that use indexed maps do not require a specific order for input. In fact, the process of conversion removes the ordering.

Postfix queries entries in a predefined order specified in the manual pages for the tables (access(5), transport(5), virtual(5), aliases(5) and canonical(5)). In other words, each map lookup actually consists of a series of single queries (derived from the original query) on single keys in the indexed map.

Linear Maps (PCRE, regexp, CIDR, and Flat Files)

Linear maps are regular text files. Postfix reads these files from top to bottom, making them different from indexed maps. This difference is quite important, because the first match in the file determines the action that Postfix will take. Postfix ignores any later entries, whether they match the query or not.

Consider the following regexp map, where a john.doe@example.com lookup returns OK, because the first line matches.

/john\.doe@example\.com/ OK
/example\.com/ REJECT

However, if you swap the lines in the regexp map, the other entry matches first, so the same john.doe@example.com lookup returns REJECT:

/example\.com/ REJECT
/john\.doe@example\.com/ OK

You do not need to convert linear maps to a binary form (in fact, you can't do it). The Postfix daemons read them at startup and do not notice any changes to the map until they are restarted. Typical Postfix linear maps include header_checks, body_checks, and mime_header_checks (see Chapter 9).

Caution:

As your linear maps grow, it takes longer for the Postfix daemons to process them. This is especially true with respect to body or header checks, because the cleanup daemon needs to check every line of the body (up to body_checks_size_limit) and headers against every line of the map.

This can cause a significant slowdown, especially if you have extensive *_checks parameters that use regexp or PCRE (Perl-compatible regular expression) type maps in order to prevent spam from entering the system. When this happens, it's usually time to hand complex spam filtering to an external application.

To make the Postfix daemons notice changes in linear maps, run postfix reload. If the timing is not critical, you can set the max_use parameter to define a time-to-live for daemons. As soon as a daemon has processed the number of tasks specified in that parameter, it quits and is restarted by master. Upon restart, it re-reads all required maps.

Databases (MySQL, PostgreSQL, LDAP)

Postfix treats a database just like an indexed map. The result of a database query is Match (along with the value returned by the query) or No match. The principal difference between a database map and an indexed map is that you do not need to restart a daemon when there is a change to the database. Postfix does not assume that the postmaster is the only person who can alter the database.

The drawback to this approach is that the database may not be able to handle the number of queries gracefully, because Postfix needs to perform at least three queries for each lookup in a map (see the "See How Postfix Queries Maps" section that follows). Under heavy load, the database backend could stop working, and your mail service would be vulnerable to a self-induced meltdown or a denial-of-service attack. This possibility should not prevent you from using database backends, but you should be aware of the risk.

Database lookups can become a problem for systems with a heavy load, but this isn't the only issue to consider--latency can be another problem. Database queries have a higher latency than indexed maps because Postfix must connect to the database backend, send the query, and then wait for the result. With an indexed map, Postfix has only to consult data that is already loaded in memory.

If your database becomes a bottleneck, and you do not have an excessively large map, you can insert a map between the database and Postfix. That is, you can create an indexed map from a complete database query, and then run Postfix with that map. You need to remember to update the map as often as necessary, but the proxymap daemon can be used to significantly reduce the number of concurrent connections.

Determining the Number of Simultaneous Connections to a Database

Postfix daemons (smtpd, smtp, and so on) run with a process limit (set by default_process_limit) of 100 simultaneous processes. Running at peak load, there would be 100 concurrent smtpd daemons, each querying the database backend for one access(5) lookup (e.g., because we use a map for checking if the client is in our personal blacklist and should then be denied from sending mail to us).

Remember that one lookup results in at least three queries, so the number of simultaneous queries to the database would be at least default_process_limit * 3 (which, in the default configuration, would be 300 queries), while the number of simultaneous connections is default_process_limit. This is only the number of queries and connections for smtpd daemons; other daemons, such as local and qmgr, may be working on other jobs, adding to the number of open connections and simultaneous queries.

How Postfix Queries Maps

Maps can be used for various tasks. Postfix has table-driven mechanisms that use maps (see access(5), aliases(5), canonical(5), and transport(5)). These maps can use different lookup mechanisms (LDAP, NIS, SQL, btree, hash, regexp, cdb, cidr, pcre, and so on). Note that the lookup order described below only applies to access(5) type maps.

  1. <localpart@domainpart> Matches the specified mail address verbatim.

  2. <domainpart> Matches domainpart as the domain part of an email address. The pattern domainpart also matches subdomains, but only when the string smtpd_access_maps is listed in the Postfix parent_domain_matches_subdomains configuration setting. Otherwise, specify .domainpart (note the initial dot) to match subdomains.

  3. <localpart@> Matches all mail addresses with the specified user part (localpart), no matter what domain they belong to.

  4. Fail If the lookups don't match, Postfix will return no match found, and the query ends with an error.

Note:

It isn't possible to look up a null sender address in some lookup table types. By default, Postfix uses <> as the lookup key for the null sender address. The value is specified with the smtpd_null_access_lookup_key parameter in the main.cf file.

This order of lookups implies that Postfix performs several lookups for each query, which isn't really a problem unless you're using high-latency maps like SQL or LDAP maps (and, of course, you should expect that a lot of lookups will need multiple queries). This is just one thing to remember before you put all your maps into LDAP and then complain on the postfix-users mailing list that "Postfix is slow. . . ."

______________________

Comments

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

Chill

Anonymous's picture

the f**k out people. you're not learning anything by having a go at each other.

Postfix is easier to install

V. Prague's picture

You say that Postfix is a useful alternative to Sendmail, but it is actually better.
Just a week ago, I was unable to set up Sendmail properly on the 64 bit Mandrake - it just refused to send mail. I asked a very experienced friend to do this, only to hear that Sentmail apparently does not respect some DNS related settings. He installed Postfix and it works perfectly. I do not wish to hear about Sendmail again.

This is THE Postfix book to have

Scott Kitterman's picture

I found the Book of Postfix invaluable to me when I was getting started with Postfix. If you want to learn Postfix, this book is the place to start.

This is THE Postfix book to have - hardly

Anonymous's picture

Postfix specialists might disagree with you. I certainly do.

2nd edition?

Chris's picture

The book has an extraordinary amount of errata -- so much that I have to keep the errata page open whenever I read the book. Are there plans for a second (proof-read) edition?

A second edition is in the

Anonymous's picture

A second edition is in the works. Unfortunately we need to backport the text from RTF to our native XML format first :(

Why bother?

Anonymous's picture

Basically, you're promoting your book, much of which I have read. The article itself is simply a brain dump for you to jack up your ego and say, "look what I know". Only a postfix administrator or developer would understand this article, so it teaches little - like your book.

Readers might need a warning: No Starch Press did a poor job of editing the book (if they did any editing at all): No copy editing, no technical editing, etc. and most of it reads like a German with little understanding of native English - "now, I will tell you how you must do this....First you must ...then you must, etc."

Additionally, the main Chapter about building a company server is broken. You won't build a functional server following their instructions.

So, consider this with caution.

Considering this is the only

Anonymous's picture

Considering this is the only article online I've found that explains clearly the process that postfix uses to process mail, and how all the pieces fit together, I hardly see it as just an ego braindump.

The entire postfix documentation is written so that only a postfix administrator or developer would understand it.

Considering that every organisation is going to do mail a little differently, standard how-to guides very rarely can be followed completely. This is why it's so important to actually understand the system. If I can understand the system, each configuration value is just a manpage away.

Why bother....

Keith Daniels's picture

Uh.... I assume you have written something better? If so I would like to hear about it...

As far as the promotion goes, the authors had nothing to do with it, if that is who you are accusing of promoting it.

RE: "Only a postfix administrator or developer would understand this book"...... duh, who else but someone who was, or wanted to be, an administrator or developer WOULD read it?

If formal American style English is your prime criteria for the excellence of a book and not having that makes it not worthy of reading -- then your basic orientation must be academic liberal arts instead of technical. Either that or you have a personal issue with the authors or their nationality and are just trying to put them, the book and their nationality down because of prejudice or personal dislike.

RE: the Chapter about building a company server -- did you check their web site for corrections they might have posted for that page? Did you actually try to build that server setup or is that just your opinion from reading it?

All the new OSs and windowing systems are oriented towards content consumption instead of content production.

--Steve Daniels 2013

Yea, yea, yea.

Anonymous's picture

Yea, yea, yea.

If only an experienced postfix admin could read it, why bother?

In other words, if you're interested in learning postfix, how about a book that guides someone with less knowledge to become more able. That seems to be the purpose of a book like this.

As far as American English - how about just English instead of the horrible language used. Try this from page 313: "Once we got this going, we will make the system more complex." then, "You should have profound understand of LDAP schema and OpenLDAP before you start to implement the company mail server we describe in this chapter".

This book is crammed full of this kind of grand work.

Take your sophomoric diatribe to Slashdot, freak.

yea, yea, yea

Keith Daniels's picture

What's the matter? Your little company having trouble using Postfix with their "Controled Email" and you are blaming it on some poor non native English speakers who wrote a book?

All the new OSs and windowing systems are oriented towards content consumption instead of content production.

--Steve Daniels 2013

yea, yea, yea

Anonymous's picture

DANG! I read two other books before this, both of those helped very little and go me nowhere. I had no previous experience other than HOWTO's I found on line, which weren't many. After I read this book, I can honestly say that I had a successful Postfix email server running and operating SPAM FREE.

These cats don't know what they are talking about! It's a good book and Postfix is definitely easier and faster to learn than bloated sendmail.

Webinar
One Click, Universal Protection: Implementing Centralized Security Policies on Linux Systems

As Linux continues to play an ever increasing role in corporate data centers and institutions, ensuring the integrity and protection of these systems must be a priority. With 60% of the world's websites and an increasing share of organization's mission-critical workloads running on Linux, failing to stop malware and other advanced threats on Linux can increasingly impact an organization's reputation and bottom line.

Learn More

Sponsored by Bit9

Webinar
Linux Backup and Recovery Webinar

Most companies incorporate backup procedures for critical data, which can be restored quickly if a loss occurs. However, fewer companies are prepared for catastrophic system failures, in which they lose all data, the entire operating system, applications, settings, patches and more, reducing their system(s) to “bare metal.” After all, before data can be restored to a system, there must be a system to restore it to.

In this one hour webinar, learn how to enhance your existing backup strategies for better disaster recovery preparedness using Storix System Backup Administrator (SBAdmin), a highly flexible bare-metal recovery solution for UNIX and Linux systems.

Learn More

Sponsored by Storix