Anatomy of Postfix

Maps are files and databases that Postfix uses to look up information. Maps have many different purposes, but they all have one thing in common--a left-hand side (LHS, or key) and a right-hand side (RHS, or value).

To use a map, you specify a key and get the associated value as a result.

Note: The keys and values here come from various files and would not make sense in one file. The preceding list is just an illustration to show that all map entries take the same basic form.

Map Types

Postfix can use many different kinds of maps. The formats available depend on the way Postfix was compiled on your particular system. To find out what formats your Postfix supports, run postconf -m on the command line. You should get a list of map types:

# postconf -m
btree
cdb
cidr
environ
hash
ldap
mysql
nis
pcre
proxy
regexp
sdbm
static
tcp
unix

Indexed Maps (hash, btree, dbm, and So On)

Indexed maps are binary databases built from regular text files with commands such as newaliases, postalias, and postmap. The binary maps have an indexed format so that Postfix can quickly retrieve the value associated with a key. As a further performance improvement, the Postfix daemons open these maps when starting up, and they do not re-read them unless they notice a change in the map files in the filesystem. To reload a map, a daemon exits and a new one is started by the master daemon.

The most common indexed maps are built from the aliases, virtual, transport, relocated, and sasl_passwd text files. You can identify a map file because its name is the original file with a suffix that also tells you the index format. For example, an aliases map file built with the postalias command is named aliases.db.

Postfix queries entries in a predefined order specified in the manual pages for the tables (access(5), transport(5), virtual(5), aliases(5) and canonical(5)). In other words, each map lookup actually consists of a series of single queries (derived from the original query) on single keys in the indexed map.

Linear Maps (PCRE, regexp, CIDR, and Flat Files)

Linear maps are regular text files. Postfix reads these files from top to bottom, making them different from indexed maps. This difference is quite important, because the first match in the file determines the action that Postfix will take. Postfix ignores any later entries, whether they match the query or not.

Consider the following regexp map, where a john.doe@example.com lookup returns OK, because the first line matches.

/john\.doe@example\.com/ OK
/example\.com/ REJECT

However, if you swap the lines in the regexp map, the other entry matches first, so the same john.doe@example.com lookup returns REJECT:

/example\.com/ REJECT
/john\.doe@example\.com/ OK

You do not need to convert linear maps to a binary form (in fact, you can't do it). The Postfix daemons read them at startup and do not notice any changes to the map until they are restarted. Typical Postfix linear maps include header_checks, body_checks, and mime_header_checks (see Chapter 9).

This can cause a significant slowdown, especially if you have extensive *_checks parameters that use regexp or PCRE (Perl-compatible regular expression) type maps in order to prevent spam from entering the system. When this happens, it's usually time to hand complex spam filtering to an external application.

To make the Postfix daemons notice changes in linear maps, run postfix reload. If the timing is not critical, you can set the max_use parameter to define a time-to-live for daemons. As soon as a daemon has processed the number of tasks specified in that parameter, it quits and is restarted by master. Upon restart, it re-reads all required maps.

Databases (MySQL, PostgreSQL, LDAP)

Postfix treats a database just like an indexed map. The result of a database query is Match (along with the value returned by the query) or No match. The principal difference between a database map and an indexed map is that you do not need to restart a daemon when there is a change to the database. Postfix does not assume that the postmaster is the only person who can alter the database.

The drawback to this approach is that the database may not be able to handle the number of queries gracefully, because Postfix needs to perform at least three queries for each lookup in a map (see the "See How Postfix Queries Maps" section that follows). Under heavy load, the database backend could stop working, and your mail service would be vulnerable to a self-induced meltdown or a denial-of-service attack. This possibility should not prevent you from using database backends, but you should be aware of the risk.

Database lookups can become a problem for systems with a heavy load, but this isn't the only issue to consider--latency can be another problem. Database queries have a higher latency than indexed maps because Postfix must connect to the database backend, send the query, and then wait for the result. With an indexed map, Postfix has only to consult data that is already loaded in memory.

If your database becomes a bottleneck, and you do not have an excessively large map, you can insert a map between the database and Postfix. That is, you can create an indexed map from a complete database query, and then run Postfix with that map. You need to remember to update the map as often as necessary, but the proxymap daemon can be used to significantly reduce the number of concurrent connections.

Determining the Number of Simultaneous Connections to a Database

Postfix daemons (smtpd, smtp, and so on) run with a process limit (set by default_process_limit) of 100 simultaneous processes. Running at peak load, there would be 100 concurrent smtpd daemons, each querying the database backend for one access(5) lookup (e.g., because we use a map for checking if the client is in our personal blacklist and should then be denied from sending mail to us).

Remember that one lookup results in at least three queries, so the number of simultaneous queries to the database would be at least default_process_limit * 3 (which, in the default configuration, would be 300 queries), while the number of simultaneous connections is default_process_limit. This is only the number of queries and connections for smtpd daemons; other daemons, such as local and qmgr, may be working on other jobs, adding to the number of open connections and simultaneous queries.

How Postfix Queries Maps

Maps can be used for various tasks. Postfix has table-driven mechanisms that use maps (see access(5), aliases(5), canonical(5), and transport(5)). These maps can use different lookup mechanisms (LDAP, NIS, SQL, btree, hash, regexp, cdb, cidr, pcre, and so on). Note that the lookup order described below only applies to access(5) type maps.

This order of lookups implies that Postfix performs several lookups for each query, which isn't really a problem unless you're using high-latency maps like SQL or LDAP maps (and, of course, you should expect that a lot of lookups will need multiple queries). This is just one thing to remember before you put all your maps into LDAP and then complain on the postfix-users mailing list that "Postfix is slow. . . ."