The System Logging Dæmons, syslogd and klog

Most UNIX-like systems since the early days of BSD (and Linux certainly falls in this category) have provided an API for application programs to send log messages to the system, where they can be centrally handled at the discretion of the system operator. Prior to the creation of this facility, each application program would handle log messages in its own way. Some would write to STDERR, some would write to a file, some would write to a pipe, and some would offer all these options or more.

As the number and complexity of applications on a system grows, so, too, does the complexity of the system administrator's job. Applications and their messages vary widely in their significance to certain audiences. If a number of applications are considered “critical” and their status is the system administrator's responsibility, he does not want to search to find out where and how every critical application logs its status. That's where syslogd comes in.

BSD added an API for logging to the standard library. Linux also offers it. This API consists of three function calls:

#include <syslog.h>
void openlog( char *ident, int option, int facility)
void syslog ( int priority, char *format, ...)
void closelog( void )

An application that wishes to use syslogd for logging uses these calls. A brief introduction to the use of this API can be found in the sidebar “Using the syslog API in Applications”. The one critical thing to know is that all messages from this API have, at a minimum, a “facility” and a “priority”.

Using the syslog API in Applications

Facilities include LOG_AUTHPRIV, LOG_CRON, LOG_KERN, LOG_DAEMON and so forth. These serve to identify the “system” of origin. Note that it is not the “program” of origin. For example, many different programs make up UUCP, but they all log as LOG_UUCP. The program name can be a component of a log message, but this has nothing to do with the facility. Some programs will log as more than one facility. For example, telnetd might log failed logins as LOG_AUTHPRIV, but it might log other messages as LOG_DAEMON. “Priorities” specify the “severity level” or “level of attention” the message merits. We'll discuss these concepts at greater length throughout this piece.

Our primary focus here is the tool that handles messages sent via this API. While syslogd was originally developed for 4.2BSD, we are going to cover the version that ships with most Linux distributions today, specifically syslogd version 1.3-3. The syslogd utility normally reads a configuration file at startup to determine how messages are to be handled. This file, normally /etc/syslog.conf, tells syslogd what to do with messages. Much of the rest of this document will describe how to use syslog.conf to customize logging on your system.

Listing 1

The syslog.conf file follows the more or less ubiquitous UNIX convention of using the pound sign (#) as a comment character. We'll use the sample syslog.conf file in Listing 1 for the rest of our discussion. This is the “out-of-the-box” syslog.conf from my Red Hat 6.1 laptop. I use other distributions personally, primarily Debian and SuSE, but Red Hat seems to be the most popular. Let's understand what this file is doing.

“Rules” in syslog.conf are a single line which consists of two parts. The first is a “selector”, which specifies the set of messages on which the rule is to act. The second is an action, which specifies what is to be done with messages that match the selector.

The selector is further divided into a “facility” and a “priority”. Yes, these match the terms mentioned above in the brief description of the syslog API. The facilities and levels have numeric values and you can use them in the syslog.conf file, but it is strongly advised that you do not. Symbolic values are supported, and if the syslog API is ever changed, the numeric values might change, whereas one would expect that the symbolic names would be kept in alignment with any such change. So, be safe and use the symbolic names.

The symbolic facility names are auth, authpriv, cron, daemon, kern, lpr, mail, mark, news, security (same as auth), syslog, user, uucp and local0 through local7. The security keyword is deprecated in favor of the auth keyword. The mark keyword is internal and should not be used by applications; the syslogd program can be set to produce a mark periodically, which provides a means to tell if you aren't getting messages, because none are being generated or because syslogd has died. The rest of them correspond to the major subsystems on your Linux box.

The priority keywords are debug, info, notice, warning, warn (same as warning), err, error (same as err), crit, alert, emerg and panic (same as emerg). The keywords error, warn and panic are deprecated and should no longer be used.

A selector consists of a facility and a priority separated by a period (.) character. Thus, mail.crit would select all critical messages from the mail facility.

The default behavior of the BSD syslog system is for all messages of the specified or higher priority to be handled by the action. The Linux syslogd does the same by default. It does have a number of extensions, however.

You may use the asterisk (*) character to indicate all facilities or all priorities (depending on whether it appears before or after the period). Thus, the authpriv.* line in the example sends all messages from the authentication facility, no matter what priority, to /var/log/secure. You may use the special priority none to indicate that no messages from a given facility rule are to be acted upon by the action.

You may specify multiple facilities with the same priority in a single rule by listing the facilities separated by commas (,) before the period. Thus, the uucp,news.crit line sends all critical and above priority messages from the mail and news facilities to /var/log/spooler.

You may specify multiple selectors for a single action by listing them separated by the semicolon character. Each subsequent selector may override the previous. Thus, the *.info;mail.none;news.none;authpriv.none rule would send all messages above info priority from all facilities (because of the *) to /var/log/messages, except messages from mail, news or authentication facilities (because of the none keyword and because rules are applied in order, left to right).

More than one rule may apply to a message! It is important to understand that a message will be sent to all actions with matching selectors. It is not as if a message, once matched, is gobbled up. That means you can store a single message to multiple actions if it is matched by multiple selectors.

There are more priority selection extensions. First, remember that the default is to select messages of the stated or higher priority. You may also reverse the sense with the exclamation mark. So, for example, a rule such as

*.!err   /var/log/routine

would send all messages not at err or above to /var/log/routine (a file meant for “routine” messages, apparently).

You can also restrict the selection to an exact priority instead of to a given priority and higher with the equals sign (=). Thus, the news.=crit rule out of our example would send only critical messages from the news facility to /var/log/news.crit.

Table 1

At this point, you may be a little foggy on what, precisely, these various priorities are meant to denote. Let's shed a little light on that issue by looking at the “Priorities” table.

A classic problem in designing software is trying to figure out an empirical way to tell the difference between a condition you would report as crit vs. alert. In fact, it is sometimes even more difficult to decide when you should use notice vs. warn. You won't find total agreement between packages on what level of message falls where. One difficulty lies in trying to decide who will read the log. An emerg to a business unit might be only a warn to a network administrator.

There's no one good answer to this problem. At least, by creating a uniform method for handling program messages, we avoid a proliferation of different reporting systems, and some conventions have emerged with time. Because we are able to match a message to more than one action, we can output messages to targeted audiences. For example, we could report all auth messages to the security department's home directory, but the system administrator might choose to receive only those of “crit” or above. The syslogd lets us do this.