Whatever else you do to secure a Linux system, it must have comprehensive, accurate and carefully watched logs. Logs serve several purposes. First, they help us troubleshoot virtually all kinds of system and application problems. Second, they provide valuable early warning signs of system abuse. And third, when all else fails (whether that means a system crash or a system compromise), logs provide us with crucial forensic data.
This article is about making sure your system processes and critical applications log the events and states you're interested in. The tried-and-true tool for achieving this is syslog. syslog accepts log data from the kernel (by way of klogd), from any and all local processes, and even from processes on remote systems. It's flexible as well, allowing you to determine what gets logged and where it gets logged. A preconfigured syslog installation is part of the base operating system in virtually all variants of UNIX and Linux.
This month, therefore, we discuss syslog configuration and use it in-depth, probably in much greater detail than you've previously considered. In my experience the vast majority of Linux users, and even administrators, tend to leave their syslog installations with default settings, tweaking them little if at all. This is seldom a good idea.
I should also mention that if you're really interested in granular, flexible logging, Balazs Scheidler's excellent syslog-ng (syslog, new generation) is well worth checking out. But it's still nowhere near as ubiquitous as syslog, so I won't do more than mention it this time. See the Resources section for more information on syslog-ng.
Whenever syslogd, the syslog dæmon, receives a log message, it acts based on the message's type (or facility) and its priority. syslog's mapping of actions to facilities and priorities is specified in /etc/syslog.conf. Each line in this file specifies one or more facility/priority selectors followed by an action. A selector consists of a facility or facilities and a (single) priority.
In the following syslog.conf line, mail.notice is the selector and /var/log/mail is the action (i.e., “write messages to /var/log/mail”):
Within the selector, “mail” is the facility (message category) and “notice” is the level of priority.
Facilities are simply categories. Supported facilities in Linux are auth, authpriv, cron, dæmon, kern, lpr, mail, mark, news, syslog, user, UUCP and local0 through local7. Some of these are self-explanatory, but of special note are:
auth: used for many security events.
authpriv: used for access-control-related messages.
dæmon: used by system processes and other dæmons.
kern: used for kernel messages.
mark: messages generated by syslogd itself that contain only a timestamp and the string “--MARK--”. To specify how many minutes should transpire between marks, invoke syslogd with the -m [minutes] flag.
user: the default facility when none is specified by an application or in a selector.
local7: boot messages.
*: wildcard signifying “any facility”.
none: wildcard signifying “no facility”.
Unlike facilities, which have no relationship to each other, priorities are hierarchical. Possible priorities in Linux are (in increasing order of urgency): debug, info, notice, warning, err, crit, alert and emerg. Note that the urgency of a given message is determined by the programmer who wrote it; facility and priority are set by the programs that generate messages, not by syslog.
As with facilities, the wildcards “*” and “none” also may be used. Only one priority or wildcard may be specified per selector. A priority may be preceded by either or both of the modifiers “=” and “!”.
If you specify a single priority in a selector (without modifiers), you're actually specifying that priority plus all higher priorities. Thus the selector mail.notice translates to “all mail-related messages having a priority of notice or higher”, i.e., having a priority of notice, warning, err, crit, alert or emerg.
This behavior can be canceled by prepending an = to the priority. The selector mail.=notice translates to “all mail-related messages having a priority of notice”. Priorities may also be negated: mail.!notice is equivalent to “all mail messages except those with priority of notice or higher”, and mail.!=notice corresponds to “all mail messages except those with the priority notice”.
Free DevOps eBooks, Videos, and more!
Regardless of where you are in your DevOps process, Linux Journal can help!
We offer here the DEFINITIVE DevOps for Dummies, a mobile Application Development Primer, and advice & help from the expert sources like:
- Linux Journal
- Integrating Trac, Jenkins and Cobbler—Customizing Linux Operating Systems for Organizational Needs
- New Products
- Tech Tip: Really Simple HTTP Server with Python
- Returning Values from Bash Functions
- Non-Linux FOSS: Remember Burning ISOs?
- EdgeRouter Lite
- RSS Feeds
- Cooking with Linux - Serious Cool, Sysadmin Style!