How to Log Friends and Influence People

 in
This month, Mark discusses how to keep track of what is happening to (and on) your Linux system.

The syslog daemon (known as syslogd) is used to receive messages from various parts of the Linux (or other Unix) system and distribute these messages to other locations. This allows you, the administrator, to review system activities by keeping a log of what goes on behind the scenes. It's a great way of tracking down problems and also of reviewing your system security.

For example, you can keep track of serious errors, such as TCP/IP problems or lack of memory (for when ghostscript and X get a bit out of hand). At the same time, you can debug that pesky mail problem that a user is having, or you can keep track of who has used the su program to assume root status, and you can also find out who failed trying to do so.

To get a copy of the syslog program, see the sidebar on obtaining syslog from an ftp site near you.

Syslogd has a setup file, called /etc/syslog.conf, which tells syslogd what messages to log, and what to do with the messages it receives. By default (i.e., the /etc/syslog.conf file is empty), messages are not stored anywhere. Most distributions of Linux have some basic setup of the syslog.conf file included, and the syslogd program is often started in the /etc/rc.d/rc.inet2 file.

The setup of your syslog.conf file is in three parts. Let's look at a sample syslog.conf file:

# /etc/syslog.conf
 *.info;*.notice    /usr/adm/messages
 *.debug           /usr/adm/debug
 *.warn            /usr/adm/syslog

The first line is a comment. Any lines starting with # or completely blank lines are considered comments and are ignored by syslogd. You can see two columns here, separated by at least one tab. The left side (the one with *.info;*.notice in it) defines the facility and the logging level. The right side defines the destination of the messages.

The facility can be defined as the “kind” of program that is running. For example, “mail” is the facility used whether you are running sendmail or smail. There are nine pre-defined facilities, plus eight facilities that are locally defined, a “wildcard” facility that means “all facilities” and one facility for issuing timestamps.

Here is what each facility is for:

Facility

Used For

user

user processes

kern

kernel messages

mail

mail

daemon

various daemons (such as ftpd)

auth

authorization (such as login)

lpr

the line printer

news

USENET news (nntp)

uucp

UUCP (Unix to Unix copy)

cron

the cron daemon

mark

a periodic message created for placing timestamps in log files

local0-local7

locally-defined facilities

*

all facilities (except mark)

One special note about the mark facility. This facility is created by syslogd every twenty minutes at the info level. This will allow you to quickly see time change in a log file, along with making sure that syslogd is running (and logging). And as noted above, the special * facility (which means all the available facilities) does not include mark.

Each facility can have a logging (or severity) level to it. This is used to indicate which types of messages you want to record. The high severity levels (such as emerg) require the most attention, as an emerg usually indicates that the program will fail soon. These severity levels decrease in importance all the way down to the debug level, which is used for properly setting up your software. Once your software is configured correctly, you can change the severity level to a higher one. Check the documentation on your software for its interpretation of the various levels. From highest severity to lowest:emergalertcriterrwarningnoticeinfodebugnone

The severity of none means that no messages from that facility are to be recorded. These levels mean that messages of that level and above are recorded. For example, say you have the following two lines:

mail.debug      /var/adm/syslog.mail
mail.emerg      /var/adm/syslog.mail.emerg

When sendmail (or smail), or whatever program is logging as the mail facility) sends syslog a message with a level of debug, it gets placed in the syslog.mail file. Any other messages, of any level from info up to emerg, also get placed in syslog.mail. Emergency messages from mail get sent to both syslog.mail and to syslog.mail.emerg. This setup makes it extremely easy to check for emergency conditions for your software, since a glance at the directory listing for syslog.mail.emerg will tell you if the file has changed recently, and you can easily type tail/var/adm/syslog.mail.emerg to glance at the ten most recent entries. In addition, you can find the emergency message in syslog.mail and see the other messages surrounding it to determine the events leading up to the emergency message.

In another example, we can see the use of the “none” severity level:

*.alert;user.none       /var/adm/syslog.alert
user.alert              /var/adm/syslog.user.alert

Here, all messages at alert severity and above are sent to syslog.alert, except for the ones in the user facility. Those get sent to the syslog.user.alert file instead, as specified by the second line.

Now you are probably wondering about the right hand side of the lines. By now, you have figured out that this can be the name of a file. But it can also be the name of a user or of a remote host. If the intended recipient of the message is a file, the name of the file must start with a /, indicating that you have to give a full path for the filename. Note that this can be just about any file, including /dev/console, which will print the message to the console of the machine, or /dev/lp2 to print your message on paper.

If the recipient is a remote host, it will start with an @ followed by the name of the host. The intended host will receive the message via TCP/IP port 514, and from this point, it is just like a message to syslog locally. A message to the remote host must go through syslog and get logged to the appropriate location. This will allow you to monitor a network of machines from one location. You can also include a comma-separated list of users and the messages will be printed on those users' screens if they are logged in. You can also include a *, which means that every user that is logged in gets the message. This is very useful for extreme emergency conditions where you want the users to realize that something is very wrong and they should log out.

#Log mail errors to the mail host for
#the postmaster to deal with
mail.*         @mailhost
#Send kernel emergency warnings to all
#users so they know what's up
kern.emerg     *

Once you have your /etc/syslog.conf file set up, you have to first set up the files that will be storing the messages, then restart syslogd.

Setting your files up for syslog merely means reviewing the permissions of the log files. By default, files are set up owned by root, in the root group, readable by everyone, and writable only by the root user. For a one-user system, this should be fine. But for a multi-user system, with (possibly) logs of who sent and received e-mail, many of these files should only be read by the root user. A good suggestion for file setup is to make it read-write by root and read-only by the wheel group. This allows you to set up some security (by controlling who is in the wheel group) without having to `su' to root every time you want to check some of the files. Re-starting syslogd does not require killing the syslogd program. All that is needed is to send a SIGHUP (signal number 1) to the process, and it will re-read its configuration files. To help you out even more, the PID of syslogd is stored in the /etc/syslog.pid file. As root:

kill -HUP `cat /etc/syslog.pid`

will re-start the syslog daemon and put your changes into effect. Some Linux packages also include a script called /etc/syslogd.restart, which is merely a script that runs the above kill program. If for some reason you want to actually kill the syslogd program, a TERM signal (or 15) will kill the program (kill -TERM `cat /etc/syslog.pid`)

If you have trouble with either of these, consider using the killall program. This is a linux-specific program, but is quite useful:

killall -HUP syslogd

will work on almost any Linux box. The only possible problem is if you are trying to learn how to administer Unix systems in general, you may be spoiled by using the convenient, but Linux-specific killall command.

For more information, look at the man pages for syslogd and syslog.conf. The Sendmail book published by O'Reilly & Associates (often referred to as the “Bat book” because it features a picture of a bat on the cover) also devotes a few pages to syslog (aside from being a great book on configuring sendmail).

______________________

Webinar
One Click, Universal Protection: Implementing Centralized Security Policies on Linux Systems

As Linux continues to play an ever increasing role in corporate data centers and institutions, ensuring the integrity and protection of these systems must be a priority. With 60% of the world's websites and an increasing share of organization's mission-critical workloads running on Linux, failing to stop malware and other advanced threats on Linux can increasingly impact an organization's reputation and bottom line.

Learn More

Sponsored by Bit9

Webinar
Linux Backup and Recovery Webinar

Most companies incorporate backup procedures for critical data, which can be restored quickly if a loss occurs. However, fewer companies are prepared for catastrophic system failures, in which they lose all data, the entire operating system, applications, settings, patches and more, reducing their system(s) to “bare metal.” After all, before data can be restored to a system, there must be a system to restore it to.

In this one hour webinar, learn how to enhance your existing backup strategies for better disaster recovery preparedness using Storix System Backup Administrator (SBAdmin), a highly flexible bare-metal recovery solution for UNIX and Linux systems.

Learn More

Sponsored by Storix