How to Log Friends and Influence People

 in
This month, Mark discusses how to keep track of what is happening to (and on) your Linux system.

The syslog daemon (known as syslogd) is used to receive messages from various parts of the Linux (or other Unix) system and distribute these messages to other locations. This allows you, the administrator, to review system activities by keeping a log of what goes on behind the scenes. It's a great way of tracking down problems and also of reviewing your system security.

For example, you can keep track of serious errors, such as TCP/IP problems or lack of memory (for when ghostscript and X get a bit out of hand). At the same time, you can debug that pesky mail problem that a user is having, or you can keep track of who has used the su program to assume root status, and you can also find out who failed trying to do so.

To get a copy of the syslog program, see the sidebar on obtaining syslog from an ftp site near you.

Syslogd has a setup file, called /etc/syslog.conf, which tells syslogd what messages to log, and what to do with the messages it receives. By default (i.e., the /etc/syslog.conf file is empty), messages are not stored anywhere. Most distributions of Linux have some basic setup of the syslog.conf file included, and the syslogd program is often started in the /etc/rc.d/rc.inet2 file.

The setup of your syslog.conf file is in three parts. Let's look at a sample syslog.conf file:

# /etc/syslog.conf
 *.info;*.notice    /usr/adm/messages
 *.debug           /usr/adm/debug
 *.warn            /usr/adm/syslog

The first line is a comment. Any lines starting with # or completely blank lines are considered comments and are ignored by syslogd. You can see two columns here, separated by at least one tab. The left side (the one with *.info;*.notice in it) defines the facility and the logging level. The right side defines the destination of the messages.

The facility can be defined as the “kind” of program that is running. For example, “mail” is the facility used whether you are running sendmail or smail. There are nine pre-defined facilities, plus eight facilities that are locally defined, a “wildcard” facility that means “all facilities” and one facility for issuing timestamps.

Here is what each facility is for:

Facility

Used For

user

user processes

kern

kernel messages

mail

mail

daemon

various daemons (such as ftpd)

auth

authorization (such as login)

lpr

the line printer

news

USENET news (nntp)

uucp

UUCP (Unix to Unix copy)

cron

the cron daemon

mark

a periodic message created for placing timestamps in log files

local0-local7

locally-defined facilities

*

all facilities (except mark)

One special note about the mark facility. This facility is created by syslogd every twenty minutes at the info level. This will allow you to quickly see time change in a log file, along with making sure that syslogd is running (and logging). And as noted above, the special * facility (which means all the available facilities) does not include mark.

Each facility can have a logging (or severity) level to it. This is used to indicate which types of messages you want to record. The high severity levels (such as emerg) require the most attention, as an emerg usually indicates that the program will fail soon. These severity levels decrease in importance all the way down to the debug level, which is used for properly setting up your software. Once your software is configured correctly, you can change the severity level to a higher one. Check the documentation on your software for its interpretation of the various levels. From highest severity to lowest:emergalertcriterrwarningnoticeinfodebugnone

The severity of none means that no messages from that facility are to be recorded. These levels mean that messages of that level and above are recorded. For example, say you have the following two lines:

mail.debug      /var/adm/syslog.mail
mail.emerg      /var/adm/syslog.mail.emerg

When sendmail (or smail), or whatever program is logging as the mail facility) sends syslog a message with a level of debug, it gets placed in the syslog.mail file. Any other messages, of any level from info up to emerg, also get placed in syslog.mail. Emergency messages from mail get sent to both syslog.mail and to syslog.mail.emerg. This setup makes it extremely easy to check for emergency conditions for your software, since a glance at the directory listing for syslog.mail.emerg will tell you if the file has changed recently, and you can easily type tail/var/adm/syslog.mail.emerg to glance at the ten most recent entries. In addition, you can find the emergency message in syslog.mail and see the other messages surrounding it to determine the events leading up to the emergency message.

In another example, we can see the use of the “none” severity level:

*.alert;user.none       /var/adm/syslog.alert
user.alert              /var/adm/syslog.user.alert

Here, all messages at alert severity and above are sent to syslog.alert, except for the ones in the user facility. Those get sent to the syslog.user.alert file instead, as specified by the second line.

Now you are probably wondering about the right hand side of the lines. By now, you have figured out that this can be the name of a file. But it can also be the name of a user or of a remote host. If the intended recipient of the message is a file, the name of the file must start with a /, indicating that you have to give a full path for the filename. Note that this can be just about any file, including /dev/console, which will print the message to the console of the machine, or /dev/lp2 to print your message on paper.

If the recipient is a remote host, it will start with an @ followed by the name of the host. The intended host will receive the message via TCP/IP port 514, and from this point, it is just like a message to syslog locally. A message to the remote host must go through syslog and get logged to the appropriate location. This will allow you to monitor a network of machines from one location. You can also include a comma-separated list of users and the messages will be printed on those users' screens if they are logged in. You can also include a *, which means that every user that is logged in gets the message. This is very useful for extreme emergency conditions where you want the users to realize that something is very wrong and they should log out.

#Log mail errors to the mail host for
#the postmaster to deal with
mail.*         @mailhost
#Send kernel emergency warnings to all
#users so they know what's up
kern.emerg     *

Once you have your /etc/syslog.conf file set up, you have to first set up the files that will be storing the messages, then restart syslogd.

Setting your files up for syslog merely means reviewing the permissions of the log files. By default, files are set up owned by root, in the root group, readable by everyone, and writable only by the root user. For a one-user system, this should be fine. But for a multi-user system, with (possibly) logs of who sent and received e-mail, many of these files should only be read by the root user. A good suggestion for file setup is to make it read-write by root and read-only by the wheel group. This allows you to set up some security (by controlling who is in the wheel group) without having to `su' to root every time you want to check some of the files. Re-starting syslogd does not require killing the syslogd program. All that is needed is to send a SIGHUP (signal number 1) to the process, and it will re-read its configuration files. To help you out even more, the PID of syslogd is stored in the /etc/syslog.pid file. As root:

kill -HUP `cat /etc/syslog.pid`

will re-start the syslog daemon and put your changes into effect. Some Linux packages also include a script called /etc/syslogd.restart, which is merely a script that runs the above kill program. If for some reason you want to actually kill the syslogd program, a TERM signal (or 15) will kill the program (kill -TERM `cat /etc/syslog.pid`)

If you have trouble with either of these, consider using the killall program. This is a linux-specific program, but is quite useful:

killall -HUP syslogd

will work on almost any Linux box. The only possible problem is if you are trying to learn how to administer Unix systems in general, you may be spoiled by using the convenient, but Linux-specific killall command.

For more information, look at the man pages for syslogd and syslog.conf. The Sendmail book published by O'Reilly & Associates (often referred to as the “Bat book” because it features a picture of a bat on the cover) also devotes a few pages to syslog (aside from being a great book on configuring sendmail).

______________________

White Paper
Linux Management with Red Hat Satellite: Measuring Business Impact and ROI

Linux has become a key foundation for supporting today's rapidly growing IT environments. Linux is being used to deploy business applications and databases, trading on its reputation as a low-cost operating environment. For many IT organizations, Linux is a mainstay for deploying Web servers and has evolved from handling basic file, print, and utility workloads to running mission-critical applications and databases, physically, virtually, and in the cloud. As Linux grows in importance in terms of value to the business, managing Linux environments to high standards of service quality — availability, security, and performance — becomes an essential requirement for business success.

Learn More

Sponsored by Red Hat

White Paper
Private PaaS for the Agile Enterprise

If you already use virtualized infrastructure, you are well on your way to leveraging the power of the cloud. Virtualization offers the promise of limitless resources, but how do you manage that scalability when your DevOps team doesn’t scale? In today’s hypercompetitive markets, fast results can make a difference between leading the pack vs. obsolescence. Organizations need more benefits from cloud computing than just raw resources. They need agility, flexibility, convenience, ROI, and control.

Stackato private Platform-as-a-Service technology from ActiveState extends your private cloud infrastructure by creating a private PaaS to provide on-demand availability, flexibility, control, and ultimately, faster time-to-market for your enterprise.

Learn More

Sponsored by ActiveState