swatch: Automated Log Monitoring for the Vigilant but Lazy

Once swatch is configured and running, we must turn our attention to the Goldilocks Goal: we want swatch to be running neither too hot (alerting us about routine or trivial events) nor too cold (never alerting us about anything). But what constitutes just right? There are as many different answers to this question as there are uses for UNIX.

Anyhow, you don't need me to tell you what constitutes nuisance-level reporting: if it happens you'll know it. You may even experience a scare or two in responding to events that set off alarms initially but turn out to be harmless nonetheless. Read the manual, tweak .swatch.rc and stay the course.

The other scenario, in which too little is watched for, is much harder to address, especially for the beginning system administrator. By definition, anomalous events don't happen too frequently, so how do you anticipate how they'll manifest themselves in the logs? My first bit of advice is to get in the habit of browsing your system logs often enough to get a feel for what the routine operation of your systems looks like.

Better still, tail the logs in real time. If you enter the command

tail -f /var/log/messages

the last 50 lines of the system log will be printed, plus all subsequent lines, as they're generated, until you kill tail with a Ctrl-C. This works for any file, even a log file that changes rapidly.

Another good thing you can do is to “beat up on” your system in one virtual console or xterm while tailing various log files in another. The tools we explored last month and the month before, Nessus and nmap, respectively, are perfect for this.

By now you may be thinking, “Hey, I thought the whole reason I installed swatch was so I wouldn't have to watch log files manually!” Nope. swatch minimizes, but does not eliminate, the need for us to parse log files.

Were you able to quit using your arithmetic skills after you got your first pocket calculator? No. For that matter, can you use a calculator in the first place unless you already know how to add, multiply, etc.? Definitely not. Same goes for log file parsing: you can't tell swatch to look for things you can't identify yourself, no more than you can ask for directions to a town whose name you've forgotten.

In the same vein, I urge you to not be complacent about swatch silence. If swatch's actions don't fire very often, it could be that your system isn't getting probed or misused often, but it's at least as likely that swatch isn't casting its net widely enough. Continue to scan through your logs manually from time to time to see if you're missing anything, and continue to tweak .swatchrc.

And don't forget to reconsider periodically the auditing/logging configurations of the dæmons that generate log messages in the first place. swatch won't catch events that aren't logged at all. Refer to the syslogd(8) man page for general instructions on managing your syslog dæmon and the man pages of the various things that log to syslog for specific instructions on changing the way they log events.

Resources

Should We Let Perl Download and Install Its Own Modules?