swatch: Automated Log Monitoring for the Vigilant but Lazy

Sifting through logs every day, looking for trouble, could leave little time for anything else—swatch helps.
Fine-Tuning swatch (in Both Directions!)

Once swatch is configured and running, we must turn our attention to the Goldilocks Goal: we want swatch to be running neither too hot (alerting us about routine or trivial events) nor too cold (never alerting us about anything). But what constitutes just right? There are as many different answers to this question as there are uses for UNIX.

Anyhow, you don't need me to tell you what constitutes nuisance-level reporting: if it happens you'll know it. You may even experience a scare or two in responding to events that set off alarms initially but turn out to be harmless nonetheless. Read the manual, tweak .swatch.rc and stay the course.

The other scenario, in which too little is watched for, is much harder to address, especially for the beginning system administrator. By definition, anomalous events don't happen too frequently, so how do you anticipate how they'll manifest themselves in the logs? My first bit of advice is to get in the habit of browsing your system logs often enough to get a feel for what the routine operation of your systems looks like.

Better still, tail the logs in real time. If you enter the command

tail -f /var/log/messages

the last 50 lines of the system log will be printed, plus all subsequent lines, as they're generated, until you kill tail with a Ctrl-C. This works for any file, even a log file that changes rapidly.

Another good thing you can do is to “beat up on” your system in one virtual console or xterm while tailing various log files in another. The tools we explored last month and the month before, Nessus and nmap, respectively, are perfect for this.

By now you may be thinking, “Hey, I thought the whole reason I installed swatch was so I wouldn't have to watch log files manually!” Nope. swatch minimizes, but does not eliminate, the need for us to parse log files.

Were you able to quit using your arithmetic skills after you got your first pocket calculator? No. For that matter, can you use a calculator in the first place unless you already know how to add, multiply, etc.? Definitely not. Same goes for log file parsing: you can't tell swatch to look for things you can't identify yourself, no more than you can ask for directions to a town whose name you've forgotten.

Why You Shouldn't Configure swatch Once and Forget about It

In the same vein, I urge you to not be complacent about swatch silence. If swatch's actions don't fire very often, it could be that your system isn't getting probed or misused often, but it's at least as likely that swatch isn't casting its net widely enough. Continue to scan through your logs manually from time to time to see if you're missing anything, and continue to tweak .swatchrc.

And don't forget to reconsider periodically the auditing/logging configurations of the dæmons that generate log messages in the first place. swatch won't catch events that aren't logged at all. Refer to the syslogd(8) man page for general instructions on managing your syslog dæmon and the man pages of the various things that log to syslog for specific instructions on changing the way they log events.

Resources

Should We Let Perl Download and Install Its Own Modules?

Mick Bauer (mick@visi.com) is a network security consultant in the Twin Cities area. He's been a Linux devotee since 1995 and an OpenBSD zealot since 1997, taking particular pleasure in getting these cutting-edge operating systems to run on obsolete junk. Mick welcomes questions, comments and greetings.

______________________

Comments

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

page not found

gadelkareem's picture

the page you provided for the source is giving 404 error

SWATCH project @ Source Forge

Joe Lee's picture

Comments

Anonymous's picture

Great article - a side note for people not well versed in Perl
If you run the Makefile and you are missing the dependecies/modules, you must install them.

Time::HiRes
Date::Calc
Date::Format
File::Tail

The easiest thing to do is to start the program CPAN, by typing in cpan
Hit enter to allow it to pick up the default when it starts asking you a slew of questions
type in -
install Time::HiRes Date::Calc Date::Format File::Tail

It will run for quite a while and will ask you if you want to install the dependencies - you do- hit enter

When it finishes, type in exit to return to the command prompt and rerun the Makefile. That should allow
everything to run pretty well

Note: Avoid typing in the install Bundle - it runs forever and makes cpan a lot more complicated than it should be.

Webinar
One Click, Universal Protection: Implementing Centralized Security Policies on Linux Systems

As Linux continues to play an ever increasing role in corporate data centers and institutions, ensuring the integrity and protection of these systems must be a priority. With 60% of the world's websites and an increasing share of organization's mission-critical workloads running on Linux, failing to stop malware and other advanced threats on Linux can increasingly impact an organization's reputation and bottom line.

Learn More

Sponsored by Bit9

Webinar
Linux Backup and Recovery Webinar

Most companies incorporate backup procedures for critical data, which can be restored quickly if a loss occurs. However, fewer companies are prepared for catastrophic system failures, in which they lose all data, the entire operating system, applications, settings, patches and more, reducing their system(s) to “bare metal.” After all, before data can be restored to a system, there must be a system to restore it to.

In this one hour webinar, learn how to enhance your existing backup strategies for better disaster recovery preparedness using Storix System Backup Administrator (SBAdmin), a highly flexible bare-metal recovery solution for UNIX and Linux systems.

Learn More

Sponsored by Storix