swatch: Automated Log Monitoring for the Vigilant but Lazy

Sifting through logs every day, looking for trouble, could leave little time for anything else—swatch helps.
Fine-Tuning swatch (in Both Directions!)

Once swatch is configured and running, we must turn our attention to the Goldilocks Goal: we want swatch to be running neither too hot (alerting us about routine or trivial events) nor too cold (never alerting us about anything). But what constitutes just right? There are as many different answers to this question as there are uses for UNIX.

Anyhow, you don't need me to tell you what constitutes nuisance-level reporting: if it happens you'll know it. You may even experience a scare or two in responding to events that set off alarms initially but turn out to be harmless nonetheless. Read the manual, tweak .swatch.rc and stay the course.

The other scenario, in which too little is watched for, is much harder to address, especially for the beginning system administrator. By definition, anomalous events don't happen too frequently, so how do you anticipate how they'll manifest themselves in the logs? My first bit of advice is to get in the habit of browsing your system logs often enough to get a feel for what the routine operation of your systems looks like.

Better still, tail the logs in real time. If you enter the command

tail -f /var/log/messages

the last 50 lines of the system log will be printed, plus all subsequent lines, as they're generated, until you kill tail with a Ctrl-C. This works for any file, even a log file that changes rapidly.

Another good thing you can do is to “beat up on” your system in one virtual console or xterm while tailing various log files in another. The tools we explored last month and the month before, Nessus and nmap, respectively, are perfect for this.

By now you may be thinking, “Hey, I thought the whole reason I installed swatch was so I wouldn't have to watch log files manually!” Nope. swatch minimizes, but does not eliminate, the need for us to parse log files.

Were you able to quit using your arithmetic skills after you got your first pocket calculator? No. For that matter, can you use a calculator in the first place unless you already know how to add, multiply, etc.? Definitely not. Same goes for log file parsing: you can't tell swatch to look for things you can't identify yourself, no more than you can ask for directions to a town whose name you've forgotten.

Why You Shouldn't Configure swatch Once and Forget about It

In the same vein, I urge you to not be complacent about swatch silence. If swatch's actions don't fire very often, it could be that your system isn't getting probed or misused often, but it's at least as likely that swatch isn't casting its net widely enough. Continue to scan through your logs manually from time to time to see if you're missing anything, and continue to tweak .swatchrc.

And don't forget to reconsider periodically the auditing/logging configurations of the dæmons that generate log messages in the first place. swatch won't catch events that aren't logged at all. Refer to the syslogd(8) man page for general instructions on managing your syslog dæmon and the man pages of the various things that log to syslog for specific instructions on changing the way they log events.

Resources

Should We Let Perl Download and Install Its Own Modules?

Mick Bauer (mick@visi.com) is a network security consultant in the Twin Cities area. He's been a Linux devotee since 1995 and an OpenBSD zealot since 1997, taking particular pleasure in getting these cutting-edge operating systems to run on obsolete junk. Mick welcomes questions, comments and greetings.

______________________

Comments

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

page not found

gadelkareem's picture

the page you provided for the source is giving 404 error

SWATCH project @ Source Forge

Joe Lee's picture

Comments

Anonymous's picture

Great article - a side note for people not well versed in Perl
If you run the Makefile and you are missing the dependecies/modules, you must install them.

Time::HiRes
Date::Calc
Date::Format
File::Tail

The easiest thing to do is to start the program CPAN, by typing in cpan
Hit enter to allow it to pick up the default when it starts asking you a slew of questions
type in -
install Time::HiRes Date::Calc Date::Format File::Tail

It will run for quite a while and will ask you if you want to install the dependencies - you do- hit enter

When it finishes, type in exit to return to the command prompt and rerun the Makefile. That should allow
everything to run pretty well

Note: Avoid typing in the install Bundle - it runs forever and makes cpan a lot more complicated than it should be.

White Paper
Linux Management with Red Hat Satellite: Measuring Business Impact and ROI

Linux has become a key foundation for supporting today's rapidly growing IT environments. Linux is being used to deploy business applications and databases, trading on its reputation as a low-cost operating environment. For many IT organizations, Linux is a mainstay for deploying Web servers and has evolved from handling basic file, print, and utility workloads to running mission-critical applications and databases, physically, virtually, and in the cloud. As Linux grows in importance in terms of value to the business, managing Linux environments to high standards of service quality — availability, security, and performance — becomes an essential requirement for business success.

Learn More

Sponsored by Red Hat

White Paper
Private PaaS for the Agile Enterprise

If you already use virtualized infrastructure, you are well on your way to leveraging the power of the cloud. Virtualization offers the promise of limitless resources, but how do you manage that scalability when your DevOps team doesn’t scale? In today’s hypercompetitive markets, fast results can make a difference between leading the pack vs. obsolescence. Organizations need more benefits from cloud computing than just raw resources. They need agility, flexibility, convenience, ROI, and control.

Stackato private Platform-as-a-Service technology from ActiveState extends your private cloud infrastructure by creating a private PaaS to provide on-demand availability, flexibility, control, and ultimately, faster time-to-market for your enterprise.

Learn More

Sponsored by ActiveState