swatch: Automated Log Monitoring for the Vigilant but Lazy

Sifting through logs every day, looking for trouble, could leave little time for anything else—swatch helps.
Advanced swatch Configuration

So far we've only considered actions we want to be triggered every time a given pattern is matched. There are several ways we can control swatch's behavior with greater granularity, however.

The first and most obvious way is to take advantage of the fact that search patterns take the form of regular expressions. Regular expressions, which really constitute a text-formatting language of their own, are incredibly powerful and responsible for a good deal of the magic that Perl, sed, vi and many other UNIX utilities can do.

It behooves you to know at least a couple of “regex” tricks, which I'll describe here. Trick number one is called alternation, and it adds a “logical or” to your regular expression in the form of a “|” sign. Consider this regular expression:

/reject|failed/

This expression will match any line containing either the word “reject” or the word “failed”. Use alternation when you want swatch to take the same action for more than one pattern.

Trick number two is the Perl-specific regular expression modifier “case-insensitive”, also known as “slash-i” since it always follows a regular expression's trailing slash. The regular expression /reject/i matches any line containing the word “reject”, whether it's spelled “Reject”, “REJECT”, “rEjEcT”, etc. Granted, this isn't nearly as useful as alternation, and in the interest of full disclosure, I'm compelled to mention that slash-i is one of the more CPU-intensive Perl modifiers. However, if despite your best efforts at log-tailing, self-attacking, etc., you aren't 100% sure how a worrisome attack might look in a log file, slash-i helps you make a reasonable guess.

If you wish to become a regular expression archimage, I recommend the book Mastering Regular Expressions by Jeffrey E. F. Friedl. See Resources for details.

Another way to control swatch to a greater degree is to specify what time of day a given action may be performed. You can do this by sticking a “when=” option after any action. For example, below I've got a .swatchrc entry for a medium-importance event I want to know about via console messages during weekdays, but I'll need e-mail messages to know about it during the weekend. To do this I set the when option:

/file system full/
      echo=red
      mail addresses=mick\@visi.com,
      subject=Volume_Full,when=7-1:1-24

The syntax of the when= option is when=range_of_days:range_of_hours. Thus, we see that any time the message “file system full” is logged, swatch will echo the log entry to the console in red ink. It will also send e-mail, but only if it's Saturday (“7”) or Sunday (“1”).

Running swatch

swatch expects .swatchrc to live in the home directory of the user who invokes swatch. swatch also keeps its temporary files there by default (each time it's invoked it creates and runs a script called a “watcher process”, whose name ends with a dot followed by the PID of the swatch process that created it).

The -c path_to_configfile and --script-dir=path flags let you specify alternate locations for swatch's configuration and script files, respectively. Never keep either in a world-writable directory, however. In fact, only these files' owners should even be able to read them.

For example, to invoke swatch so it reads my custom configuration file in /var/log and also uses that directory for its watcher process script, I'd use this command:

swatch -c /var/log/.swatchrc.access --script-dir=/var/log &

I also need to tell swatch which file to tail, and for that I need the -t filename flag. If I wanted to use the above command to have swatch monitor /var/log/apache/access_log, it would look like this:

swatch -c /var/log/.swatchrc.access --script-dir=/var/log \
       -t /var/log/apache/access_log &
swatch generally doesn't clean up after itself very well; it tends to leave watcher-process scripts behind. Keep an eye out for and periodically delete these in your home directory or in the script directories you tend to specify with --script-dir.

Again, if you want swatch to monitor multiple files, you'll need to run swatch multiple times, with at least a different tailing-target (-t value) specified each time and probably a different configuration file for each as well.

______________________

Comments

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

page not found

gadelkareem's picture

the page you provided for the source is giving 404 error

SWATCH project @ Source Forge

Joe Lee's picture

Comments

Anonymous's picture

Great article - a side note for people not well versed in Perl
If you run the Makefile and you are missing the dependecies/modules, you must install them.

Time::HiRes
Date::Calc
Date::Format
File::Tail

The easiest thing to do is to start the program CPAN, by typing in cpan
Hit enter to allow it to pick up the default when it starts asking you a slew of questions
type in -
install Time::HiRes Date::Calc Date::Format File::Tail

It will run for quite a while and will ask you if you want to install the dependencies - you do- hit enter

When it finishes, type in exit to return to the command prompt and rerun the Makefile. That should allow
everything to run pretty well

Note: Avoid typing in the install Bundle - it runs forever and makes cpan a lot more complicated than it should be.

White Paper
Linux Management with Red Hat Satellite: Measuring Business Impact and ROI

Linux has become a key foundation for supporting today's rapidly growing IT environments. Linux is being used to deploy business applications and databases, trading on its reputation as a low-cost operating environment. For many IT organizations, Linux is a mainstay for deploying Web servers and has evolved from handling basic file, print, and utility workloads to running mission-critical applications and databases, physically, virtually, and in the cloud. As Linux grows in importance in terms of value to the business, managing Linux environments to high standards of service quality — availability, security, and performance — becomes an essential requirement for business success.

Learn More

Sponsored by Red Hat

White Paper
Private PaaS for the Agile Enterprise

If you already use virtualized infrastructure, you are well on your way to leveraging the power of the cloud. Virtualization offers the promise of limitless resources, but how do you manage that scalability when your DevOps team doesn’t scale? In today’s hypercompetitive markets, fast results can make a difference between leading the pack vs. obsolescence. Organizations need more benefits from cloud computing than just raw resources. They need agility, flexibility, convenience, ROI, and control.

Stackato private Platform-as-a-Service technology from ActiveState extends your private cloud infrastructure by creating a private PaaS to provide on-demand availability, flexibility, control, and ultimately, faster time-to-market for your enterprise.

Learn More

Sponsored by ActiveState