swatch: Automated Log Monitoring for the Vigilant but Lazy

So far we've only considered actions we want to be triggered every time a given pattern is matched. There are several ways we can control swatch's behavior with greater granularity, however.

The first and most obvious way is to take advantage of the fact that search patterns take the form of regular expressions. Regular expressions, which really constitute a text-formatting language of their own, are incredibly powerful and responsible for a good deal of the magic that Perl, sed, vi and many other UNIX utilities can do.

It behooves you to know at least a couple of “regex” tricks, which I'll describe here. Trick number one is called alternation, and it adds a “logical or” to your regular expression in the form of a “|” sign. Consider this regular expression:

/reject|failed/

This expression will match any line containing either the word “reject” or the word “failed”. Use alternation when you want swatch to take the same action for more than one pattern.

Trick number two is the Perl-specific regular expression modifier “case-insensitive”, also known as “slash-i” since it always follows a regular expression's trailing slash. The regular expression /reject/i matches any line containing the word “reject”, whether it's spelled “Reject”, “REJECT”, “rEjEcT”, etc. Granted, this isn't nearly as useful as alternation, and in the interest of full disclosure, I'm compelled to mention that slash-i is one of the more CPU-intensive Perl modifiers. However, if despite your best efforts at log-tailing, self-attacking, etc., you aren't 100% sure how a worrisome attack might look in a log file, slash-i helps you make a reasonable guess.

If you wish to become a regular expression archimage, I recommend the book Mastering Regular Expressions by Jeffrey E. F. Friedl. See Resources for details.

Another way to control swatch to a greater degree is to specify what time of day a given action may be performed. You can do this by sticking a “when=” option after any action. For example, below I've got a .swatchrc entry for a medium-importance event I want to know about via console messages during weekdays, but I'll need e-mail messages to know about it during the weekend. To do this I set the when option:

/file system full/
      echo=red
      mail addresses=mick\@visi.com,
      subject=Volume_Full,when=7-1:1-24

The syntax of the when= option is when=range_of_days:range_of_hours. Thus, we see that any time the message “file system full” is logged, swatch will echo the log entry to the console in red ink. It will also send e-mail, but only if it's Saturday (“7”) or Sunday (“1”).

swatch expects .swatchrc to live in the home directory of the user who invokes swatch. swatch also keeps its temporary files there by default (each time it's invoked it creates and runs a script called a “watcher process”, whose name ends with a dot followed by the PID of the swatch process that created it).

The -c path_to_configfile and --script-dir=path flags let you specify alternate locations for swatch's configuration and script files, respectively. Never keep either in a world-writable directory, however. In fact, only these files' owners should even be able to read them.

For example, to invoke swatch so it reads my custom configuration file in /var/log and also uses that directory for its watcher process script, I'd use this command:

swatch -c /var/log/.swatchrc.access --script-dir=/var/log &

I also need to tell swatch which file to tail, and for that I need the -t filename flag. If I wanted to use the above command to have swatch monitor /var/log/apache/access_log, it would look like this:

swatch -c /var/log/.swatchrc.access --script-dir=/var/log \
       -t /var/log/apache/access_log &

Again, if you want swatch to monitor multiple files, you'll need to run swatch multiple times, with at least a different tailing-target (-t value) specified each time and probably a different configuration file for each as well.