swatch: Automated Log Monitoring for the Vigilant but Lazy
So far we've only considered actions we want to be triggered every time a given pattern is matched. There are several ways we can control swatch's behavior with greater granularity, however.
The first and most obvious way is to take advantage of the fact that search patterns take the form of regular expressions. Regular expressions, which really constitute a text-formatting language of their own, are incredibly powerful and responsible for a good deal of the magic that Perl, sed, vi and many other UNIX utilities can do.
It behooves you to know at least a couple of “regex” tricks, which I'll describe here. Trick number one is called alternation, and it adds a “logical or” to your regular expression in the form of a “|” sign. Consider this regular expression:
This expression will match any line containing either the word “reject” or the word “failed”. Use alternation when you want swatch to take the same action for more than one pattern.
Trick number two is the Perl-specific regular expression modifier “case-insensitive”, also known as “slash-i” since it always follows a regular expression's trailing slash. The regular expression /reject/i matches any line containing the word “reject”, whether it's spelled “Reject”, “REJECT”, “rEjEcT”, etc. Granted, this isn't nearly as useful as alternation, and in the interest of full disclosure, I'm compelled to mention that slash-i is one of the more CPU-intensive Perl modifiers. However, if despite your best efforts at log-tailing, self-attacking, etc., you aren't 100% sure how a worrisome attack might look in a log file, slash-i helps you make a reasonable guess.
If you wish to become a regular expression archimage, I recommend the book Mastering Regular Expressions by Jeffrey E. F. Friedl. See Resources for details.
Another way to control swatch to a greater degree is to specify what time of day a given action may be performed. You can do this by sticking a “when=” option after any action. For example, below I've got a .swatchrc entry for a medium-importance event I want to know about via console messages during weekdays, but I'll need e-mail messages to know about it during the weekend. To do this I set the when option:
/file system full/ echo=red mail addresses=mick\@visi.com, subject=Volume_Full,when=7-1:1-24
The syntax of the when= option is when=range_of_days:range_of_hours. Thus, we see that any time the message “file system full” is logged, swatch will echo the log entry to the console in red ink. It will also send e-mail, but only if it's Saturday (“7”) or Sunday (“1”).
swatch expects .swatchrc to live in the home directory of the user who invokes swatch. swatch also keeps its temporary files there by default (each time it's invoked it creates and runs a script called a “watcher process”, whose name ends with a dot followed by the PID of the swatch process that created it).
The -c path_to_configfile and --script-dir=path flags let you specify alternate locations for swatch's configuration and script files, respectively. Never keep either in a world-writable directory, however. In fact, only these files' owners should even be able to read them.
For example, to invoke swatch so it reads my custom configuration file in /var/log and also uses that directory for its watcher process script, I'd use this command:
swatch -c /var/log/.swatchrc.access --script-dir=/var/log &
I also need to tell swatch which file to tail, and for that I need the -t filename flag. If I wanted to use the above command to have swatch monitor /var/log/apache/access_log, it would look like this:
swatch -c /var/log/.swatchrc.access --script-dir=/var/log \ -t /var/log/apache/access_log &swatch generally doesn't clean up after itself very well; it tends to leave watcher-process scripts behind. Keep an eye out for and periodically delete these in your home directory or in the script directories you tend to specify with --script-dir.
Again, if you want swatch to monitor multiple files, you'll need to run swatch multiple times, with at least a different tailing-target (-t value) specified each time and probably a different configuration file for each as well.
Practical Task Scheduling Deployment
July 20, 2016 12:00 pm CDT
One of the best things about the UNIX environment (aside from being stable and efficient) is the vast array of software tools available to help you do your job. Traditionally, a UNIX tool does only one thing, but does that one thing very well. For example, grep is very easy to use and can search vast amounts of data quickly. The find tool can find a particular file or files based on all kinds of criteria. It's pretty easy to string these tools together to build even more powerful tools, such as a tool that finds all of the .log files in the /home directory and searches each one for a particular entry. This erector-set mentality allows UNIX system administrators to seem to always have the right tool for the job.
Cron traditionally has been considered another such a tool for job scheduling, but is it enough? This webinar considers that very question. The first part builds on a previous Geek Guide, Beyond Cron, and briefly describes how to know when it might be time to consider upgrading your job scheduling infrastructure. The second part presents an actual planning and implementation framework.
Join Linux Journal's Mike Diehl and Pat Cameron of Help Systems.
Free to Linux Journal readers.Register Now!
- Stunnel Security for Oracle
- SourceClear Open
- SUSE LLC's SUSE Manager
- My +1 Sword of Productivity
- Murat Yener and Onur Dundar's Expert Android Studio (Wrox)
- Tech Tip: Really Simple HTTP Server with Python
- Managing Linux Using Puppet
- Non-Linux FOSS: Caffeine!
- Doing for User Space What We Did for Kernel Space
- Google's SwiftShader Released