Managing your Logs with Chklogs

An introduction to a program written by Mr. Grimaldo to manage system logs.

One of the attributes that characterizes Unix systems, and therefore Linux, is true multitasking. Because of it, there are usually many processes running on your machine, including the not-so-evil daemons and other important programs such as uucico (Unix-to-Unix copy-in copy-out). If your system is properly configured, those programs leave traces of their activities in the system logs. These logs usually contain lots of data that must be filtered to generate more readable reports. In the case of system problems, these logs are a valuable source of information for tracking, and possibly solving, the problem.

Years ago, when I was faced with the prospect of spending some of my free time looking through raw logs and trimming them so that they wouldn't eat up disk space, I decided it was time to take action—I wrote the program Chklogs. ChkLogs is an acronym for “Check Logs” in the Unix tradition.

Whether you are an experienced Linux user, system administrator or a newcomer to the Linux world, you will certainly find this subject of interest. Although it is mainly a system administration tool, it also has applications in the user world.

In this article I will introduce you to version 2.0 of Chklogs which should be out by the time this is published. Currently version 1.9 release/build 2 (1.9-2) is available, which is the last Perl 4.0x compatible version. Version 2.0 and higher require Perl 5.003—which is not much to ask considering about 99% of registered users have that Perl version.

Features

Here are some of the features offered by Chklogs:

  • Individual specification of thresholds and action(s) for each log

  • A choice of compression program

  • Log management by size or age

  • Addition of user extensions

  • Logical Log Groups with pre- and post-processing

  • Global and local repositories (Alternate Repository feature)

  • Log shuffling (also known as log rotation)

  • Directory lumping

  • Requires no programming experience

  • User resource file

  • Fully based on Perl 5.003, except for the user interface

  • Nice Tcl/Tk user interface, available separately (See Resources at the end of this article.)

Logical Log Groups are groups of logs that have something in common; this could be a collection of UUCP logs, INN logs or anything you consider a group. You can use virtually any name; the only restriction is that the name must be valid to create a directory. Chklogs has two built-in groups: syslog and common (reserved).

Alternate Repository (AR) is a directory where the archived logs are stored after they have been processed if they met the threshold condition. By default, archived logs are created with tar and GNU zip. A local AR is a directory named OldLogs under the directory in which the log resides. A global AR is a directory hierarchy (you define the root of this AR) where log archives are divided into logical groups. Here, you will always see the “common” subdirectory where all “orphan” logs go, i.e., those that have not been explicitly declared to belong to a group.

Log Shuffling is well known and is also called log rotation. The “phased-out” log is assigned a number or tag each time until a maximum is reached, at which point the oldest one is removed.

Directory Lumping is another nice feature. Instead of specifying each log separately by name, you supply a directory name. Chklogs treats any non-archived, non-directory file (as recognized by Chklogs) as a log and acts on it according to the action specification. A real-world application of this option is a site that gives UUCP access to a large number of subdomains (very cumbersome to do by hand), and a separate UUCP transfer log is kept for each site. Chklogs always determines first if you have specified a directory or a file.

Installing Chklogs

Every time I make the first release/build of a new version (e.g., v2.0-1), I submit it to Sunsite and Funet FTP sites. Whenever a fix is required (and therefore no new features), I put out a new release/build (e.g., 2.0-2) only on the primary site, my ISP. To make sure that you have the latest version, check out the primary site and/or the Chklogs web page at http://www.iaehv.nl/users/grimaldo/info/.

Now, get “version 2.0 build 1” installed on your system. Unpack it and go to the root of the directory tree:

gunzip chklogs-2.0-1.tar.gz
tar xvf chklogs-2.0-1.tar
cd chklogs-2.0-1

Under the root tree is the bin directory containing the scripts and modules which comprise the Chklogs package. The /doc directory contains all the necessary documentation, including man pages in both troff and HTML format. A plug-out directory contains some extra utilities to scan your logs—use them as is or as examples to build your own. Last is the /contrib directory. In the root of the tree there is the README file, release notes, a makefile for installation and, most important, the GuideMe script. Type:

GuideMe
And it will do just that or at least make the attempt . This script performs a probe into your system and indicates which configuration parameters must be changed in which files. Follow its advice closely. In the end, it will ask you if you wish to send in your registration. If you select “no” that's fine. If “yes”, you will receive mail regarding updates and major fixes.

Assuming you have made the necessary configuration parameter changes (mailer, compress program, administrative account, library location, Perl location etc.), you are now ready to actually make Chklogs work for your system. I will also assume it has been installed (see Makefile) in /usr/local/sbin and /usr/local/lib/Degt/ by the make command (make install). The same library directory is shared by the graphical user interface.

If you are not yet sure you wish to commit your logs to Chklogs, you should make a back-up copy of them in a local directory. Run Chklogs on those copies until you feel safe, and you will. Doing this requires use of the resource file (see below).

______________________

White Paper
Linux Management with Red Hat Satellite: Measuring Business Impact and ROI

Linux has become a key foundation for supporting today's rapidly growing IT environments. Linux is being used to deploy business applications and databases, trading on its reputation as a low-cost operating environment. For many IT organizations, Linux is a mainstay for deploying Web servers and has evolved from handling basic file, print, and utility workloads to running mission-critical applications and databases, physically, virtually, and in the cloud. As Linux grows in importance in terms of value to the business, managing Linux environments to high standards of service quality — availability, security, and performance — becomes an essential requirement for business success.

Learn More

Sponsored by Red Hat

White Paper
Private PaaS for the Agile Enterprise

If you already use virtualized infrastructure, you are well on your way to leveraging the power of the cloud. Virtualization offers the promise of limitless resources, but how do you manage that scalability when your DevOps team doesn’t scale? In today’s hypercompetitive markets, fast results can make a difference between leading the pack vs. obsolescence. Organizations need more benefits from cloud computing than just raw resources. They need agility, flexibility, convenience, ROI, and control.

Stackato private Platform-as-a-Service technology from ActiveState extends your private cloud infrastructure by creating a private PaaS to provide on-demand availability, flexibility, control, and ultimately, faster time-to-market for your enterprise.

Learn More

Sponsored by ActiveState