Managing your Logs with Chklogs
The Chklogs package consists of an administrative program (chklogsadm), the main program (chklogs) and some Perl modules (Smtp.pm, Chklogs.pm, Interp.pm). For the second part of the configuration, we need the Chklogs configuration file (chklogs.conf) and, optionally, the resource file.
The resource file (.chklogsrc) resides in your home directory. You need it if you wish to first test on a copy of your logs or if you don't have root permissions (therefore, not used for the system logs but for others). When Chklogs runs, it first gets configuration information that is coded in the scripts configuration variables (the configuration section is clearly marked in the source), then it looks for a Personal Resource File (PRF) and then the command-line options, in that order.
In the PRF you can override some variables from the script configuration, for example, the admin user and others. Here is an example provided in the distribution:
# Personal resource file for Chklogs 2.x set ChklogsConf ~/devel/test/chklogs.conf set ChklogsDb ~grimaldo/devel/test/.chklogsdb set VarRun ~/devel/test set RelativePath MyOldLogs set Admin grimaldo mode ignore on set SyslogConf /etc/syslog.conf
You will notice it has a syntax similar to Tcl. That is, with the exception of the mode lines, everything between ignore on and ignore off is not interpreted. The Tcl/Tk GUI does have a dummy mode routine, but all variable settings are performed as this ignore mode is only meaningful in the Perl mode. Personally, I find the dollar sign in front of normal variables annoying. This same resource file (with some extensions) is used by the GUI. The PRF example from above does not show the variables that are used exclusively by the GUI. However, for the examples in this article, I will assume you are working on the actual logs and not a copy.
In the above script, all variables are overrides for the internal configuration variables. ChklogsConf indicates the location of the configuration file (introduced in the next section), ChklogsDb is the internal database (you must never edit or move this file around), and VarRun is the directory where the PID lock files are kept (including where to find Syslog's and Chklogs' PID file). RelativePath is used when you choose the local repository, and Admin is the e-mail address where the reports are sent when the mail command-line option is specified. Normally, I don't need this file when running Chklogs, but I do use it during testing as a non-privileged user.
Listing 1 is a sample configuration file demonstrating the various features. This file contains three types of lines: comments/spacers, directives and specifications. It is divided into two major sections, the header and the body:
A directive line begins with the two characters #: and because this pattern is checked first, it is not confused for a comment.
A comment line is introduced by the # character. Note that you cannot have comments at the end of a specification line.
The specification line is any line that is not empty or a directive or comment. It defines the attributes for each of the logs you wish ChkLogs to manage.
The header is comprised of the comment lines that identify this configuration for a particular host, and the two directive lines which must appear in the order shown. The header can be created with either of the following two administrative commands:
chklogsadm newconf -global /var/log/Chklogs
chklogsadm newconf -localThe first directive (options) is used to set global options for this configuration. At present only two option names are recognized, and both deal with the Alternate Repository Feature explained earlier. The two acceptable options are global and local. The second directive (global) is also related to the Alternate Repository. It must be set to the root directory where the logs are going to be archived (i.e., those with “archive” attribute) by group hierarchy.
In the instruction section or body, we find mostly specification lines. These lines follow a particular syntax:
LogName Threshold Action [Parameter(s)]
The first field is a fully-qualified log name, the second is a threshold that can be a size or age. The size must be in bytes or, optionally, append the K qualifier for kilobytes. If you opt for an age, it must be in days by appending a D (e.g., 7d) or in months by appending an M (e.g., 2m). The qualifiers D, M and K are case insensitive. The log is always compared to the threshold, if it is reached then the specified “Action” is taken.
The Action field can have a value of archive, execute or truncate, which has no arguments, Chklogs uses the truncate system call to chop the file to a length of zero. Use this option if you don't care about this log or after you have used the log with either the archive or execute options as shown in Listing 1.
The archive action takes one parameter that indicates how many logs to keep archived before rotating/shuffling. If no parameter is given, an internal one is used (see documentation). Use this option, if you plan to look at this log at a later time.
The execute action is used to spawn a plug-out, that is, an external program that will operate on the log. This is useful if you have a particular log scanner that will filter the log and give you a human-readable report. You can specify as many parameters as you wish, the special parameter %L is replaced by Chklogs with the full name of the log and is used to inform the plug-out which file to process.
Listing 1 also shows a Logical Group definition. There, a group named NetNews is created so that, if a global alternate repository is used, there will be a NetNews directory under it in addition to the /common directory. For each group, you can associate an external program to execute in the plug-out interface. Parameters are allowed, but %L is meaningless in that context. Pre and Post specify which program to execute before and after the group has been processed respectively. A group definition always starts with the three directive lines in that order, then one or several logs and, finally, an empty line or a comment line. Do not include any of these in between the lines of the group definition.
In the case of the sample NetNews group, we want the daemon to stop using the logs before we touch them and restart (or continue) after we are finished. This is particularly useful for INN (which has a program to control the daemon) and HTTP. Chklogs handles syslog automatically; it knows what to do and how to do it.
Finally, notice that the last specification line refers to a directory as explained earlier in the Directory Lumping section.
|Non-Linux FOSS: libnotify, OS X Style||Jun 18, 2013|
|Containers—Not Virtual Machines—Are the Future Cloud||Jun 17, 2013|
|Lock-Free Multi-Producer Multi-Consumer Queue on Ring Buffer||Jun 12, 2013|
|Weechat, Irssi's Little Brother||Jun 11, 2013|
|One Tail Just Isn't Enough||Jun 07, 2013|
|Introduction to MapReduce with Hadoop on Linux||Jun 05, 2013|
- Containers—Not Virtual Machines—Are the Future Cloud
- Non-Linux FOSS: libnotify, OS X Style
- Linux Systems Administrator
- Validate an E-Mail Address with PHP, the Right Way
- Lock-Free Multi-Producer Multi-Consumer Queue on Ring Buffer
- Senior Perl Developer
- Technical Support Rep
- UX Designer
- RSS Feeds
- Introduction to MapReduce with Hadoop on Linux
Free Webinar: Hadoop
How to Build an Optimal Hadoop Cluster to Store and Maintain Unlimited Amounts of Data Using Microservers
Realizing the promise of Apache® Hadoop® requires the effective deployment of compute, memory, storage and networking to achieve optimal results. With its flexibility and multitude of options, it is easy to over or under provision the server infrastructure, resulting in poor performance and high TCO. Join us for an in depth, technical discussion with industry experts from leading Hadoop and server companies who will provide insights into the key considerations for designing and deploying an optimal Hadoop cluster.
Some of key questions to be discussed are:
- What is the “typical” Hadoop cluster and what should be installed on the different machine types?
- Why should you consider the typical workload patterns when making your hardware decisions?
- Are all microservers created equal for Hadoop deployments?
- How do I plan for expansion if I require more compute, memory, storage or networking?