SMART (Smart Monitoring and Rebooting Tool)

If you want an agent to monitor and control services, you'll need to get SMART.
Check Scripts

There are some optional executables files, the check scripts, responsible for checking whether the monitored services really are operative and responding to petitions. These files are written in Shell (.sh extension) and Expect (.exp extension). Expect is a tool that requires Tcl and allows for automation of interactive applications that use textual representation.

These scripts could be written in any programming language, because only the exit status is taken into account. If it's not equal to 0, we suppose that there has been no answer or that the answer given by the service has not been the expected one. This means that a check script not only can monitor services, but it also can achieve any check that returns a Boolean value, for example, to check whether the size of a directory exceeds a certain value, whether the amount of logged users is greater than a desired number, whether a kernel module is loaded and so on (Listing 4).

Files with the .nag extension are also Shell scripts, but unlike the former ones, they call an external program (plugin) passing to it the parameters received from check-service, following the order and format that the plugin expects. This checks the service and returns the information gathered to the check script, which will interpret and convert it into the exit status that check-service is waiting for (Listing 5).

Plugins are programmed in C, Perl and Shell and belong to Nagios. Their sources can be downloaded independently of the Nagios distribution, and some of them require the additional installation of certain programs and libraries.

Installation, Configuration and Usage

Software requirements include the following:

  • sudo: allows a user to execute a command as another user. This will be necessary if you are planning to allow a nonroot user to execute SMART.

  • awk: a pattern scanning and processing language. SMART uses it and expects to find it at /bin/awk. If that's not your case, edit the check-service and smart files of the SMART distribution and modify the line where AWK=“/bin/awk” is specified.

  • Nagios plugins: sources can be downloaded independently of the Nagios distribution, and some of them require the additional installation of certain programs and libraries. You can use the plugins distributed with SMART or download the newest ones.

  • Some shell scripts (in the scripts directory of SMART) may require some specific commands to check some services, such as dig for dns, wget for Web services, nmblookup for nmbd (Samba), ntpq for NTP, ldapsearch for OpenLDAP and so on. The paths of these commands are defined in a variable at the beginning of each script, so you can change their location, use any other command that might work better for your system or even rewrite the whole script at your convenience.

With sudo you can permit another user to run SMART. If you're not interested in creating such a user, you can omit steps 1, 2 and 3 below.

  1. Create user sysman and group sysman.

  2. Create the SMART directory. It's a good idea to install it at sysman home and to set the appropriate owner and permissions:

    mkdir /home/sysman
    chown root:sysman /home/sysman
    chmod 750 /home/sysman
    

  3. Edit the sudo configuration file /etc/sudoers, and add the following lines:

    ...
    sysman  hostname=(root) NOPASSWD: /home/sysman/check-service
    sysman  hostname=(root) NOPASSWD: /sbin/reboot
    

  4. Download the SMART software.

  5. Untar and unzip the distribution:

    tar -zxf smart-X.Y.tar.gz
    

  6. Go to the distribution directory and copy the files to the destination directory. If you choose a destination different from /home/sysman, you will have to edit the smart file and modify the line where dir=“/home/sysman” is specified:

    cd smart-X.
    cp check-service /home/sysman/
    cp smart /home/sysman/
    cp host.conf.dist /home/sysman/host.conf
    cp services.conf.dist /home/sysman/services.conf
    cp -r scripts /home/sysman/
    cp -r plugins /home/sysman/
    

  7. Go to the destination directory, and check/set file permissions and owners:

    cd /home/sysman
    chown -R root:root check-service scripts plugins host.conf services.conf
    chown root:sysman smart
    chmod -R 700 check-service scripts plugins
    chmod 750 smart
    chmod 644 host.conf services.conf
    

Configuration is as follows. First, edit the SMART host configuration file host.conf, and modify it according to your preferences (hostname, mail addresses, commands paths and so on). Then, edit the SMART services configuration file services.conf, and uncomment/modify/add any service/dæmon you want to check. Every line describes one service, with the following semicolon-separated parameters:

  • NAME (non-empty string): descriptive service name (for example, IMAP).

  • process_name[:port] (non-empty string[:integer]): parent process name and its operational port (for example, couriertcpd:143).

  • process_param (string): parameters of running process. Some services run with the same process name, so parameters are useful to distinguish them. For example, the parent process of Courier IMAP and POP3 is couriertcpd, but one is executed with the parameter pop3d and the other one with imapd.

  • max_procs (non-empty integer): the highest number of running processes allowed (for example, 10). Leave it at 0 if what you're monitoring runs no processes (for example, disk space).

  • min_procs (non-empty integer): the lowest number of running processes allowed (for example, 1). Leave it at 0 if what you're monitoring runs no processes (for example, disk space).

  • start_command (string): the command to start the service or script to be executed when the service is down (for example, /courier/libexec/imapd.rc).

  • pid_file (string): pid file path (for example, /var/run/imapd.pid).

  • sock_file (string): socket file path.

  • start_mode (0/1): the service can be started/stopped by adding start/stop to the start command (1), or it may not be necessary (0).

  • check_script (string): the name of the script used to check the service. This script has to be in the scripts directory (for example, imap.nag).

Leave the parameters empty if they are not applicable, except NAME, process_name, max_procs, min_procs and start_mode, which can't be empty.

Now, you should be able to run SMART as user root or sysman:

/home/sysman/smart

Try using -h to get more information about available parameters. Running SMART through crond might be a good thing. You can run it as frequently as you want, but doing it every five minutes seems to be reasonable enough.

______________________

Webinar
One Click, Universal Protection: Implementing Centralized Security Policies on Linux Systems

As Linux continues to play an ever increasing role in corporate data centers and institutions, ensuring the integrity and protection of these systems must be a priority. With 60% of the world's websites and an increasing share of organization's mission-critical workloads running on Linux, failing to stop malware and other advanced threats on Linux can increasingly impact an organization's reputation and bottom line.

Learn More

Sponsored by Bit9

Webinar
Linux Backup and Recovery Webinar

Most companies incorporate backup procedures for critical data, which can be restored quickly if a loss occurs. However, fewer companies are prepared for catastrophic system failures, in which they lose all data, the entire operating system, applications, settings, patches and more, reducing their system(s) to “bare metal.” After all, before data can be restored to a system, there must be a system to restore it to.

In this one hour webinar, learn how to enhance your existing backup strategies for better disaster recovery preparedness using Storix System Backup Administrator (SBAdmin), a highly flexible bare-metal recovery solution for UNIX and Linux systems.

Learn More

Sponsored by Storix