SMART (Smart Monitoring and Rebooting Tool)

If you want an agent to monitor and control services, you'll need to get SMART.
Check Scripts

There are some optional executables files, the check scripts, responsible for checking whether the monitored services really are operative and responding to petitions. These files are written in Shell (.sh extension) and Expect (.exp extension). Expect is a tool that requires Tcl and allows for automation of interactive applications that use textual representation.

These scripts could be written in any programming language, because only the exit status is taken into account. If it's not equal to 0, we suppose that there has been no answer or that the answer given by the service has not been the expected one. This means that a check script not only can monitor services, but it also can achieve any check that returns a Boolean value, for example, to check whether the size of a directory exceeds a certain value, whether the amount of logged users is greater than a desired number, whether a kernel module is loaded and so on (Listing 4).

Files with the .nag extension are also Shell scripts, but unlike the former ones, they call an external program (plugin) passing to it the parameters received from check-service, following the order and format that the plugin expects. This checks the service and returns the information gathered to the check script, which will interpret and convert it into the exit status that check-service is waiting for (Listing 5).

Plugins are programmed in C, Perl and Shell and belong to Nagios. Their sources can be downloaded independently of the Nagios distribution, and some of them require the additional installation of certain programs and libraries.

Installation, Configuration and Usage

Software requirements include the following:

  • sudo: allows a user to execute a command as another user. This will be necessary if you are planning to allow a nonroot user to execute SMART.

  • awk: a pattern scanning and processing language. SMART uses it and expects to find it at /bin/awk. If that's not your case, edit the check-service and smart files of the SMART distribution and modify the line where AWK=“/bin/awk” is specified.

  • Nagios plugins: sources can be downloaded independently of the Nagios distribution, and some of them require the additional installation of certain programs and libraries. You can use the plugins distributed with SMART or download the newest ones.

  • Some shell scripts (in the scripts directory of SMART) may require some specific commands to check some services, such as dig for dns, wget for Web services, nmblookup for nmbd (Samba), ntpq for NTP, ldapsearch for OpenLDAP and so on. The paths of these commands are defined in a variable at the beginning of each script, so you can change their location, use any other command that might work better for your system or even rewrite the whole script at your convenience.

With sudo you can permit another user to run SMART. If you're not interested in creating such a user, you can omit steps 1, 2 and 3 below.

  1. Create user sysman and group sysman.

  2. Create the SMART directory. It's a good idea to install it at sysman home and to set the appropriate owner and permissions:

    mkdir /home/sysman
    chown root:sysman /home/sysman
    chmod 750 /home/sysman
    

  3. Edit the sudo configuration file /etc/sudoers, and add the following lines:

    ...
    sysman  hostname=(root) NOPASSWD: /home/sysman/check-service
    sysman  hostname=(root) NOPASSWD: /sbin/reboot
    

  4. Download the SMART software.

  5. Untar and unzip the distribution:

    tar -zxf smart-X.Y.tar.gz
    

  6. Go to the distribution directory and copy the files to the destination directory. If you choose a destination different from /home/sysman, you will have to edit the smart file and modify the line where dir=“/home/sysman” is specified:

    cd smart-X.
    cp check-service /home/sysman/
    cp smart /home/sysman/
    cp host.conf.dist /home/sysman/host.conf
    cp services.conf.dist /home/sysman/services.conf
    cp -r scripts /home/sysman/
    cp -r plugins /home/sysman/
    

  7. Go to the destination directory, and check/set file permissions and owners:

    cd /home/sysman
    chown -R root:root check-service scripts plugins host.conf services.conf
    chown root:sysman smart
    chmod -R 700 check-service scripts plugins
    chmod 750 smart
    chmod 644 host.conf services.conf
    

Configuration is as follows. First, edit the SMART host configuration file host.conf, and modify it according to your preferences (hostname, mail addresses, commands paths and so on). Then, edit the SMART services configuration file services.conf, and uncomment/modify/add any service/dæmon you want to check. Every line describes one service, with the following semicolon-separated parameters:

  • NAME (non-empty string): descriptive service name (for example, IMAP).

  • process_name[:port] (non-empty string[:integer]): parent process name and its operational port (for example, couriertcpd:143).

  • process_param (string): parameters of running process. Some services run with the same process name, so parameters are useful to distinguish them. For example, the parent process of Courier IMAP and POP3 is couriertcpd, but one is executed with the parameter pop3d and the other one with imapd.

  • max_procs (non-empty integer): the highest number of running processes allowed (for example, 10). Leave it at 0 if what you're monitoring runs no processes (for example, disk space).

  • min_procs (non-empty integer): the lowest number of running processes allowed (for example, 1). Leave it at 0 if what you're monitoring runs no processes (for example, disk space).

  • start_command (string): the command to start the service or script to be executed when the service is down (for example, /courier/libexec/imapd.rc).

  • pid_file (string): pid file path (for example, /var/run/imapd.pid).

  • sock_file (string): socket file path.

  • start_mode (0/1): the service can be started/stopped by adding start/stop to the start command (1), or it may not be necessary (0).

  • check_script (string): the name of the script used to check the service. This script has to be in the scripts directory (for example, imap.nag).

Leave the parameters empty if they are not applicable, except NAME, process_name, max_procs, min_procs and start_mode, which can't be empty.

Now, you should be able to run SMART as user root or sysman:

/home/sysman/smart

Try using -h to get more information about available parameters. Running SMART through crond might be a good thing. You can run it as frequently as you want, but doing it every five minutes seems to be reasonable enough.

______________________

White Paper
Linux Management with Red Hat Satellite: Measuring Business Impact and ROI

Linux has become a key foundation for supporting today's rapidly growing IT environments. Linux is being used to deploy business applications and databases, trading on its reputation as a low-cost operating environment. For many IT organizations, Linux is a mainstay for deploying Web servers and has evolved from handling basic file, print, and utility workloads to running mission-critical applications and databases, physically, virtually, and in the cloud. As Linux grows in importance in terms of value to the business, managing Linux environments to high standards of service quality — availability, security, and performance — becomes an essential requirement for business success.

Learn More

Sponsored by Red Hat

White Paper
Private PaaS for the Agile Enterprise

If you already use virtualized infrastructure, you are well on your way to leveraging the power of the cloud. Virtualization offers the promise of limitless resources, but how do you manage that scalability when your DevOps team doesn’t scale? In today’s hypercompetitive markets, fast results can make a difference between leading the pack vs. obsolescence. Organizations need more benefits from cloud computing than just raw resources. They need agility, flexibility, convenience, ROI, and control.

Stackato private Platform-as-a-Service technology from ActiveState extends your private cloud infrastructure by creating a private PaaS to provide on-demand availability, flexibility, control, and ultimately, faster time-to-market for your enterprise.

Learn More

Sponsored by ActiveState