SMART (Smart Monitoring and Rebooting Tool)
There are a lot of excellent monitoring tools (Big Brother, Nagios and so on), and some of them allow recovery from dead services, but with great complexity in their configuration, which becomes even more complicated when you want to supervise local services that are not remotely accessible, such as syslog, xinet, mrtg, iptables or Nagios itself.
The purpose with SMART was to have a simple, flexible and quick-to-implement application for monitoring the most critical system dæmons that made it possible to add new ones without modifying the code and to avoid installation and configuration complexities. It also needed to be capable of making decisions and solving problems (or at least trying to do that).
After a first version of “passive” monitoring, we tried to go a step further and obtain an “active” application, that is to say, to add the possibility of auto-recovery. By executing the application periodically through crond, it should detect dæmons that were down and boot them without the intervention of the system administrator.
Later, we considered the possibility that a nonprivileged user could execute this application from a console or remotely (via Telnet or SSH). Centralization of detection and error recovery in only one script made integration with sudo easier. Furthermore, it allowed delegating some stronger recovery actions needed in critical situations, such as rebooting the whole system, to this nonroot-privileged user.
With the ps command, we can list all the active processes in the system, but being “active” is not the same as being “operative”, so this led us to include the check scripts, which are small programs to test services and determine whether they really are operative and answering requests. The difficulties we found suggested that we not waste efforts re-inventing the wheel and profit from plugins included in Nagios (monitoring software that we were using satisfactorily for almost three years).
The distribution of SMART has two shell scripts (smart and check-service), two configuration files (host.conf and services.conf) and two directories (scripts and plugins), which contain the check scripts and the plugins (Listing 1).
Listing 1. The SMART Installation Files and Directories
[root@server /]# ls -la /home/sysman/ drwxr-x--- 4 root sysman 4096 May 27 11:49 . drwxr-xr-x 3 root root 4096 Jul 8 2003 .. -rwxr-x--- 1 root sysman 1448 May 27 11:51 smart -rwx------ 1 root root 7815 May 27 11:51 check-service -rw-r--r-- 1 root root 242 May 27 11:49 host.conf drwx------ 2 root root 4096 Apr 29 13:38 plugins drwx------ 2 root root 4096 Apr 29 13:39 scripts -rw-r--r-- 1 root root 883 May 17 10:40 services.conf
Permissions of files and directories allow a nonprivileged user called sysman to execute the application, but deny sysman the ability to modify the contents to use it in an inadequate way.
The SMART program reads the configuration files services.conf and host.conf and executes check-service for each defined service. If a check script has been assigned to a service, for example, services 1 and 2 in Figure 1, check-service will execute it, passing the needed parameters and then will wait for the exit status to determine whether the service is alive. If this check script executes some other external script (plugin), such as service 1 in Figure 1, this one will be responsible for checking the service status.
If no check script has been assigned to a service (service 3 in Figure 1), the check-service file will determine the service status by getting the number of active processes. According to this information, the SMART command-line parameters and the configuration parameters, it will decide what actions to carry out.
Integration with the sudo (superuser do) tool allows the system administrator to permit another user (sysman) to start dead services, restart all the services or reboot the whole system. Advantages of this are:
Simple configuration: there's no need to give privileges to that user to stop and start every service, and no need to use administrative tools (ps, kill, rm and so on). The check-service script centralizes the whole operation.
Security: user sysman can't read, write or execute the check-service file.
Easy to use: scripts are managed by sudo, so its usage will be transparent for the user.
For a user sysman, who needs privileges on the host server, the configuration file of sudo (/etc/sudoers) should be as shown in Listing 2.
Free Webinar: Hadoop
How to Build an Optimal Hadoop Cluster to Store and Maintain Unlimited Amounts of Data Using Microservers
Realizing the promise of Apache® Hadoop® requires the effective deployment of compute, memory, storage and networking to achieve optimal results. With its flexibility and multitude of options, it is easy to over or under provision the server infrastructure, resulting in poor performance and high TCO. Join us for an in depth, technical discussion with industry experts from leading Hadoop and server companies who will provide insights into the key considerations for designing and deploying an optimal Hadoop cluster.
Some of key questions to be discussed are:
- What is the “typical” Hadoop cluster and what should be installed on the different machine types?
- Why should you consider the typical workload patterns when making your hardware decisions?
- Are all microservers created equal for Hadoop deployments?
- How do I plan for expansion if I require more compute, memory, storage or networking?
|Using Salt Stack and Vagrant for Drupal Development||May 20, 2013|
|Making Linux and Android Get Along (It's Not as Hard as It Sounds)||May 16, 2013|
|Drupal Is a Framework: Why Everyone Needs to Understand This||May 15, 2013|
|Home, My Backup Data Center||May 13, 2013|
|Non-Linux FOSS: Seashore||May 10, 2013|
|Trying to Tame the Tablet||May 08, 2013|
- RSS Feeds
- Making Linux and Android Get Along (It's Not as Hard as It Sounds)
- Using Salt Stack and Vagrant for Drupal Development
- New Products
- Validate an E-Mail Address with PHP, the Right Way
- Drupal Is a Framework: Why Everyone Needs to Understand This
- A Topic for Discussion - Open Source Feature-Richness?
- Download the Free Red Hat White Paper "Using an Open Source Framework to Catch the Bad Guy"
- Tech Tip: Really Simple HTTP Server with Python
- Readers' Choice Awards
- Android is Linux -- why no better inter-operation
2 hours 13 min ago
- Connecting Android device to desktop Linux via USB
2 hours 42 min ago
- Find new cell phone and tablet pc
3 hours 40 min ago
5 hours 9 min ago
- Automatically updating Guest Additions
6 hours 17 min ago
- I like your topic on android
7 hours 4 min ago
- Reply to comment | Linux Journal
7 hours 25 min ago
- This is the easiest tutorial
13 hours 40 min ago
- Ahh, the Koolaid.
19 hours 18 min ago
- git-annex assistant
1 day 1 hour ago