Home ›

SMART (Smart Monitoring and Rebooting Tool)

Issue 151

From Issue #151
November 2006

Sep 30, 2006  By Albert Martorell
 in
If you want an agent to monitor and control services, you'll need to get SMART.

Listing 2. sudo Configured for SMART Access

# Defaults specification
Defaults:root !syslog

# User privilege specification
root    ALL=(ALL) ALL
sysman  server=(root) NOPASSWD: /home/sysman/check-service
sysman  server=(root) NOPASSWD: /sbin/reboot

This way, we disable syslog logging when sudo is executed by user root, and we assign root privileges to user sysman, at the host server, only for the execution of commands /home/sysman/check-service and /sbin/reboot, without asking sysman for the password every time.

Verifications

Through the PID file defined in the configuration file, we obtain the parent process identifier (PID), and we determine the number of active processes generated by this service. Next we check whether:

  • The service is responding to petitions within the defined time period.

  • The number of processes generated by the service doesn't exceed the maximum and minimum defined thresholds.

Status Determination

Considering the results obtained in former verifications, we classify the service status:

  • 0: service is responding to requests within the defined time period, the number of processes generated by service remains between the defined thresholds, and the information provided by the PID file is correct.

  • 1: service is responding to requests within the defined time period and the number of processes generated by service remains between the defined thresholds, but either the information provided by the PID file is incorrect or this file doesn't exist, even though it has been defined.

  • 2: service is responding to requests within the defined time period, but the number of processes generated by the service is beyond the defined thresholds (this could be the case of an overloaded but operative Web server).

  • 3: the number of generated processes is out of thresholds, and we don't have any tool (script) to check whether the service is operative (this could be the case of processes such as syslogd, crond and xinetd).

  • 4: service is not responding to requests within the defined time period.

We group the above five situations in three more general cases:

  • OK (status 0 and 1).

  • WARN (status 2).

  • DOWN (status 3 and 4).

Decision

When executing the program with no parameters, it simply will determine the status of services defined in the configuration file and will display the results. If we want the program to work in an active way, we need to use some of the following parameters:

  • -w: restart services in WARN status and send a notification (e-mail) for each one of them.

  • -d: restart services in DOWN status and send a notification for each one of them.

  • -wd: restart services in WARN and DOWN status and send a notification for each one of them.

  • --all: restart all services independently of their status and send a notification for each service with WARN or DOWN status.

  • --reboot: restart the whole system independently of service's status and send a general notification.

Once the service status has been determined, and according to the parameter specified in the execution, the action carried out for each service will consist of that shown in Table 1.

Table 1. Service Actions

StatusParametersAction
OK--allRestart the service
WARN-w, -wd, --allRestart the service
  Send a notification relating to service
 -dSend a notification relating to service
DOWN-d, -wd, --allRestart the service
  Send a notification relating to service

Furthermore, independently of the service's status, with the parameters --all and --reboot, a notification via e-mail is sent to the administrator about the performed action.

Listing 3 shows a sample of SMART in action, executed from a console with parameter -d (recovery of services in DOWN status).

______________________

Webcast
More Resources
How to Build an Optimal Hadoop Cluster to Store and Maintain Unlimited Amounts of Data Using Microservers

Realizing the promise of Apache® Hadoop® requires the effective deployment of compute, memory, storage and networking to achieve optimal results. With its flexibility and multitude of options, it is easy to over or under provision the server infrastructure, resulting in poor performance and high TCO. Join us for an in depth, technical discussion with industry experts from leading Hadoop and server companies who will provide insights into the key considerations for designing and deploying an optimal Hadoop cluster.

Learn More

Sponsored by AMD

White Paper
More Resources
Red Hat White Paper: Using an Open Source Framework to Catch the Bad Guy

Built-in forensics, incident response, and security with Red Hat Enterprise Linux 6

Every security policy provides guidance and requirements for ensuring adequate protection of information and data, as well as high-level technical and administrative security requirements for a system in a given environment. Traditionally, providing security for a system focuses on the confidentiality of the information on it. However, protecting the data integrity and system and data availability is just as important. For example, when processing United States intelligence information, there are three attributes that require protection: confidentiality, integrity, and availability.

Learn more about catching the bad guy in this free white paper.

Learn More

Sponsored by DLT Solutions