SNMP Monitoring with Nagios

Because I did not have a spare Dell Power Edge server sitting around to test the modified script, I had to test it another way. Reading the man page for snmpd.conf, I found that you could have external programs answer certain OIDs using “pass-through” scripts. The bash script (dell_open_manager_test.sh) below serves as my pass-through script for testing. With this script, I can simulate all of the states that the Dell server could be in:

#!/bin/bash
#
# bash script to replicate a working Dell OpenManage SNMP agent
# works with Net-SNMP daemon. infotek@gmail.com
#

REQUEST_OID="$2"

echo "$REQUEST_OID";
case "$REQUEST_OID" in
  .1.3.6.1.4.1.674.10892.1.200.10.1.4.1)
    echo "integer"; echo "3"; exit 0 ;;
  .1.3.6.1.4.1.674.10892.1.200.10.1.9.1)
    echo "integer"; echo "5"; exit 0 ;;
  .1.3.6.1.4.1.674.10892.1.200.10.1.12.1)
    echo "integer"; echo "3"; exit 0 ;;
  .1.3.6.1.4.1.674.10892.1.200.10.1.21.1)
    echo "integer"; echo "4"; exit 0 ;;
  .1.3.6.1.4.1.674.10892.1.200.10.1.24.1)
    echo "integer"; echo "3"; exit 0 ;;
  .1.3.6.1.4.1.674.10892.1.200.10.1.27.1)
    echo "integer"; echo "3"; exit 0 ;;
  .1.3.6.1.4.1.674.10892.1.200.10.1.30.1)
    echo "integer"; echo "3"; exit 0 ;;
  .1.3.6.1.4.1.674.10892.1.200.10.1.41.1)
    echo "integer"; echo "3"; exit 0 ;;
  .1.3.6.1.4.1.674.10893.1.20.110.13.0)
    echo "integer"; echo "3"; exit 0 ;;
  *)
    echo "string"; echo "$@"; exit 0 ;;
esac
exit

To use the script, I added the following lines to the end of ./etc/snmp/snmpd.conf:

### dell open manager test
view systemview included .1.3.6.1.4.1.674
pass .1.3.6.1.4.1.674 /bin/bash \
              /usr/local/bin/dell_open_manager_test.sh

To make the changes in the configuration file take effect, restart the snmpd dæmon. On Slackware, this is done via the following:

# /etc/rc.d/rc.snmpd restart
Shutting down snmpd: . DONE
Starting snmpd: /usr/sbin/snmpd -A -p \
              /var/run/snmpd -a -c /etc/snmp/snmpd.conf

To query the SNMP server, we use Net-SNMP's command-line snmpget utility:

# snmpget -v 1 -c public 127.0.0.1 \
                .1.3.6.1.4.1.674.10892.1.200.10.1.9.1
SNMPv2-SMI::enterprises.674.10892.1.200.10.1.9.1 = INTEGER: 3

The response is an integer value of 3. The value 3 in the DellStatus (see above) maps to “ok(3) The object's status is OK”. This tells us that the pass-through script is working. Now, we test the /check_dell_openmanager.pl Perl script:

# ./check_dell_openmanager.pl -H 127.0.0.1 -C public -T pe2950
OK

To test other values, simply modify the dell_open_manager_test.sh shell script. For example, to simulate an error in the Cooling Device OID (.1.3.6.1.4.1.674.10892.1.200.10.1.21), modify that OID's line in the script to return a code of 4 for nonCritical:

.1.3.6.1.4.1.674.10892.1.200.10.1.21.1)
  echo "integer"; echo "4"; exit 0 ;;

Now, running the Perl script produces a warning:

# ./check_dell_openmanager.pl -H 127.0.0.1 -C public -T pe2950
WARNING:Cooling Device Status=Non-Critical

To simulate a critical error, let's modify the Power Supply OID to reply with a 5:

.1.3.6.1.4.1.674.10892.1.200.10.1.9.1)
  echo "integer"; echo "5"; exit 0 ;;


# ./check_dell_openmanager.pl -H 127.0.0.1 -C public -T pe2950
CRITICAL:Cooling Device Status=Non-Critical, \
                 Power Supply Status=Critical

To test the script on the live production systems, we added the check_dell_openmanager.pl command to a working Nagios server. We opened the case cover on a live system to generate a Chassis Intrusion Status error to test the plugin. Within a few seconds, we had an SMS message on the IT administrator's phone letting us know that there was a problem with the chassis subsystem on the server we just opened.

After writing this plugin, I uploaded it to a Web site that hosts third-party addons for Nagios named Nagios Exchange. In short order, I was getting e-mail messages from all over the world concerning the Nagios plugin I had written. Some were suggestions, and some were from people in need of help. It was not an overwhelming number of messages. At most, two a week and sometimes none. It was just enough to let me know that people other than me actually were using this thing.

I would like to make a few improvements to the module. For one, I think there may be a way to reduce the SNMP queries to only one query to obtain the overall global status of the machine. Then, only if the state is not “ok(3)”, move to query the other OIDs so that a more specific error can be reported.

It also would be nice to be able to evaluate the existence of the various subsystems, that way, for example, if a machine has a RAID array, it is monitored, and if not, the script skips it.

One of the most common e-mail messages I get is about missing the Net::SNMP Perl module. I would like to test for these common-case scenarios. If the test fails, I would like to print the problem with a solution. In the case of “Net::SNMP”, it should print:

You are missing the Net::SNMP perl module.
Please install it using:
  perl -MCPAN -e shell
  cpan> install "Net::SNMP"

This would improve end-user experience significantly, especially for users new to Linux.