SNMP Monitoring with Nagios

Using Nagios, you can monitor Dell servers with SNMP via Dell's server administration tools.
Testing

Because I did not have a spare Dell Power Edge server sitting around to test the modified script, I had to test it another way. Reading the man page for snmpd.conf, I found that you could have external programs answer certain OIDs using “pass-through” scripts. The bash script (dell_open_manager_test.sh) below serves as my pass-through script for testing. With this script, I can simulate all of the states that the Dell server could be in:

#!/bin/bash
#
# bash script to replicate a working Dell OpenManage SNMP agent
# works with Net-SNMP daemon. infotek@gmail.com
#

REQUEST_OID="$2"

echo "$REQUEST_OID";
case "$REQUEST_OID" in
  .1.3.6.1.4.1.674.10892.1.200.10.1.4.1)
    echo "integer"; echo "3"; exit 0 ;;
  .1.3.6.1.4.1.674.10892.1.200.10.1.9.1)
    echo "integer"; echo "5"; exit 0 ;;
  .1.3.6.1.4.1.674.10892.1.200.10.1.12.1)
    echo "integer"; echo "3"; exit 0 ;;
  .1.3.6.1.4.1.674.10892.1.200.10.1.21.1)
    echo "integer"; echo "4"; exit 0 ;;
  .1.3.6.1.4.1.674.10892.1.200.10.1.24.1)
    echo "integer"; echo "3"; exit 0 ;;
  .1.3.6.1.4.1.674.10892.1.200.10.1.27.1)
    echo "integer"; echo "3"; exit 0 ;;
  .1.3.6.1.4.1.674.10892.1.200.10.1.30.1)
    echo "integer"; echo "3"; exit 0 ;;
  .1.3.6.1.4.1.674.10892.1.200.10.1.41.1)
    echo "integer"; echo "3"; exit 0 ;;
  .1.3.6.1.4.1.674.10893.1.20.110.13.0)
    echo "integer"; echo "3"; exit 0 ;;
  *)
    echo "string"; echo "$@"; exit 0 ;;
esac
exit

To use the script, I added the following lines to the end of ./etc/snmp/snmpd.conf:

### dell open manager test
view systemview included .1.3.6.1.4.1.674
pass .1.3.6.1.4.1.674 /bin/bash \
              /usr/local/bin/dell_open_manager_test.sh

To make the changes in the configuration file take effect, restart the snmpd dæmon. On Slackware, this is done via the following:

# /etc/rc.d/rc.snmpd restart
Shutting down snmpd: . DONE
Starting snmpd: /usr/sbin/snmpd -A -p \
              /var/run/snmpd -a -c /etc/snmp/snmpd.conf

To query the SNMP server, we use Net-SNMP's command-line snmpget utility:

# snmpget -v 1 -c public 127.0.0.1 \
                .1.3.6.1.4.1.674.10892.1.200.10.1.9.1
SNMPv2-SMI::enterprises.674.10892.1.200.10.1.9.1 = INTEGER: 3

The response is an integer value of 3. The value 3 in the DellStatus (see above) maps to “ok(3) The object's status is OK”. This tells us that the pass-through script is working. Now, we test the /check_dell_openmanager.pl Perl script:

# ./check_dell_openmanager.pl -H 127.0.0.1 -C public -T pe2950
OK

To test other values, simply modify the dell_open_manager_test.sh shell script. For example, to simulate an error in the Cooling Device OID (.1.3.6.1.4.1.674.10892.1.200.10.1.21), modify that OID's line in the script to return a code of 4 for nonCritical:

.1.3.6.1.4.1.674.10892.1.200.10.1.21.1)
  echo "integer"; echo "4"; exit 0 ;;

Now, running the Perl script produces a warning:

# ./check_dell_openmanager.pl -H 127.0.0.1 -C public -T pe2950
WARNING:Cooling Device Status=Non-Critical

To simulate a critical error, let's modify the Power Supply OID to reply with a 5:

.1.3.6.1.4.1.674.10892.1.200.10.1.9.1)
  echo "integer"; echo "5"; exit 0 ;;


# ./check_dell_openmanager.pl -H 127.0.0.1 -C public -T pe2950
CRITICAL:Cooling Device Status=Non-Critical, \
                 Power Supply Status=Critical

To test the script on the live production systems, we added the check_dell_openmanager.pl command to a working Nagios server. We opened the case cover on a live system to generate a Chassis Intrusion Status error to test the plugin. Within a few seconds, we had an SMS message on the IT administrator's phone letting us know that there was a problem with the chassis subsystem on the server we just opened.

Moving Forward

After writing this plugin, I uploaded it to a Web site that hosts third-party addons for Nagios named Nagios Exchange. In short order, I was getting e-mail messages from all over the world concerning the Nagios plugin I had written. Some were suggestions, and some were from people in need of help. It was not an overwhelming number of messages. At most, two a week and sometimes none. It was just enough to let me know that people other than me actually were using this thing.

I would like to make a few improvements to the module. For one, I think there may be a way to reduce the SNMP queries to only one query to obtain the overall global status of the machine. Then, only if the state is not “ok(3)”, move to query the other OIDs so that a more specific error can be reported.

It also would be nice to be able to evaluate the existence of the various subsystems, that way, for example, if a machine has a RAID array, it is monitored, and if not, the script skips it.

One of the most common e-mail messages I get is about missing the Net::SNMP Perl module. I would like to test for these common-case scenarios. If the test fails, I would like to print the problem with a solution. In the case of “Net::SNMP”, it should print:

You are missing the Net::SNMP perl module.
Please install it using:
  perl -MCPAN -e shell
  cpan> install "Net::SNMP"

This would improve end-user experience significantly, especially for users new to Linux.

______________________

Comments

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

why reinvent the wheel :-)

natxo asenjo's picture

http://folk.uio.no/trondham/software/check_openmanage.html

check_openmanage does all you need and is (at this point) a better solution for nagios.

Wheel wasn't reinvented

Trond H. Amundsen's picture

I feel the need to comment on this. I am the author of the check_openmanage plugin. I know for a fact that Jason's plugin existed long before check_openmanage, so to be precise it was I who reinvented the wheel. Also, our two plugins are different in their focus, and I believe that both are needed. Users can choose whichever plugin they want, among these two and many others. Isn't open source great :)

Besides that, I really enjoyed Jason's article. It explains in a detailed and concise manner how one goes about to monitor something with SNMP, and how to integrate this with Nagios. This is universally useful to many out there.

Thanks for a great article, Jason!

-trond

MonitorSNMP is a free

Anonymous's picture

MonitorSNMP is a free monitoring service, basic but provides notification based on rules. Easy to setup and use. Take a look

White Paper
Linux Management with Red Hat Satellite: Measuring Business Impact and ROI

Linux has become a key foundation for supporting today's rapidly growing IT environments. Linux is being used to deploy business applications and databases, trading on its reputation as a low-cost operating environment. For many IT organizations, Linux is a mainstay for deploying Web servers and has evolved from handling basic file, print, and utility workloads to running mission-critical applications and databases, physically, virtually, and in the cloud. As Linux grows in importance in terms of value to the business, managing Linux environments to high standards of service quality — availability, security, and performance — becomes an essential requirement for business success.

Learn More

Sponsored by Red Hat

White Paper
Private PaaS for the Agile Enterprise

If you already use virtualized infrastructure, you are well on your way to leveraging the power of the cloud. Virtualization offers the promise of limitless resources, but how do you manage that scalability when your DevOps team doesn’t scale? In today’s hypercompetitive markets, fast results can make a difference between leading the pack vs. obsolescence. Organizations need more benefits from cloud computing than just raw resources. They need agility, flexibility, convenience, ROI, and control.

Stackato private Platform-as-a-Service technology from ActiveState extends your private cloud infrastructure by creating a private PaaS to provide on-demand availability, flexibility, control, and ultimately, faster time-to-market for your enterprise.

Learn More

Sponsored by ActiveState