Custom checks and notifications for Nagios

A while back, I wrote an article for Linux Journal's web edition entitled “Howto be a good (and lazy) System Administrator.” A couple astute readers, after reading the article, asked if I was familiar with the Nagios monitoring system, and I am. I've been using Nagios for a few years now.

I had intended to write this article as a How-to on getting Nagios configured and running for the first time. However, it turns out that the documentation that comes with Nagios is really pretty good. And even if you do have problems, and I did, the user community is also quite responsive. So, rather than beating a dead horse, (with sympathy to horse lovers) I decided to continue the Good and Lazy Administrator Theme and discuss extending Nagios with custom service checks and custom notifications.

Nagios uses a plug-in mechanism to implement all of it's server and service checks as well as all of it's notifications. This is good news for hackers, as it allows us to build new functionality that either no one else has though of, or has need of. I wrote a couple scripts for my Nagios system. One does a custom service check to see if I have voicemail waiting for me at the Help Desk, and the other does a custom notification by telephone. Before I go on, I should give a little bit of background.

I maintain several servers, both for myself and for customers. These servers range from web servers to phone systems running Asterisk. Just like most System Administrators, I don't have any “optional” servers or services; the stuff just has to work, and when it isn't, I need to know. But I'll tell you, I'm not interested in sitting at the desk watching the Help Desk phone or the monitoring screen. I'm either too lazy, or too busy. Either way, that's what silicon is for, right?

My phone system at the house runs on Asterisk. You can read more about my home infrastructure at http://www.linuxjournal.com/article/9111. My Nagios server runs on the same server, so it just makes sense to integrate the two services.

I've created a Nagios script that monitors the Help Desk voicemailbox and sets a service alert if there are any critical alerts in Nagios. I've also written a script that can call me, perhaps on my cellphone, in the event of a service outage. With these two scripts in place, I can get a call on my cellphone any time someone calls my Help Desk and leaves a message. I can also get a call if any of my monitored services fail. Theoretically, I can be at a park playing with my boys and know that my servers are happy... until the cellphone rings.

I understand that I have kind of a unique situation, but the same concept is applicable in a business production environment, so lets get down to looking at code.

First, let's talk about the Help Desk monitoring script. Essentially, this script checks to see if there are any files in the INBOX in the Help Desk mailbox. Here is the code:

#!/usr/bin/perl -w

local *DIR;
my ($file, $error);

$error = 0;

opendir DIR, "/var/spool/asterisk/voicemail/customers/611/INBOX/"
       or die("Error: Permission denied\n");

while ($file = readdir(DIR)) {
       if ($file eq ".") { next; }
       if ($file eq "..") { next; }

       $error++;
}

$error = $error/4;

if (!$error) {
       print "OK\n";
       exit 0;
} else {
       print "CRITICAL: $error\n";
}

exit 2;

Of course, you need to make sure that the Nagios user has access to the Asterisk voicemailbox, but that can be taken care of by setting the script set-uid. The script, as you can see, is pretty simple. If there are any other files in the directory, the script assumes that there is a voicemail and sends a CRITICAL alert to Nagios. Otherwise everything is OK.

To enable Nagios to use this check script, we need to define it in checkcommands.cfg. Here is the definition, I used:

define command{
       command_name    check_help
       command_line    /etc/nagios/local/check_611.pl
}

Now, I can refer to the check_help check script in the services.cfg file. Here's how I did it:

define service {
       use generic-service
       name                    Help_Desk
       host_name               my_server
       service_description     Help Desk Voicemail
       check_command           check_help
       register 1
}

With this configuration in place, Nagios can indicate an alarm any time there is voicemail in the Help Desk mailbox. But that's only half of what I promised to write about. The next script allows Nagios to call me to let me know that I've got a fire to put out. Here is that script:

#!/usr/bin/perl

foreach $main::phone ("15055551234") {

       $main::call = <
MaxRetries: 0
RetryTime: 1
WaitTime: 120
Account: Enterprise
Context: apps
Extension: OUTAGE
Priority: 1
EOF
;

       open FILE, ">/tmp/outage.call";
       print FILE $main::call;
       close FILE;

       system("mv /tmp/outage.call /var/spool/asterisk/outgoing");
}

As you can see, this script isn't complicated, either. It simply creates an Asterisk “call file” and puts it in Asterisk's outgoing spool directory. The script is capable of calling multiple numbers... just in case. It's important to that the call file be created in another directory and moved into the spool directory. Otherwise bad things can happen if Asterisk tries to read the file while the script is still writing it.

Obviously this script relies on some configuration in the Asterisk dial plan. Here is the relevant part of the dial plan:

exten => OUTAGE,1,answer
exten => OUTAGE,2,playback(/etc/asterisk/sounds/OUTAGE)
exten => OUTAGE,3,hangup

At this point, you're probably realizing that I'm not doing anything complicated. All that is needed from Asterisk's point of view is an audio message in /etc/asterisk/sounds/OUTAGE (.wav or .au) that indicates that something is on fire. Asterisk will select the most reasonable file extension and play the file when the call is answered.

So all that is left to do is configure Nagios to use this notification method. This is configured in the misccommands.cfg file. Here is how I did it:

# 'notify_by_phone' command definition
define command{
       command_name    notify_by_phone
       command_line    /etc/nagios/local/notify_by_phone.pl
}

Now that all of the configuration is done, we restart Nagios and reload the Asterisk dial plan. To do this, we type “/etc/init.d/nagios restart” at the command line and “extensions reload” at the Asterisk console.

So now, anytime I have voicemail at the Help Desk, it's indicated in the Nagios monitoring screen as a critical alert. Also, anytime any of my servers or services are unavailable, I can get a phone call on either my home phone or my cell phone. This means that my customers don't HAVE to have those phone numbers and I can still provide quality service to them.

Now I realize that I have a unique situation, but I hope that this article serves as an example of how to create custom Nagios service checks and notifications, as well as hinting at some of the integration options available in Asterisk.