Custom checks and notifications for Nagios

A while back, I wrote an article for Linux Journal's web edition entitled “Howto be a good (and lazy) System Administrator.” A couple astute readers, after reading the article, asked if I was familiar with the Nagios monitoring system, and I am. I've been using Nagios for a few years now.

I had intended to write this article as a How-to on getting Nagios configured and running for the first time. However, it turns out that the documentation that comes with Nagios is really pretty good. And even if you do have problems, and I did, the user community is also quite responsive. So, rather than beating a dead horse, (with sympathy to horse lovers) I decided to continue the Good and Lazy Administrator Theme and discuss extending Nagios with custom service checks and custom notifications.

Nagios uses a plug-in mechanism to implement all of it's server and service checks as well as all of it's notifications. This is good news for hackers, as it allows us to build new functionality that either no one else has though of, or has need of. I wrote a couple scripts for my Nagios system. One does a custom service check to see if I have voicemail waiting for me at the Help Desk, and the other does a custom notification by telephone. Before I go on, I should give a little bit of background.

I maintain several servers, both for myself and for customers. These servers range from web servers to phone systems running Asterisk. Just like most System Administrators, I don't have any “optional” servers or services; the stuff just has to work, and when it isn't, I need to know. But I'll tell you, I'm not interested in sitting at the desk watching the Help Desk phone or the monitoring screen. I'm either too lazy, or too busy. Either way, that's what silicon is for, right?

My phone system at the house runs on Asterisk. You can read more about my home infrastructure at http://www.linuxjournal.com/article/9111. My Nagios server runs on the same server, so it just makes sense to integrate the two services.

I've created a Nagios script that monitors the Help Desk voicemailbox and sets a service alert if there are any critical alerts in Nagios. I've also written a script that can call me, perhaps on my cellphone, in the event of a service outage. With these two scripts in place, I can get a call on my cellphone any time someone calls my Help Desk and leaves a message. I can also get a call if any of my monitored services fail. Theoretically, I can be at a park playing with my boys and know that my servers are happy... until the cellphone rings.

I understand that I have kind of a unique situation, but the same concept is applicable in a business production environment, so lets get down to looking at code.

First, let's talk about the Help Desk monitoring script. Essentially, this script checks to see if there are any files in the INBOX in the Help Desk mailbox. Here is the code:

#!/usr/bin/perl -w

local *DIR;
my ($file, $error);

$error = 0;

opendir DIR, "/var/spool/asterisk/voicemail/customers/611/INBOX/"
       or die("Error: Permission denied\n");

while ($file = readdir(DIR)) {
       if ($file eq ".") { next; }
       if ($file eq "..") { next; }

       $error++;
}

$error = $error/4;

if (!$error) {
       print "OK\n";
       exit 0;
} else {
       print "CRITICAL: $error\n";
}

exit 2;

Of course, you need to make sure that the Nagios user has access to the Asterisk voicemailbox, but that can be taken care of by setting the script set-uid. The script, as you can see, is pretty simple. If there are any other files in the directory, the script assumes that there is a voicemail and sends a CRITICAL alert to Nagios. Otherwise everything is OK.

To enable Nagios to use this check script, we need to define it in checkcommands.cfg. Here is the definition, I used:

define command{
       command_name    check_help
       command_line    /etc/nagios/local/check_611.pl
}

Now, I can refer to the check_help check script in the services.cfg file. Here's how I did it:

define service {
       use generic-service
       name                    Help_Desk
       host_name               my_server
       service_description     Help Desk Voicemail
       check_command           check_help
       register 1
}

With this configuration in place, Nagios can indicate an alarm any time there is voicemail in the Help Desk mailbox. But that's only half of what I promised to write about. The next script allows Nagios to call me to let me know that I've got a fire to put out. Here is that script:

#!/usr/bin/perl

foreach $main::phone ("15055551234") {

       $main::call = <
MaxRetries: 0
RetryTime: 1
WaitTime: 120
Account: Enterprise
Context: apps
Extension: OUTAGE
Priority: 1
EOF
;

       open FILE, ">/tmp/outage.call";
       print FILE $main::call;
       close FILE;

       system("mv /tmp/outage.call /var/spool/asterisk/outgoing");
}

As you can see, this script isn't complicated, either. It simply creates an Asterisk “call file” and puts it in Asterisk's outgoing spool directory. The script is capable of calling multiple numbers... just in case. It's important to that the call file be created in another directory and moved into the spool directory. Otherwise bad things can happen if Asterisk tries to read the file while the script is still writing it.

Obviously this script relies on some configuration in the Asterisk dial plan. Here is the relevant part of the dial plan:

exten => OUTAGE,1,answer
exten => OUTAGE,2,playback(/etc/asterisk/sounds/OUTAGE)
exten => OUTAGE,3,hangup

At this point, you're probably realizing that I'm not doing anything complicated. All that is needed from Asterisk's point of view is an audio message in /etc/asterisk/sounds/OUTAGE (.wav or .au) that indicates that something is on fire. Asterisk will select the most reasonable file extension and play the file when the call is answered.

So all that is left to do is configure Nagios to use this notification method. This is configured in the misccommands.cfg file. Here is how I did it:

# 'notify_by_phone' command definition
define command{
       command_name    notify_by_phone
       command_line    /etc/nagios/local/notify_by_phone.pl
}

Now that all of the configuration is done, we restart Nagios and reload the Asterisk dial plan. To do this, we type “/etc/init.d/nagios restart” at the command line and “extensions reload” at the Asterisk console.

So now, anytime I have voicemail at the Help Desk, it's indicated in the Nagios monitoring screen as a critical alert. Also, anytime any of my servers or services are unavailable, I can get a phone call on either my home phone or my cell phone. This means that my customers don't HAVE to have those phone numbers and I can still provide quality service to them.

Now I realize that I have a unique situation, but I hope that this article serves as an example of how to create custom Nagios service checks and notifications, as well as hinting at some of the integration options available in Asterisk.

______________________

Mike Diehl is a freelance Computer Nerd specializing in Linux administration, programing, and VoIP. Mike lives in Albuquerque, NM. with his wife and 3 sons. He can be reached at mdiehl@diehlnet.com

Comments

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

Nagios Checks

Anonymous's picture

Hi Mike, great post!
I´m working with nagios since several months and find it great!
Now I´m trying to write my own check scripts, but mostly I´m adapting existing ones.
I´m not quite sure how to set the ok/warning/critical status on the script output, or simply how to inform it to nagios...
I´ve looked into your asterix example but it´s confusing to me...
The status is set by the $error variable?? or just by the text you printed at the end?
I hope you can help me on this, send a guide, an example or just point me the right direction.
Thanks in advance for your time.
Facundo

Gr8

mike...'s picture

This article is gr8.
Thanks alot!

Fish Hatchery

FredR's picture

At an ISP I worked for, my manager, a good friend, would often say "Someday I'm going to quit and go work for a fish hatchery". So I thought it would be funny if I wrote a nagios plugin to page all the techs and admins that "CRITICAL: FISH HATCHERY IS DOWN". It actually didn't do anything, just complain, and obviously produced several laughs.

-- FLR or flrichar is a superfan of Linux Journal, and goofs around in the LJ IRC Channel

In case of problems like

Dieter_be's picture

In case of problems like "Error: Permission denied" it can also be useful to use an appropriate exit code. such as 3 (UNKNOWN) or even 2 (CRITICAL)

White Paper
Linux Management with Red Hat Satellite: Measuring Business Impact and ROI

Linux has become a key foundation for supporting today's rapidly growing IT environments. Linux is being used to deploy business applications and databases, trading on its reputation as a low-cost operating environment. For many IT organizations, Linux is a mainstay for deploying Web servers and has evolved from handling basic file, print, and utility workloads to running mission-critical applications and databases, physically, virtually, and in the cloud. As Linux grows in importance in terms of value to the business, managing Linux environments to high standards of service quality — availability, security, and performance — becomes an essential requirement for business success.

Learn More

Sponsored by Red Hat

White Paper
Private PaaS for the Agile Enterprise

If you already use virtualized infrastructure, you are well on your way to leveraging the power of the cloud. Virtualization offers the promise of limitless resources, but how do you manage that scalability when your DevOps team doesn’t scale? In today’s hypercompetitive markets, fast results can make a difference between leading the pack vs. obsolescence. Organizations need more benefits from cloud computing than just raw resources. They need agility, flexibility, convenience, ROI, and control.

Stackato private Platform-as-a-Service technology from ActiveState extends your private cloud infrastructure by creating a private PaaS to provide on-demand availability, flexibility, control, and ultimately, faster time-to-market for your enterprise.

Learn More

Sponsored by ActiveState