Are Your Web Services Working Correctly?

How to use Linux, Perl and other free software to check your web services around the clock.

In December of 1999, I started to work for an Italian firm as a system administrator. The firm, Infogroup S.p.A., is a provider of banking and financial applications for Italian banks. Its core business is to provide web portals and data sharing for web-trading systems. We get data from the satellite, put it in the databases and present it to internet customers for trading via a web application.

One of the most critical aspects of these activities is time. While not as important for the bank account holder, for those who invest money in the stock markets, time is a primary concern. Therefore, if our web applications hang for seconds it's a big problem. Time is important even for the development cycle. The time to market needs to be short, so not all of the application components may work correctly at launch time.

When I started my job, we had several problems with applications. In particular, the ones using old Oracle Application Server (OAS) were problematic. Two or three times in a day the server hung, forcing a restart. I think everyone can understand that we needed something more robust to build our applications. Even with support from Oracle, we didn't reach the goal of no more than one restart in a day.

Moreover, restarts didn't always execute successfully, especially when the system didn't stop correctly. So our solution was to use two application servers for load balancing (or high availability) with a load balancer in front. To do this, we needed an automated control system in order to identify rapidly the application server not working or the application error in building a web page (e.g., the wrong market data). When one of the application servers hangs, we can restart it, but in the meantime, the other server is working.

To help with this identification process, I started to think about an application that would periodically perform a series of checks on URLs to alert us in case of problems. I'd previously found that the perfect language for me was Perl. I'd learned it writing some little CGI scripts, and I've enough confidence with it to prefer it to other languages.

The following is the list of necessities we came up with for the identification application:

  • periodically check URLs

  • maintain history data for analysis

  • administer via the Web

  • view quickly all the checks and statuses

This lists, then, brought about an application formed in three parts: the web administrator (whatsdown.admin), the analyzer (whatsdown.robot) and the data viewer and monitor (whatsdown.archive). To build the application, I used a lot of open-source software: Apache, PostgreSQL (for data repository), Festival (for voice alerts) and Perl (with a lot of modules). The following list is the tools I used to build the application:

  • Net-DNS-0.12

  • Digest-MD5-2.09

  • GD-1.19

  • HTML-Parser-3.05

  • HTML-Template-1.8

  • libnet-1.0607

  • libwww-perl-5.47

  • MD5-1.7

  • MIME-Base64-2.11

  • Net_SSLeay-1.08

  • SNMP

  • URI-1.04

  • CGI

  • Net::Ping

  • DBI-1.14

  • DBD-Pg

  • DBD-Oracle-1.03

During the development cycle of the application, with URLs checked via HTTP and HTTPS, other functions were added: server disk space, UPS status, availability of servers, space on Oracle databases and generic service availability. For every check it is now possible to define the frequency (e.g., check is run every 4 hours); to define an e-mail for send alert messages; to define a mobile phone to alert an administrator via SMS; to disable a server; and to create groups of checks. All checks return a status of OK, ALARM or PROBLEM.

I'm not a developer, and my core work is system administration, so I probably haven't written perfect code. The purpose of this article, however, is to explain how I realized our goal in a short timeframe by using Perl and other useful software I found on the Net. I won't spend time explaining how to use Perl for interacting with PostgreSQL, nor how I installed Festival, because these topics are covered elsewhere abundantly.

As for the application, I mentioned that it had been divided into four parts: a web administrator, a web monitor, a robot and an archiver/viewer via the Web. All these parts of software were builded around a PostgreSQL database, so I built a little module to separate the program business logic from the database. If the application needs the list of active tests to execute, the user can initialize the database connection with use Database ; and request the data with my @tests = Database::getAllTests() ;. This function returns an array of hashes, containing all the details about the tests.