Creating Web Plots on Demand

I work at Virginia Power, an electric utility that services a large section of the southeastern United States. Part of my group's job is to collect and archive data from thousands of devices in the field, such as electrical transformers and circuits. Our use of Linux to collect this data is a story in itself (see “SCADA—Linux Still Hard At Work” in the January and February 1995 issues of Linux Journal).

To make this vast amount of data accessible throughout our company, my group built a set of Intranet applications. These applications are accessible to anyone on our company network using a standard web browser.

Using one of these applications, our users can generate graphic plots of data from our databases. For example, an engineer may want to see a plot of the electrical load on a particular transformer during the course of a month. Our program can extract data from our database, create a plot of that data and display the graphic through the user's web browser in a matter of seconds.

Using tools that come with most standard Linux distributions, you too can build web-based plots on demand. The tools you will need include Perl, the gnuplot graphics package, and the NetPBM graphics conversion tools. I'll assume you already have a web server up and running. We use the Apache web server, but the techniques described here will work with other web servers too.

Before I show you how to tie these tools together, you'll need to get and install gnuplot and the NetPBM packages. See “Resources” for information on Internet sites and Linux distributions where you can get these packages.

gnuplot is a device-independent plotting package that has been available for various Unices, as well as other platforms, for many years. It features its own language for creating and displaying two- and three-dimensional plots. Commands can be entered interactively, one at a time, or can be placed together into a gnuplot script file and run as a batch job.

gnuplot supports a wide variety of output devices. The most useful of these devices can be an X window. You can run gnuplot from an xterm under X Windows, and it can plot your data into a separate X window. This is a good way for those unfamiliar with gnuplot to learn all about the various commands.

Since gnuplot can take its commands from script files and its data from simple text files, it's the perfect tool to automate the generation of graphics when requested by a web server.

I doubt you have much interest in plotting load curves of electrical transformers, so I'll present a more generally useful example. The example is a CGI Perl script that plots web server hits over the course of a given day.

The script will require some understanding of Perl; the small amount of CGI knowledge required will be explained as needed.

First, let's describe what the script needs to do. Every time a person requests information from an Apache web server, the request is logged to a file called access_log. This log resides in the web server's log directory. The exact location of this directory depends on how you have set up your web server. Each line of the log represents a single web hit. A line from an Apache server's access_log file looks something like this:

foobar.mydomain.org
- - [01/Sep/1997:17:14:31 -0400]
"GET /images/gnuplot_10270.gif HTTP/1.0" 200 9538

This line indicates that a web browser at foobar.mydomain.org requested the file /images/gnuplot_10270.gif to be sent to it at 5:14 PM on September 1. All lines in the log have the same format, so we should be able to extract just those lines that match the date we're seeking.

We can also tell the hour and minute of the access. We'll want to keep a count of the number of accesses for each minute of our target day. This data will be sent to gnuplot so it can create our plot. We'll want to create a simple x-y line plot of the data, with time of day on the x axis and number of hits on the y axis.

Once the graph has been plotted, we'll need to convert it to a graphics format that can be displayed with a web browser. Finally, we'll need to build an HTML page to send back to the browser. This page will have an image tag in it that points back to the graphics file we just created.

Don't worry if this seems like a lot of hoops to jump through. It can all be done in seven short steps with less than 100 lines of code. The complete Perl script is shown in Listing 1. The code is divided into numbered sections for easy reference. Let's look at the script one section at a time.