Creating Web Plots on Demand
I work at Virginia Power, an electric utility that services a large section of the southeastern United States. Part of my group's job is to collect and archive data from thousands of devices in the field, such as electrical transformers and circuits. Our use of Linux to collect this data is a story in itself (see “SCADA—Linux Still Hard At Work” in the January and February 1995 issues of Linux Journal).
To make this vast amount of data accessible throughout our company, my group built a set of Intranet applications. These applications are accessible to anyone on our company network using a standard web browser.
Using one of these applications, our users can generate graphic plots of data from our databases. For example, an engineer may want to see a plot of the electrical load on a particular transformer during the course of a month. Our program can extract data from our database, create a plot of that data and display the graphic through the user's web browser in a matter of seconds.
Using tools that come with most standard Linux distributions, you too can build web-based plots on demand. The tools you will need include Perl, the gnuplot graphics package, and the NetPBM graphics conversion tools. I'll assume you already have a web server up and running. We use the Apache web server, but the techniques described here will work with other web servers too.
Before I show you how to tie these tools together, you'll need to get and install gnuplot and the NetPBM packages. See “Resources” for information on Internet sites and Linux distributions where you can get these packages.
gnuplot is a device-independent plotting package that has been available for various Unices, as well as other platforms, for many years. It features its own language for creating and displaying two- and three-dimensional plots. Commands can be entered interactively, one at a time, or can be placed together into a gnuplot script file and run as a batch job.
gnuplot supports a wide variety of output devices. The most useful of these devices can be an X window. You can run gnuplot from an xterm under X Windows, and it can plot your data into a separate X window. This is a good way for those unfamiliar with gnuplot to learn all about the various commands.
Since gnuplot can take its commands from script files and its data from simple text files, it's the perfect tool to automate the generation of graphics when requested by a web server.
I doubt you have much interest in plotting load curves of electrical transformers, so I'll present a more generally useful example. The example is a CGI Perl script that plots web server hits over the course of a given day.
The script will require some understanding of Perl; the small amount of CGI knowledge required will be explained as needed.
First, let's describe what the script needs to do. Every time a person requests information from an Apache web server, the request is logged to a file called access_log. This log resides in the web server's log directory. The exact location of this directory depends on how you have set up your web server. Each line of the log represents a single web hit. A line from an Apache server's access_log file looks something like this:
foobar.mydomain.org - - [01/Sep/1997:17:14:31 -0400] "GET /images/gnuplot_10270.gif HTTP/1.0" 200 9538
This line indicates that a web browser at foobar.mydomain.org requested the file /images/gnuplot_10270.gif to be sent to it at 5:14 PM on September 1. All lines in the log have the same format, so we should be able to extract just those lines that match the date we're seeking.
We can also tell the hour and minute of the access. We'll want to keep a count of the number of accesses for each minute of our target day. This data will be sent to gnuplot so it can create our plot. We'll want to create a simple x-y line plot of the data, with time of day on the x axis and number of hits on the y axis.
Once the graph has been plotted, we'll need to convert it to a graphics format that can be displayed with a web browser. Finally, we'll need to build an HTML page to send back to the browser. This page will have an image tag in it that points back to the graphics file we just created.
Don't worry if this seems like a lot of hoops to jump through. It can all be done in seven short steps with less than 100 lines of code. The complete Perl script is shown in Listing 1. The code is divided into numbered sections for easy reference. Let's look at the script one section at a time.
|Speed Up Your Web Site with Varnish||Jun 19, 2013|
|Non-Linux FOSS: libnotify, OS X Style||Jun 18, 2013|
|Containers—Not Virtual Machines—Are the Future Cloud||Jun 17, 2013|
|Lock-Free Multi-Producer Multi-Consumer Queue on Ring Buffer||Jun 12, 2013|
|Weechat, Irssi's Little Brother||Jun 11, 2013|
|One Tail Just Isn't Enough||Jun 07, 2013|
- Speed Up Your Web Site with Varnish
- Containers—Not Virtual Machines—Are the Future Cloud
- Linux Systems Administrator
- Non-Linux FOSS: libnotify, OS X Style
- Lock-Free Multi-Producer Multi-Consumer Queue on Ring Buffer
- Senior Perl Developer
- Technical Support Rep
- UX Designer
- RSS Feeds
- Reply to comment | Linux Journal
3 hours 16 min ago
- Yeah, user namespaces are
4 hours 32 min ago
- Cari Uang
8 hours 4 min ago
- user namespaces
10 hours 57 min ago
11 hours 23 min ago
- One advantage with VMs
13 hours 52 min ago
- about info
14 hours 25 min ago
14 hours 26 min ago
14 hours 27 min ago
14 hours 29 min ago
Free Webinar: Hadoop
How to Build an Optimal Hadoop Cluster to Store and Maintain Unlimited Amounts of Data Using Microservers
Realizing the promise of Apache® Hadoop® requires the effective deployment of compute, memory, storage and networking to achieve optimal results. With its flexibility and multitude of options, it is easy to over or under provision the server infrastructure, resulting in poor performance and high TCO. Join us for an in depth, technical discussion with industry experts from leading Hadoop and server companies who will provide insights into the key considerations for designing and deploying an optimal Hadoop cluster.
Some of key questions to be discussed are:
- What is the “typical” Hadoop cluster and what should be installed on the different machine types?
- Why should you consider the typical workload patterns when making your hardware decisions?
- Are all microservers created equal for Hadoop deployments?
- How do I plan for expansion if I require more compute, memory, storage or networking?