As the Log Scrolls By...

As a web hosting company, there are times when it's vitally important to see what our Apache web server is dishing up to the outside world at any one time—and to see this as quickly as possible.

Just as a system administrator needs to be able to monitor system log files, a web administrator should be able to do the same with web logs. Noting the number of utilities that display system log information in real time, I was sure there'd be similar programs for monitoring web logs. After a search through Freshmeat.net and other on-line resources, I didn't find anything that met all of my needs. Several came close, but most would only monitor one file, and the few that monitored several files would leave me lost trying to wade through tabs for the 30 or so logs that I watch.

The solution I found isn't in one program, but in making a few changes in Apache's httpd.conf file I found I could have a specialized disposable log file containing only the information I require, for all of our web hosts. The “disposable” monitoring log is then displayed using colortail (with additional configuration settings) on an external monitor in our workshop. I can see at a glance which of our hosts have current web activity, where the traffic is coming from and the pages that are being accessed. This has also allowed us to deal quickly with problems such as script kiddies and rogue search engine robots. The system has worked so well for us that we added system logging to it as well.

In addition to the LogFormat for general logging, I added a new format labeled “webmonitor”:

LogFormat "[%v] %h %u \"%r\"%>s%b\n\"%{Referrer}i\" \"%{User-Agent}i\"%t" webmonitor

This displays log information with the Referrer and User-Agent on a second line, making it clearer to read. The log file could be in any format—even the “common” one we use for standard logging. I decided to change it for purposes of clarity and because Apache is flexible enough to allow this.

Since graphic files such as GIFs, JPEGs or PNGs files can clutter up the display, I exclude them by adding the following three lines to the general log section in httpd.conf:

SetEnvIf Request_URI \.gif$ unwanted
SetEnvIf Request_URI \.jpg$ unwanted
SetEnvIf Request_URI \.png$ unwanted

We use name-based virtual hosts, and each host has their own <VirtualHost></VirtualHost> container. In addition to their permanent log file, we add an additional CustomLog command for our webmonitor file for each of our hosts. For example:

<VirtualHost someisp.com>
...
CustomLog /var/log/httpd/someisp.com-access_log combined
CustomLog /var/log/httpd/webmonitor_log webmonitor env=!unwanted
...
</VirtualHost>

CustomLog /var/log/httpd/webmonitor_log webmonitor env=!unwanted

The ability to see what's happening on the server with an external monitor proved so useful that we also included system logging information in the same file. To do this, we edited /etc/tem syslog.conf to include the following command:

kern.*;authpriv.*;*.crit;*.error;*.warning;*.emerg /var/log/httpd/webmonitor_log

Colortail was written by Joakim Andersson (pt98jan@student.hk-r.se) and is available from www.student.hk-r.se/~pt98jan/colortail.html under the GNU Public License.

While we could simply tail the webmonitor log file, adding color to the display is a nice touch and gives us an indication of which web host is seeing activity even if we happen to be on the other side of the workshop.

Colortail comes with several sample configuration files; none really suited web logs, although conf.xferlog comes close. After a bit of tweaking, this is the format we've been using. It's a hybrid as it includes both web and system-log-related items.

Listing 1. colortail.conf