Configuring, Tuning and Debugging Apache

RPMs, mod_perl, apachectl, telnet, mod_status, DSO, apxs, Apache::Status...Scared yet? Mr. Lerner tells us about all of these so we can better manage our web servers.
mod_perl

mod_perl, the Apache module which makes it possible to write new modules in Perl and to configure existing ones using the popular language, can be complied as a DSO. This requires the use of apxs, as in the case of mod_speling. However, mod_perl is much more complicated than a single module, and depends on many outside pieces of information for its compilation. mod_perl is thus configured and compiled similarly to stand-alone Perl modules, with perl Makefile.PL, followed by a make, make test and make install.

In order to compile a new version of mod_perl into an existing version of Apache, issue the following command:

perl Makefile.PL \
USE_APXS=1 WITH_APXS=/usr/local/apache/bin/apxs

If you want to enable mod_perl for all of the different Apache handlers, rather than just the default PerlHandler (for creating dynamic content), turn on the EVERYTHING switch. For example:

perl Makefile.PL \
USE_APXS=1 WITH_APXS=/usr/local/apache/bin/apxs  \
EVERYTHING=1
Once you have created the Makefile in this way, you can compile and install mod_perl into the existing Apache server with:
make
make test
make install
This method works not only for installing a new copy of mod_perl into Apache, but also for upgrading an existing copy. You can then test to see if mod_perl has been compiled into the server by telneting to the server's address and port number, and issuing the command:
HEAD / HTTP/1.0
This will return the HTTP headers associated with the / document on
the server. Among other things, there should be a "Server" header
indicating what kind of server is running. mod_perl adds a tag to
this output string, so you should see output like the following:
Server: Apache/1.3.12 (UNIX) mod_perl/1.24
Because mod_perl updates come out at different times than Apache updates, I have found it to be extremely useful to install and upgrade mod_perl in this way.

Apache::Status

While mod_status can tell us what is happening with each Apache process, it cannot tell us what is happening inside a particular module. In general, this is not such a bad thing; do I really care what is happening inside mod_mime or mod_speling?

But in the case of mod_perl, where so many complex things are going on, it would be nice to be able to find out what is going on. Perl Apache::Status module, which works with mod_perl, provides this information.

In order to activate Apache::Status, we need to insert another section into httpd.conf. As with mod_status, we will create a new Location section which associates a particular handler with a virtual URL, traditionally known as “/perl-status”:

PerlModule Apache::Status
<Location /perl-status>
    SetHandler  perl-script
    PerlHandler Apache::Status
</Location>

Once we have restarted the server or sent it a HUP signal, requesting the URL /perl-status will produce a menu of options, such as “Environment” and “Inheritance tree”. Other Perl modules for mod_perl, such as HTML::Mason, can install their own hooks for Apache::Status, making it possible to look through their environments. For example, Mason provides a simple interface for viewing the current configuration, as well as a list of components that have been compiled and cached.

A Short Story

Over the days preceding my writing of this column, I used all of the above techniques to track down a problem with my installation of HTML::Mason. The colocated server that I help run was having some problems handling a growing load. Every few hours, all of the Mason-based sites on the server would fail, followed one or two hours later by the non-Mason sites. Actually, “fail” is too strong a word—the browser would send a request to the server, but would time out (after ten minutes or so) of waiting to receive a connection. What was going on, and how was I going to fix it?

My first reaction was to think that our server was running out of RAM. I used “top” and “free” to inspect the system, but did not see anything out of the ordinary. On the one hand, this was a relief. At the same time, this meant we were somehow running out of available servers, even though I had configured the system for a maximum of 150 simultaneous clients. My server might be popular, but 150 simultaneous accesses is highly unusual, even for me. Something else was obviously going on here.

There was clearly some connection to Mason, and perhaps to mod_perl. I decided it was about time to upgrade mod_perl to the latest version (1.24), using the technique I described above. So I upgraded the copy of HTML::Mason and several other related modules, restarted Apache with apachectl, and hoped the problem would go away.

Unfortunately, the simple upgrade did not do the trick. The system still stopped responding to requests after several hours of activity. I used Apache::Status to look at the state of mod_perl, and nothing seemed to be out of the ordinary.

I looked at the system status with mod_status, and discovered that over time, many of the Apache processes were getting stuck in the “write” state. In other words, the list of Apache processes was slowly, but surely, being transformed from a list of “.” (unallocated) processes to a list of “W” (writing an HTTP response) processes. Every invocation of a Mason component on the system was using up another Apache process, and never letting it go! It was no surprise, then, that the system was locking up after only a few hours; if the Mason-related parts of the site had been more popular, the system would have gone down even more quickly.

I looked through the Mason configuration file and determined that my use of the Apache::Session module, which allows mod_perl programs to track a user's movements and actions (as we saw in my article, “Session Management with Mason” in the August Linux Journal) was failing to return. So each time a Mason component was invoked, everything would work fine—until the component needed to return, at which point the executing program would simply spin its wheels, waiting to connect to the MySQL database.

My solution was not particularly elegant, but did the trick: I decided to stop using the MySQL version of Apache::Session (known as Apache::Session::MySQL) and start using the simple file-based version (known as Apache::Session::File). I restarted the server, and was delighted to discover that everything was working as it should.

______________________

Webinar
One Click, Universal Protection: Implementing Centralized Security Policies on Linux Systems

As Linux continues to play an ever increasing role in corporate data centers and institutions, ensuring the integrity and protection of these systems must be a priority. With 60% of the world's websites and an increasing share of organization's mission-critical workloads running on Linux, failing to stop malware and other advanced threats on Linux can increasingly impact an organization's reputation and bottom line.

Learn More

Sponsored by Bit9

Webinar
Linux Backup and Recovery Webinar

Most companies incorporate backup procedures for critical data, which can be restored quickly if a loss occurs. However, fewer companies are prepared for catastrophic system failures, in which they lose all data, the entire operating system, applications, settings, patches and more, reducing their system(s) to “bare metal.” After all, before data can be restored to a system, there must be a system to restore it to.

In this one hour webinar, learn how to enhance your existing backup strategies for better disaster recovery preparedness using Storix System Backup Administrator (SBAdmin), a highly flexible bare-metal recovery solution for UNIX and Linux systems.

Learn More

Sponsored by Storix