Connecting Apache's Web Server to Multiple Instances of Tomcat

Recently, I was asked to reorganize some of our Web applications to
improve their stability. The major push was to get each of our applications
running in its own instance of Tomcat. These applications all are in
various stages of development, and if a single instance allocates all of
the memory available, the entire Tomcat server must be restarted. This,
in turn, brings down all of the other applications.

An obstacle to running multiple instances is each instance would
have to run on a unique port. Enter Apache's Web server. With the use
of mod_jk, we were able to forward requests to specific hosts and contexts
to the respective running instance of Tomcat.

Pulling all of this together was not the hardest thing I've done;
however, it did prove challenging due to a lack of documentation. By no
means, though, is this statement a stab at any of the development teams. In fact,
the developers themselves helped me on numerous occasions when I found myself
in over my head. When the documentation had me confused or I could not find
exactly what I was looking for, I would head over to #tomcat or #apache on
irc.freenode.org and pose my question there. Usually, I received a response
within minutes.

The following software was used for the purposes of this article. See
the resources section at the end of this article for download locations:

• J2sdk1.4.2_09
• Tomcat 5.0.28
• Apache 2.0.54
• mod_jk 1.2.14

I am certain some of you are wondering why I used the Java software
development kit (SDK) rather than the Java runtime environment (JRE).
The answer is simple: Tomcat requires tools.jar to compile JSP pages,
and tools.jar is provided in the SDK. If you do not wish to use the SDK,
you need to place tools.jar in $CATALINA_HOME\common\lib.
Assumptions
For the purposes of this article, I assume that you already
have Apache, Java and Tomcat installed. With all of the software listed
above installed, except mod_jk, let's set up our environment.

In order for Tomcat to start, you need to set two environment
variables, JAVA_HOME and CATALINA_HOME. JAVA_HOME should point to the
J2sdk installation directory, and CATALINA_HOME should point to the installation
directory for Tomcat. To make life easier, I have placed the
following lines in /etc/bashrc:

export JAVA_HOME=/usr/java/j2sdk1.4.2_09
export CATALINA_HOME=/opt/tomcat 

Notice that I have set CATALINA_HOME to /opt/tomcat. Although this is
the case, /opt/tomcat actually is a symlink to
/opt/jakarta-tomcat-5.0.28. Although you can tell which version of
Tomcat you have installed by running $CATALINA_HOME/bin/catalina.sh
version, I prefer to be able to note the version immediately by
looking at the installation directory.

Before we dive into compiling mod_jk, I would like to dispel some
confusion I came across while working on this project. Specifically, the
confusion centers around the myth that bigger is better. In the world of
Tomcat connectors, there exists mod_jk and mod_jk2. In my mind mod_jk2
was the logical choice, as it had a higher version number. After a bit of
probing and inquiring, however, I discovered that mod_jk2 is
deprecated--it is no longer being developed. Unfortunately for me, I
had invested several hours in getting mod_jk2 to work when I discovered
this fact. Hopefully, you will have saved yourself some time by reading this article.
Compiling, Installing and Configuring mod_jk
Open a terminal and change directories to the location where you have
saved the mod_jk source. See the Resources section at the end of this
article for the download link. In the terminal window, extract the
source archive and install with the following four lines:

tar -xzvf jakarta-tomcat-connectors-1.2.14.1-src.tar.gz
cd jakarta-tomcat-connectors-1.2.14.1/jk/native
./configure --with-axps=/usr/sbin/axps 
make && make install

The --with-axps=/usr/sbin/apxs configuration option allows us to build
Apache modules without requiring the Apache source. If all goes well,
you should see a message near the end informing you of the location of
the libraries. On my distribution, it is /usr/lib/httpd/modules.

With mod_jk installed, we must now configure Apache to load the module
by editing httpd.conf. httpd.conf is located in /etc/httpd/conf on my
system. Configuring Apache to load mod_jk is a simple two-line step:

#Load the mod_jk connector LoadModule jk_module
/usr/lib/httpd/modules/mod_jk.so

A mistake I initially made was naming the module mod_jk, as in
LoadModule mod_jk /usr/lib/httpd/modules/mod_jk.so.
The above command is certain to make Apache complain and refuse to start. With
that said, it is a good idea to start or restart Apache now to verify
that it can load the newly compiled module. If Apache starts
without a problem, it is safe to continue.
Setting Up Multiple Tomcat Instances
Multiple Tomcat instances are possible to create with the use of the CATALINA_BASE
environment variable. Each instance uses a common binary distribution
but uses its own conf, webapps, temp, logs and work directories.
Each instance also has its own JVM and, thereby, its own memory
pool. If you have defined the maximum memory to be 512MB via JAVA_OPTS,
each instance will attempt to allocate a maximum of 512MB.

Let's proceed now to set up these directories. As I mentioned before,
Tomcat is installed in /opt/tomcat. To keep things somewhat organized,
I created the following folders in /opt: /opt/tomcat_instance1,
/opt/tomcat_instance2 and /opt/tomcat_instance3. It probably is more
appropriate, however, to name these folders based on their purposes or
applications. Remember that each of the three folders will contain
conf, webapps, temp and work directories.
Configuring the First Instance
Tomcat uses a server.xml configuration file to determine
the ports, connector engines and various other "server"
configuration options. We are going to copy the installed
server.xml from $CATALINA_HOME/conf/server.xml to
/opt/tomcat_instance1/conf/server.xml. While we are at
it, we might as well copy $CATALINA_HOME/con/server.xml
to /opt/tomcat_instance2/conf/server.xml and
/opt/tomcat_instance3conf/server.xml as well.

Tomcat also uses a global web.xml file. By global, I mean it is used
for each instance. The web.xml file provides the default configuration
for each Web application running under the given instance. If an option is not defined in the
individual Web application, the default web.xml option is used.
We can copy $CATALINA_HOME/conf/web.xml to /opt/tomcat_instance1/conf,
/opt/tomcat_instance2/conf and /opt/tomcat_instance3/conf.

Now we must make some edits to server.xml. First, we need to disable
the Coyote connector. To do this, we comment out the Coyote connector
information. This is an XML file, so it uses the same comment syntax as
HTML. After we are done commenting out the the connector, it should look
something like this:

 <!-- 
  <Connector port="8080"
               maxThreads="150" minSpareThreads="25" maxSpareThreads="75"
               enableLookups="false" redirectPort="8443" acceptCount="100"
               debug="0" connectionTimeout="20000"
               disableUploadTimeout="true" />
 -->

Because this is the first running instance, we do not need to modify
any more of this file. For subsequent instances, we are required to
change the shutdown port and the AJP connector port. The AJP connector
port is the port that Apache uses to forward requests.

Next, copy the servlets-examples file provided with the
installation of Tomcat from $CATALINA_HOME/webapps/servlets-examples to
/opt/tomcat_instance1/webapps/servlets-examples. Again, copy the sample
application to /opt/tomcat_instance2/webapps and
/opt/tomcat_instance3/webapps as well. At this point, the set up of
tomcat_instance1 is complete. We now need to set up the second and third
instances before we pull it all together.
Configuring the Second Instance
We already copied server.xml from the installation directory. We now
need to make the same edit to /opt/tomcat_instance2/conf/server.xml as we
did for the first instance. That is, comment out the Coyote connector exactly as
we did above.

Additional required edits are to change the SHUTDOWN port from 8005 to
8105. We must change the port from 8005 because the first instance
already is using it. You can change the second instance's port to be
any unused port above 1024, but for simplicity and organization's sake,
let's use 8105. Here is the line as it should be in the file:

<Server port="8105" shutdown="SHUTDOWN" debug="0">

Now we must change the AJP connector from 8009 to 8109. Again, this is
required because the first instance already is using 8009. Below
is the line to change with the required edit:

    <Connector port="8109"
	enableLookups="false" redirectPort="8443" debug="0"
      protocol="AJP/1.3" />

If you are using SSL, you also should change the redirectPort to
the appropriate SSL port that Apache is listening on, normally 443.
Configuring the Third Instance
We need to make the same edits to /opt/tomcat_instance3/conf/server.xml as we
did for the second instance, except we substitute 8205 for 8005 and
8209 for 8009. In case this does not make sense, here are the respective
sections of server.xml:

<Server port="8205" shutdown="SHUTDOWN" debug="0">

    <Connector port="8209"
	enableLookups="false" redirectPort="8443" debug="0"
      protocol="AJP/1.3" />

Configuring mod_jk
mod_jk uses a file named workers.properties. I recommend placing this
file with the rest of your Apache configuration files. workers.properties
is used to define where Apache looks for the Tomcat instances. Here, I
cover only the items we are going to use for the three instances we have
set up. Below is the workers.properties file we are going to use,
followed by an explanation of the options:

worker.list=worker1,worker2,worker3
# Set properties for worker1
worker.worker1.type=ajp13
worker.worker1.host=localhost
worker.worker1.port=8009
# Set properties for worker2
worker.worker2.type=ajp13
worker.worker2.host=localhost
worker.worker2.port=8109
# Set properties for worker3
worker.worker3.type=ajp13
worker.worker3.host=localhost
worker.worker3.port=8209

worker.list is a comma-separated list of worker names. You could have
Tomcat workers defined later in the file that will not be used. Any worker
defined is not used unless the worker is listed in the worker.list value.
Workers are defined in format of
worker.NAMEOFWORKER.type, with the value being
the type of connector. All of our workers are of type ajp13. In the
above example, we have defined three workers: worker1, worker2 and worker3.

You may have noticed the host portion of the configuration. This can
be used to configure Apache to forward to Tomcat instances on separate
machines. In fact, this is an option that one might choose to employ in
order to make a site more secure. Because Tomcat and Apache both reside
on the same machine, we use localhost.

Each worker also needs to define the port on which the connector is configured
to work. If you remember, earlier we configured instance1 to listen
on port 8009, instance2 to listen on port 8109 and instance3 to listen
on port 8209.
Configuring Apache with mod_jk
To get all of this started, we need to tell Apache where to find the
workers.properties file and where to log mod_jk requests. We also need
to specify the format of the log files and the options specific to
mod_jk. I did this by adding the following lines to httpd.conf; I
placed all mod_jk configuration directives just before the virtual
host declarations:

JkWorkersFile "/etc/httpd/conf/workers.properties"
JkLogFile     "/var/logs/www/mod_jk.log"
JkLogLevel  info
JkLogStampFormat "[%a %b %d %H:%M:%S %Y] "
JkOptions     +ForwardKeySize +ForwardURICompat -ForwardDirectories
JkRequestLogFormat     "%w %V %T"

The above options tell Apache to use /etc/httpd/conf/workers.properties
for the worker definitions and to use the /var/logs/www/mod_jk.log
log file. If you are experiencing trouble with mod_jk, adjust the
JkLogLevel to "debug" in order to get more verbose messages. JKLogStampFormat
and JkRequestLogFormat define the logging formats. The option
ForwardKeySize instructs mod_jk to forward the SSL key size along with
the request. ForwardURICompat instructs mod_jk to forward the URL to
Tomcat normally. -FowardDirectories instructs mod_jk not to return a
directory listing from Tomcat.
Configuring Apache to Forward
We use domain1, domain2 and domain3 as our virtual hosts. To pull this
off, we also have to edit /etc/hosts to ensure that domain1, domain2
and domain3 are resolved properly. To do this, we make three
VirtualDirectory declarations, each corresponding to a worker defined
in workers.properties. Below is the VirtualHosts section, followed
by an explanation of the options.

<VirtualHost *:80>
    ServerName domain1
    JkMount  /servlets-examples/* worker1
</VirtualHost>
<VirtualHost *:80>
    ServerName domain2
    JkMount  /servlets-examples/* worker2
</VirtualHost>
<VirtualHost *:80>
    ServerName domain3
    JkMount  /servlets-examples/* worker3
</VirtualHost>

Configuring the Instances to Start at Boot
Now that we have most of the configuration complete, it is time to set
up the instances to start at boot. This partially is done by creating
a bash script and placing it in /etc/init.d. I used the start-up
script found
here.

#!/bin/bash
#
# tomcat        
#
# chkconfig: 
# description:  Start up the Tomcat servlet engine.

# Source function library.
. /etc/init.d/functions


RETVAL=$?
export CATALINA_BASE="/opt/tomcat_instance1"
export CATALINA_HOME="/opt/tomcat"

case "$1" in
 start)
        if [ -f $CATALINA_HOME/bin/startup.sh ];
          then
            echo $"Starting Tomcat"
            /bin/su tomcat $CATALINA_HOME/bin/startup.sh
        fi
        ;;
 stop)
        if [ -f $CATALINA_HOME/bin/shutdown.sh ];
          then
            echo $"Stopping Tomcat"
            /bin/su tomcat $CATALINA_HOME/bin/shutdown.sh
        fi
        ;;
 *)
        echo $"Usage: $0 {start|stop}"
        exit 1
        ;;
esac
exit $RETVAL

Copy the contents above and place them in /etc/init.d/tomcat_instance1, /etc/init.dtomcat_instance2 and
/etc/init.d/tomcat_instance3. Be sure to change CATALINA_BASE to
the appropriate directory for each script. Now, we need to link to
these in /etc/rc5.d. To create the symlinks, issue the following as root:

cd /etc/rc5.d
ln -s /etc/init.d/tomcat_instance1  S71tomcat_service1
ln -s /etc/init.d/tomcat_instance2 S71tomcat_service2
ln -s /etc/init.d/tomcat_instance3 S71tomcat_service3

Testing the Setup
With all of our configuration complete, it now is time to test our
setup. We begin by starting Apache or restarting if it already is
running. Next, bring up the first instance:

/etc/init.d/httpd stop
/etc/init.d/httpd start
/etc/init.d/tomcat_service1 start

Now, open a browser window and go to http://domain1/servlets-examples/.
If all goes well, you should see something similar to Figure 1.
Figure 1. Checking Your Setup
If you do not see a page similar to the above, look in the
log files. Specifically, check /var/logs/httpd/mod_jk.log and
/opt/tomcat_instance1/logs/catalina.out for any errors that may have
occurred.

If everything looks correct, go ahead and start the remaining
two contexts:

/etc/init.d/tomcat_instance2 start
/etc/init.d/tomcat_instance3 start

Point your browser to http://domain2/servlets-examples and
http://domain3/servlets-examples. You should see the exact same page
for all three instances.

To make things a bit more clear and to verify that we are hitting the
correct instance, we can modify a line
in /opt/tomcat_instance2/webapps/servlets-examples/index.html and
/opt/tomcat_instance3/webapps/servlets-examples/index.html. The line to
modify in both files is:

<b><font face="Arial, Helvetica, sans-serif"><font size=+2>Servlet
Examples with Code</font></font></b>

For the second instance, we want the line to read as follows:

<b><font face="Arial, Helvetica, sans-serif"><font size=+2>Domain 2 Servlet
Examples with Code</font></font></b>

Make a similar modification to the third instance:

<b><font face="Arial, Helvetica, sans-serif"><font size=+2>Domain 3 Servlet
Examples with Code</font></font></b>

If you now point your browser to http://domain2/servlets-examples and
http://domain3/servlets-examples, you should see Domain 2 and Domain 3 at
the beginning of the respective page.
Conclusion
Hopefully this HOWTO has given you a step-by-step understanding of what
it takes to get multiple instances of Tomcat running when Apache is
the front-end. Many more options are available that would produce similar
setups. You could, for example, use Apache as a front-end and load
balance between several servers running Tomcat. See the documentation
for mod_jk for more information about the available options.
Resources
The Apache
Jakarta Tomcat Connector

Jakarta-Tomcat
Connectors Source Files

Apache Tomcat

J2SE

Apache