Efficiently Updating Web Sites on Clusters

Management said the requirements for the web site infrastructure were simple. We could expect thousands of visitors to our site at any time over the next few months, as the marketing effort took hold. At any given time we would need to support up to a few hundred concurrent downloads of our desktop product demo. A noticeable performance drop in our web site was not acceptable.

At the time, our web site ran on a 2.2.x Linux distribution on a dual Dell 2450. Its performance was rock solid, but we were uneasy relying on a single machine for our entire business. We determined that the answer was to replace the single server with a cluster. For less than the cost of the Dell, we built a cluster of four 1U one-processor machines. The cluster's performance was excellent. By stripping down Apache, we could support at least 400 downloads over HTTP and still have a responsive site. This left one problem: we needed to be able to update the site often without affecting the performance.

Typical strategies for doing these frequent updates were not satisfactory. Either the site would be down for more than a few moments when the update occurred, or the site would be in an inconsistent state during the update. Worst of all, the site could be left in an inconsistent state if the update failed part-way through the process. To overcome these drawbacks I applied a little cross-discipline creativity. By applying the page flipping technique from the graphics world, I was able to achieve a quick and non-intrusive method of updating the clustered web site.

The clustered web site consists of a director machine and a number of worker machines. The specifics of clustering are beyond this article, but loosely the director machine accepts the request from the client browser and then routes the request to a worker machine. The worker machine then responds directly to the client browser. The multiple worker machines provide scalability and a high level of reliability.

The cluster resides off site at a colocation facility, and a staging server is kept on site. This staging server contains the entire functioning web site for testing. Once the changes are acceptable, the site can be mirrored out to the worker machines in the cluster.

I used a number of techniques to ease the administration of the cluster. The worker machines in the cluster are all accessible from the web account on the staging server using a no-passphrase key pair over SSH. Various scripts were written using sudo so the web user can stop, start and reload Apache and Tomcat. Thus a single user logged into the web account on the staging server could control all the worker machines in the cluster with a small number of command-line tools.

The update-web script is part of this collection of tools and is used to update the web sites in the cluster. With this script, a user can update the entire cluster with a single command:

$update-web acme.com

where acme.com is the name of the site being updated. The names of the worker machines are kept in a common file used by all the scripts.

Double buffering and page flipping are similar concepts. The idea is you draw an image or widgets to an area of memory that is currently not being displayed to the user. Once you are finished drawing, the complete image is displayed to the user all at once. With double buffering, the in-memory image is moved to the display area all at once using a block line transfer (blit). With page flipping, the area of memory written to is a special location in video memory, hidden from the user. When the drawing is done, the video hardware effectively changes a pointer and starts using the new in-memory image as the active display memory.

For our situation I chose to apply the page-flipping technique. We did this by creating two copies of the web site on each worker machine. A symbolic link then acts as the pointer that can be flipped. The directory structure is as follows:

The httpd.conf files for this site refer only to the symbolic link path. Therefore, we can quickly change which copy of the site is being used simply by changing the symbolic link.

To assist in this process some shell scripts are placed in the ~/bin directory of the web account on each worker machine. They are:

These three scripts allow us to determine easily the inactive site, update it in whatever manner we choose and then make it the active site.

#!/bin/bash
# minniger@minnigerode.com
# first make sure this is a site that can be flipped
if [ ! -L /web sites/$1 ]; then
        echo "FAILED: $1 is not a sym link! Therefore this is not a flippable web site."
        exit 1
fi
# just return the destination of the symlink
OUT=`ls -l /web sites/$1 | awk -F "->" '{print $2}'`
# using out will trim the spaces around the return value
echo $OUT

#!/bin/bash
# minniger@minnigerode.com
# first make sure this is a site that can be flipped
if [ ! -L /web sites/$1 ]; then
        echo "FAILED: $1 is not a sym link! Therefore this is not a flippable web site."
        exit 1
fi
# Now assume that there is a sym link to the active web site.  Also the web sites
# must be "/web sites/$1.1" and "/web sites/$1.2".   The line below lists the
# symlink.  Checks the last char to see if it
# is a 1 and if so returns a 2 else it returns a 1 (awk is my friend).
OUT=`ls -l /web sites/$1  | awk '{ val = /1$/} END {  if ( val )   print 2; else print 1; }'`
# now just output the given web site with the found value appended to it
echo "/web sites/$1.$OUT"

#!/bin/bash -login
#
# minniger@minnigerode.com
#
# Use this script to "flip" the sym link for a web site between the two directories
# that have the web site data.
#
# /web sites/acme.com  is a symlink to /web sites/acme.com.1
# after running this script it will be a symlink to /web sites/acme.com.2
# Run the script again it will be pointed back to the first one.
#
if [ ! $# = 1 ]; then
        echo "web siteflipper is a script for doing a page flip like thing with a web site"
        echo "Usage: web siteflipper <web sitename>"
        exit 1
fi
# get the inactive path
HOLD=`getInactiveSitePath $1`
# the above script will test to see if we've given it a sym link location...
if [ ! $? = 0 ]; then
        echo "Failed to flip site: /web sites/$1"
        exit 1
fi
# remove the symbolic link
rm /web sites/$1
# make a new one pointing to the inactive path
ln -s $HOLD /web sites/$1
echo $1 is now symlink to $HOLD

With the three scripts in place on each of the worker machines, we are ready to do updates from the staging server. In order to minimize bandwidth and still keep things simple we use rsync over SSH. The script update-web implements our algorithm. We'll walk through it section by section.

The lead in is the typical shell invocation and usage messages

#!/bin/bash
#
# update-web
#
# Update he production web servers with the data at the given sites.
# Note that this script assumes the sites will be in the /web sites directory.
#
# minnniger@minnigerode.com
#
#
#
if [ $# = 0 ]; then
        echo "Usage: update-web web sitedir [web sitedir2, ...]"
        exit 1
fi

Next, we keep a list of the worker machines in a local variable SERVERLIST. This could also be taken from a common file. The sites that we're going to update are passed via the command line and stored in SITELIST.

# list of the known servers
SERVERLIST=`cat workers`
# the target www sites
SITELIST=$*

Now comes some looping. For each site we'll update each server in turn.

for SITE in $SITELIST; do
   echo "Updating all servers for site: "$SITE
   for SERVER in $SERVERLIST; do

From inside the loop we can get the location of the inactive site from the current worker. The dosshcmd script is a wrapper around SSH; it's simply a convenient way to send commands to the worker machines (see example 4). If the command fails for any reason, a message is printed and processing stops. The result of the command is stored in TARGET.

        TARGET=`./dosshcmd $SERVER bin/getInactiveSitePath $SITE`
        if [ ! $? = 0 ]; then
           echo "some failure with $SERVER:$SITE aborting."
           exit 1
        fi

Next, we'll use rsync to update the inactive site. Again, if there is an error we simply stop processing. Because we are updating the inactive copy of the site, the active (production) sites are not being affected in any way.

        echo "rsync /web sites/$SITE/ to $SERVER:$TARGET/"
        rsync --delete --force -re 'ssh ' /web sites/$SITE/ $SERVER:$TARGET/
        if [ ! $? = 0 ]; then
           echo "some failure with $SERVER:$SITE aborting."
           exit 1
        fi
   done

At this point we have successfully updated all of the inactive sites. If there were any problems, the update would have stopped in a safe state. Now we can flip the symbolic links and reload the web servers so all the workers can use the update site information. vscontrol.sh is another helper script that reloads the given SITE when called in this manner:

   echo "Flip the symlink '$SITE' and restart the web server for each server in parallel"
   for SERVER in $SERVERLIST; do
        echo "flip for: $SERVER"
        ./dosshcmd $SERVER bin/web siteflipper $SITE
        ./dosshcmd $SERVER bin/vscontrol.sh $SITE &
   done

Finally we'll use rsync to synchronize the sites on each worker from the newly active site to the newly inactive site. Strictly speaking, this is not needed but it minimizes the network traffic at the time of the next update, and it doesn't really cost us much right now. The script syncActiveToInactive handles this step for us.

for SERVER in $SERVERLIST; do
        echo "rsync the newly active path to the old path for: $SERVER"
        ./dosshcmd $SERVER bin/syncActiveToInactive $SITE
   done

The complete listing of update-web is available in Example 5.

#!/bin/bash
# do the given cmd on the indicated server
# Uses the files:
# serverlist  -  a list of the workers
# lvsdirlist - a list of the linux virtual server directors
# weblist - a list of the web sites we know about
if [  $# = 0 ]; then
        echo "Usage: dosshcmd <all|dir|web|servername> cmd"
        exit 1
fi
case $1 in
        all)
                LIST=`cat serverlist`
                SKIP=1
                ;;
        dir)
                LIST=`cat lvsdirlist`
                SKIP=1
                ;;
        web)
                LIST=`cat weblist`
                SKIP=1
                ;;
        *)
                LIST=$1
                SKIP=0
                ;;
esac
shift
for X in $LIST; do
        if [  $SKIP = 1 ]; then
            echo $X" -------------------"
            echo "   " `ssh web@$X $*`
        else
            ssh web@$X $*
        fi
done

#!/bin/bash
#
# update-web
#
# Update he production web servers with the data at the given sites.
# Note that this script assumes the sites will be in the /web sites directory.
#
# minnniger@minnigerode.com
#
#
#
if [ $# = 0 ]; then
        echo "Usage: update-web web sitedir [web sitedir2, ...]"
        exit 1
fi
# list of the known servers
SERVERLIST=`cat workers`
# the target www sites
SITELIST=$*
for SITE in $SITELIST; do
   echo "Updating all servers for site: "$SITE
   for SERVER in $SERVERLIST; do
        TARGET=`./dosshcmd $SERVER bin/getInactiveSitePath $SITE`
        if [ ! $? = 0 ]; then
           echo "some failure with $SERVER:$SITE aborting."
           exit 1
        fi
        echo "rsync /web sites/$SITE/ to $SERVER:$TARGET/"
        rsync --delete --force -re 'ssh ' /web sites/$SITE/ $SERVER:$TARGET/
        if [ ! $? = 0 ]; then
           echo "some failure with $SERVER:$SITE aborting."
           exit 1
        fi
   done
   echo "Flip the symlink '$SITE' and restart the web server for each server in parallel"
   for SERVER in $SERVERLIST; do
        echo "flip for: $SERVER"
        ./dosshcmd $SERVER bin/web siteflipper $SITE
        ./dosshcmd $SERVER bin/vscontrol.sh $SITE &
   done
   wait
   for SERVER in $SERVERLIST; do
        echo "rsync the newly active path to the old path for: $SERVER"
        ./dosshcmd $SERVER bin/syncActiveToInactive $SITE
   done
   echo ""
done

The alternatives to the web update solution we chose fall into two categories: shut the whole thing down or update the cluster one subset at a time. Clearly shutting the entire cluster down to do an update is the least desirable option. One of the main goals of the cluster is to achieve higher availability than what a single server offers. This method for doing updates treats the cluster as a single machine. In short, it's convenient to set up but eliminates the benefits of the cluster, so it is not a valid choice.

The other alternative is to shut down only a part of the cluster at a given time, so we only have a subset of the cluster off-line at any one time. While this preserves availability, this method presents a number of other problems. This example illustrates them: We can divide the cluster in half, into group A machines and group B machines. We disable group A and rely on our cluster director machine to note that the machines are down and remove them from the active worker list. We then update group A while group B continues to function. Once we finish updating group A, we can disable group B and enable group A. The director machine eventually will see this, and the users will begin using the new site. Meanwhile, we can update group B. Once finished, we can enable group B, thus the entire site is updated.

The two major drawbacks of this solution are 1) the cluster is in an inconsistent state at the time we switch from group A to group B and 2) the users sessions are dropped when the groups are disabled. We also can see that coordinating this effort can quickly become complex; if there are failures during the update, the entire cluster can be left in an inconsistent state.

Further, we are losing half of our cluster for the entire time it takes to update it. If we do subsets by thirds or quarters, we're keeping more of the cluster available, but we're going to have serious issues with the site being inconsistent while the updates occur.

The most obvious benefit of the web site flipping solution is it will only cause a small hiccup in the availability and performance of the sites being hosted by the cluster. If the updates to the sites are simple, then the reload of the site will not affect even the current user sessions. If the updates are complex, then we lose the session but at least the site is consistent when it reappears.

Next, the update scheme is not too complex and scales well. As currently implemented, you don't need to worry about the number of nodes that you have--simply let the scripts do the work. If some of the nodes are removed or some new nodes are added, run the update-web script after altering the list of worker machines and the scripts will do the right thing.

The update method is relatively failsafe. All the sites are synchronized before the web servers are reloaded. If any of the synchronizations fail, the whole process is stopped. Contrast this to the other schemes where a failure can easily leave the cluster in an inconsistent state.

The main drawback of this approach is shared by all of the methods outlined here. If you have a slow link, it can take quite a while to transfer the entire site to each worker machine. Even on a fast link, the transfer is wasteful. The use of rsync mitigates this to some extent, but a more elegant solution still would be nice. Even something as simple as mirroring the staging server to a machine that is "next to" the cluster would be helpful for all of these options.

This article has demonstrated a method for updating a web site that is inspired by the page-flipping technique. Using this method we can update an arbitrarily sized cluster, with the following benefits:

I have been using this method on a production cluster since the middle of 2001 and have experienced no errors. In practice, even these rather simple scripts have been stable and reliable. This project demonstrates the value of building solutions on time-tested foundations like rsync and SSH.

If you have questions or comments, please contact me at minniger@minnigerode.com.