Xen Virtualization and Linux Clustering, Part 1

Have you heard about Xen virtualization and want to get some hands-on
experience? Do you want to experiment with Linux clustering but only
have a single computer to devote to the cause? If you answered yes to
either of these questions, keep reading.

In this article, I briefly introduce the concepts of Xen virtualization
and Linux clustering. From there, I show you how to set up multiple
operating systems on a single computer using Xen and how to configure them
for use with clustering. I should point out that a cluster implemented
in this manner does not provide the computational power of multiple
physical computers. It does, however, offer a way to prototype a
cluster as well as provide a cost-effective development environment for
cluster-based software. Even if you're not interested in clustering,
this article gives you hands-on experience using Xen virtualization.
What is Xen Virtualization?
Xen is an open-source software project that provides
high-performance, resource-managed virtualization on the x86
processor architecture.

It allows multiple operating system
instances to run concurrently on a single physical computer.
Xen manages the computer's hardware resources so they are shared
effectively among the operating system instances, called domains.
So what does all this mean? It means that by using Xen virtualization,
you can run and use several different operating systems at the same
time on a single computer.

Xen supports a variety of operating systems, which increases its
applicability for running many popular applications on whichever OS
you desire. Xen currently supports Linux kernels through an approach
called paravirtualization in which the operating system is modified to
become aware of Xen. Other ports have been developed or are in progress
for other operating systems, such as NetBSD and FreeBSD. Intel and AMD
also are developing new virtualization technology in their processors
to allow an OS to run on Xen without modification. That means
even Microsoft Windows will be an OS option on Xen in the near future.

Xen virtualization provides many exciting benefits over traditional single
OS computers, including server consolidation, application mobility,
secure computing and research/testing. What sets Xen apart
from other virtualization technologies, however, is its performance. In fact,
running over Xen, Linux's performance in a variety of benchmarks is
consistently close to native Linux.

So how is a Xen system structured? Each Xen system has a single
privileged OS, called Domain-0, that is responsible for starting and
managing the other unprivileged OS instances. Domain-0 is the OS that
boots when you start your computer, and it has the tools necessary to
manage other domains.
What is Clustering?
Clustering essentially is the unification of multiple independent
computers into a single system through the use of software and networking. By
definition, a cluster must have at least two computers (called nodes), one
master and one slave. The master node typically has a job scheduler that
schedules the work to be done by the slave nodes. As the name implies,
the slave nodes perform whatever work has been scheduled for them to do.
The nodes of the cluster communicate using a message passing interface.
Two of the most commonly used message passing interfaces are PVM and MPI.
For this tutorial, we are using PVM 3.4 to implement clustering.

For the tutorial that follows, the master node is the privileged
Xen Domain-0 and the slave nodes are the unprivileged Xen Domains.
This approach allows us to create and use as many slave nodes as
we want.
Assumed Configuration
The first step to getting started with Xen is to install it.
This process involves several steps:

1. Install the required libraries.
2. Install and/or compile the Xen privileged/unprivileged
   kernels.
3. Configure the bootloader to boot
   Domain-0.
4. Set up unprivileged domains.

I do not cover these first three steps here, because
adequate documentation already is available on-line describing
this process. A good place to start is the

Xen User Manual.
In addition, a wide variety of tutorials and troubleshooting articles
are
available on-line. Be sure to find one that matches the Linux distribution
you are using as Domain-0. During the installation process, you need
to consider which Linux kernels are appropriate for your use. The Xen
binary release contains pre-compiled kernels for both the privileged
and unprivileged domains. If you require anything special that's not
supported by these kernels, you may need to compile a custom kernel
with Xen support. On my system, I compiled my own Domain-0 kernel but
used the stock kernel for the unprivileged domains. Another important
consideration for this tutorial is to leave an extra, unused partition
on your hard drive for later use with the unprivileged domains.

For the tutorial that follows, I assume you have Xen 2.0.7 installed and
can boot Domain-0. You also must be able to start xend, the Xen control
daemon, which must be running in order to start and manage other domains.

Xen 3.0 was released recently. If you are more daring, you
might consider installing that version instead, although you may not be
able to follow along with this article as easily. I also assume that your Domain-0 is a
Debian-based distribution. If not, you simply need to use whatever
mechanism your distribution provides for downloading and installing
packages, typically where the apt-get command is used below.
Configuring Domain-0
Now that we have all the formalities out of the way, let's get started
by configuring Domain-0. The first thing we need to do is install PVM
3.4.5 on our master node.

Installing PVM on the Master Node

1. Remove any existing PVM installation to ensure we are working
   with a fresh install:

   
         # apt-get remove pvm xpvm pvm-dev libpvm3
         # rm -R /usr/lib/pvm3
   
   

2. Set the $PVM_ROOT environment variable to /usr/local/pvm3. If you
   are using the bash shell, this can be done by adding the following two
   lines to the $HOME/.bashrc file:

      
   
      PVM_ROOT=/usr/local/pvm3 
      export PVM_ROOT
   
   

   Next, apply and verify your changes:

   
         # source $HOME/.bashrc
         # echo $PVM_ROOT
   
   

3. Download and extract PVM:
   

   
         # cd /usr/local
         # wget http://www.netlib.org/pvm3/pvm3.4.5.tgz
         # tar xvfz pvm3.4.5.tgz
         # rm pvm3.4.5.tgz
         # cd pvm3
   
   

4. Set up the use of SSH instead of rsh by editing
   /usr/local/pvm3/conf/LINUX.def, replacing
   /usr/bin/rsh with
   /usr/bin/ssh
5. Compile PVM from /usr/local/pvm3 by typing
   make
6. Update /etc/hosts to include IP addresses of the master and all slaves.
   The IP addresses you use here are used later when configuring
   the slaves. An example /etc/hosts file is shown below:

   
         127.0.0.1 localhost
         192.168.0.200 master
         192.168.0.201 debian_slave1
         192.168.0.202 debian_slave2
         192.168.0.203 debian_slave3
   
   

If you run into any challenges during this process, the $PVM_ROOT/Readme
file contains a lot of useful information.
Creating Your First Unprivileged Xen Domain
Now that we have Domain-0 configured, we can begin installing
and configuring the unprivileged Xen domains. We start by
creating a minimal Debian Sarge domain as a template for all future
unprivileged domains, which will save us a considerable amount of time
down the road.
The filesystems for the unprivileged domains should be set up as 1GB LVM
volumes in a volume group named VG. Now, you may be thinking "What
is LVM?" LVM stands for logical volume management, and it allows you to
create, delete and resize multiple logical volumes without having to
repartition your hard drive, which is extremely useful for our situation.
For the sections that follow, you need to have LVM2 installed.
Go ahead and try running the following commands. If they aren't
available, install LVM2 using apt-get install
lvm2.

Creating the Filesystems

This is where the extra hard drive partition comes into play. We
start by initializing the extra partition, call it /dev/hda6, for use
by LVM. We then create a new volume group on that partition:

# pvcreate /dev/hda6
# vgcreate VG /dev/hda6

You can check the man pages for the pvcreate and vgcreate commands to
find out what's going on behind the scenes. We now are ready to create
the filesystems for our first unprivileged domain. The naming scheme
I use is Debian_Slave#_Root and Debian_Slave#_Swap, where # is replaced
with the number of the slave and Root and Swap denote the root and swap
filesystems, respectively. So, let's create and initialize the root
and swap filesystems:

# lvcreate -L1024M -n Debian_Slave1_Root VG
# lvcreate -L64M -n Debian_Slave1_Swap VG
# mke2fs -j /dev/VG/Debian_Slave1_Root
# mkswap /dev/VG/Debian_Slave1_Swap

The first lvcreate command creates a logical volume in the existing "VG"
volume group with a size of 1,024MB and a name of "Debian_Slave1_Root".
An ext3 filesystem is created on this volume using the mke2fs command.
The second lvcreate command creates a smaller logical volume that is
set up as a swap area using the mkswap command.

Now that the filesystems
are created and initialized, we can mount the new root filesystem at
/mnt/xen and install a minimal Debian Sarge root filesystem on it.
The debootstrap utility allows you to do exactly this:

# apt-get install debootstrap
# mkdir /mnt/xen
# mount /dev/VG/Debian_Slave1_Root /mnt/xen
# debootstrap --arch i386 sarge /mnt/xen http://www.uk.debian.org/debian

Now we need to configure certain files manually in the new filesystem,
which should be mounted at /mnt/xen. The files that need to be modified
are listed below, followed by the text they should contain:

1. /mnt/xen/etc/fstab

   
         /dev/sda1     /      ext3     defaults     0     1
         /dev/sda2     swap   swap     defaults     0     0
         proc          /proc  proc     defaults     0     0
   
   

2. /mnt/xen/etc/hostname
   

         debian_slave1
   

3. /mnt/xen/etc/hosts
   

         127.0.0.1   localhost
   

4. /mnt/xen/etc/network/interfaces (be sure to modify these settings
   based on your own network configuration)

   
         auto lo iface lo inet loopback
   
         auto eth0
         iface eth0 inet static
            address 192.168.0.201
            netmask 255.255.255.0
            gateway 192.168.0.1
            dns-nameservers 192.168.0.1
   
   

5. /mnt/xen/etc/apt/sources.list
   

   
         deb ftp://ftp.debian.org/debian sarge main contrib non-free
         deb ftp://non-us.debian.org/debian-non-US sarge/non-US main contrib non-free
         deb ftp://security.debian.org/debian-security sarge/updates main contrib non-free
   
   

   Finally, rename /mnt/xen/lib/tls as /mnt/xen/lib/tls.disabled and unmount /mnt/xen:

   
         # mv /mnt/xen/lib/tls /mnt/xen/lib/tls.disabled
         # umount /mnt/xen
   
   

Creating the Domain Config File

The unprivileged domain now must have a Xen configuration file created so
Xen knows how to boot it. A sample configuration file is shown below.
The actual values may be adjusted as necessary to reflect your own
system and network configuration. You can find more details about the
domain configuration file and the available options in the
Xen
User Manual.
I save my configuration files in the /etc/xen directory. The following
file is saved as debian_slave1.conf:

name="debian_slave1"
memory=64
kernel="/boot/vmlinuz-2.6.11-xenU"
nics=1
vif=[ 'mac=aa:00:00:00:00:01, bridge=xen-br0' ]
disk=[ 'phy:VG/Debian_Slave1_Root,sda1,w', 'phy:VG/Debian_Slave1_Swap,sda2,w' ]
root="/dev/sda1 ro"

Booting the Slave

The minimal Debian Sarge domain now is ready to boot. If your Domain-0
currently is taking up all RAM on the machine, you may need to free
some RAM for use with your new domain before booting it. Type
xm
help balloon for information on how to do this. Alternately, you can
modify your bootloader configuration to start Domain-0 using a specified
amount of RAM. The Xen User Manual shows how to do this for
GRUB.
Once that's taken care of, boot the new domain using the command:

# xm create /etc/xen/debian_slave1.conf -c

The xm tool provides a command-line interface for managing Xen. In this
case, it boots the new Debian Sarge OS for the first time, turning
the current console session into the console of the debian_slave1 domain.
Log in as root and change the root password using the passwd command.

Verifying Network Connectivity

You now should verify your network connectivity from the unprivileged
domain's console. If you encounter errors at any of these steps, you
need to troubleshoot the network configuration for the unprivileged
domain before proceeding.

Start by trying to ping Domain-0 using its IP address, for example,
ping 192.168.0.200. If Domain-0 has a firewall enabled, you may need
to disable it or come up with a workaround. Next, check Internet connectivity,
which is needed to install additional required packages, for example,
ping 4.2.2.1.

Finally, check whether DNS resolution is working correctly, for
example, ping www.google.com. On my machine, I received an unknown
host error message. I found that this can be resolved by running the
command apt-get install bind. Unfortunately, without DNS resolution
working, apt-get is not able to download and install the package
using the Web sites currently specified in the /etc/apt/sources.list file.
To work around this, manually look up the IP address for ftp.debian.org and
use it in the following command to download and install the BIND package:

      # wget ftp://128.101.80.133/debian/pool/main/b/bind/bind_8.4.6-1_i386.deb
      # dpkg -i bind_8.4.6-1_i386.deb

Installing PVM

You now are ready to install PVM on the slave. From the debian_slave1
console, perform the following steps:

1. Install the packages necessary for compiling PVM:

      # apt-get update
      # apt-get install make build-essential m4

2. Compile and install PVM using the same process described above for
the master node (in the section "Installing PVM on the Master Node").

3. Install an SSH client and sshd server:

      # apt-get install ssh

The SSH configuration asks the following questions. Answer them as
indicated below:

• Allow SSH Protocol 2 only? NO
• Do you want /usr/lib/ssh-keysign to be installed
  SUID root? YES
• Do you want to run the sshd server?
  YES

4. Set up SSH to accept connections from the master without a password:

• On the debian_slave1 console:

  
        # cd $HOME
        # mkdir .ssh
  
  

• On the master console:
  

  
        # cd $HOME
        # ssh-keygen -t dsa -f .ssh/id_dsa
        Leave the password blank (just press Enter)
        # cd .ssh
        There should be 2 new files in this directory: id_dsa and id_dsa.pub
        Copy the public key to the slave:
        # scp id_dsa.pub root@debian_slave1:~/.ssh/id_dsa.pub
        Enter the root password for the slave
        Connect to the slave using ssh:
        # ssh root@debian_slave1
        Using the remote ssh connection, add the master's public key to the known public keys on the slave:
        (remote)# cd .ssh
        (remote)# cat id_dsa.pub >> authorized_keys2
        (remote)# chmod 640 authorized_keys2
        (remote)# rm id_dsa.pub
        (remote)# exit
  
  

  Now, SSH will not prompt for a password when logging into the slave from the master.
  This will be useful later on.

At this point, you have completed the construction of your first unprivileged domain.
Congratulations! In the next installment of this article, we'll look at
how we easily can create additional unprivileged domains using the one we
just created as a template. We'll then move on to configuring
the PVM cluster and testing it using an open-source parallel ray tracer.

Ryan Mauer is a Computer Science graduate student at Eastern Washington
University. In addition to Xen virtualization, he also dabbles in 3-D
computer graphics programming as he attempts to finish his Master's
thesis.