Xen Virtualization and Linux Clustering, Part 1

Interested in seeing what all the fuss is about Xen? Use the steps in this tutorial to get Xen installed and domains configured.
Configuring Domain-0

Now that we have all the formalities out of the way, let's get started by configuring Domain-0. The first thing we need to do is install PVM 3.4.5 on our master node.

Installing PVM on the Master Node

  1. Remove any existing PVM installation to ensure we are working with a fresh install:

          # apt-get remove pvm xpvm pvm-dev libpvm3
          # rm -R /usr/lib/pvm3
  2. Set the $PVM_ROOT environment variable to /usr/local/pvm3. If you are using the bash shell, this can be done by adding the following two lines to the $HOME/.bashrc file:

       export PVM_ROOT

    Next, apply and verify your changes:

          # source $HOME/.bashrc
          # echo $PVM_ROOT
  3. Download and extract PVM:

          # cd /usr/local
          # wget http://www.netlib.org/pvm3/pvm3.4.5.tgz
          # tar xvfz pvm3.4.5.tgz
          # rm pvm3.4.5.tgz
          # cd pvm3
  4. Set up the use of SSH instead of rsh by editing /usr/local/pvm3/conf/LINUX.def, replacing /usr/bin/rsh with /usr/bin/ssh

  5. Compile PVM from /usr/local/pvm3 by typing make

  6. Update /etc/hosts to include IP addresses of the master and all slaves. The IP addresses you use here are used later when configuring the slaves. An example /etc/hosts file is shown below:


If you run into any challenges during this process, the $PVM_ROOT/Readme file contains a lot of useful information.

Creating Your First Unprivileged Xen Domain

Now that we have Domain-0 configured, we can begin installing and configuring the unprivileged Xen domains. We start by creating a minimal Debian Sarge domain as a template for all future unprivileged domains, which will save us a considerable amount of time down the road. The filesystems for the unprivileged domains should be set up as 1GB LVM volumes in a volume group named VG. Now, you may be thinking "What is LVM?" LVM stands for logical volume management, and it allows you to create, delete and resize multiple logical volumes without having to repartition your hard drive, which is extremely useful for our situation. For the sections that follow, you need to have LVM2 installed. Go ahead and try running the following commands. If they aren't available, install LVM2 using apt-get install lvm2.

Creating the Filesystems

This is where the extra hard drive partition comes into play. We start by initializing the extra partition, call it /dev/hda6, for use by LVM. We then create a new volume group on that partition:

# pvcreate /dev/hda6
# vgcreate VG /dev/hda6

You can check the man pages for the pvcreate and vgcreate commands to find out what's going on behind the scenes. We now are ready to create the filesystems for our first unprivileged domain. The naming scheme I use is Debian_Slave#_Root and Debian_Slave#_Swap, where # is replaced with the number of the slave and Root and Swap denote the root and swap filesystems, respectively. So, let's create and initialize the root and swap filesystems:

# lvcreate -L1024M -n Debian_Slave1_Root VG
# lvcreate -L64M -n Debian_Slave1_Swap VG
# mke2fs -j /dev/VG/Debian_Slave1_Root
# mkswap /dev/VG/Debian_Slave1_Swap

The first lvcreate command creates a logical volume in the existing "VG" volume group with a size of 1,024MB and a name of "Debian_Slave1_Root". An ext3 filesystem is created on this volume using the mke2fs command. The second lvcreate command creates a smaller logical volume that is set up as a swap area using the mkswap command.

Now that the filesystems are created and initialized, we can mount the new root filesystem at /mnt/xen and install a minimal Debian Sarge root filesystem on it. The debootstrap utility allows you to do exactly this:

# apt-get install debootstrap
# mkdir /mnt/xen
# mount /dev/VG/Debian_Slave1_Root /mnt/xen
# debootstrap --arch i386 sarge /mnt/xen http://www.uk.debian.org/debian

Now we need to configure certain files manually in the new filesystem, which should be mounted at /mnt/xen. The files that need to be modified are listed below, followed by the text they should contain:

  1. /mnt/xen/etc/fstab

          /dev/sda1     /      ext3     defaults     0     1
          /dev/sda2     swap   swap     defaults     0     0
          proc          /proc  proc     defaults     0     0
  2. /mnt/xen/etc/hostname

  3. /mnt/xen/etc/hosts

  4. /mnt/xen/etc/network/interfaces (be sure to modify these settings based on your own network configuration)

          auto lo iface lo inet loopback
          auto eth0
          iface eth0 inet static
  5. /mnt/xen/etc/apt/sources.list

          deb ftp://ftp.debian.org/debian sarge main contrib non-free
          deb ftp://non-us.debian.org/debian-non-US sarge/non-US main contrib non-free
          deb ftp://security.debian.org/debian-security sarge/updates main contrib non-free

    Finally, rename /mnt/xen/lib/tls as /mnt/xen/lib/tls.disabled and unmount /mnt/xen:

          # mv /mnt/xen/lib/tls /mnt/xen/lib/tls.disabled
          # umount /mnt/xen

Creating the Domain Config File

The unprivileged domain now must have a Xen configuration file created so Xen knows how to boot it. A sample configuration file is shown below. The actual values may be adjusted as necessary to reflect your own system and network configuration. You can find more details about the domain configuration file and the available options in the Xen User Manual. I save my configuration files in the /etc/xen directory. The following file is saved as debian_slave1.conf:

vif=[ 'mac=aa:00:00:00:00:01, bridge=xen-br0' ]
disk=[ 'phy:VG/Debian_Slave1_Root,sda1,w', 'phy:VG/Debian_Slave1_Swap,sda2,w' ]
root="/dev/sda1 ro"

Booting the Slave

The minimal Debian Sarge domain now is ready to boot. If your Domain-0 currently is taking up all RAM on the machine, you may need to free some RAM for use with your new domain before booting it. Type xm help balloon for information on how to do this. Alternately, you can modify your bootloader configuration to start Domain-0 using a specified amount of RAM. The Xen User Manual shows how to do this for GRUB. Once that's taken care of, boot the new domain using the command:

# xm create /etc/xen/debian_slave1.conf -c

The xm tool provides a command-line interface for managing Xen. In this case, it boots the new Debian Sarge OS for the first time, turning the current console session into the console of the debian_slave1 domain. Log in as root and change the root password using the passwd command.

Verifying Network Connectivity

You now should verify your network connectivity from the unprivileged domain's console. If you encounter errors at any of these steps, you need to troubleshoot the network configuration for the unprivileged domain before proceeding.

Start by trying to ping Domain-0 using its IP address, for example, ping If Domain-0 has a firewall enabled, you may need to disable it or come up with a workaround. Next, check Internet connectivity, which is needed to install additional required packages, for example, ping

Finally, check whether DNS resolution is working correctly, for example, ping www.google.com. On my machine, I received an unknown host error message. I found that this can be resolved by running the command apt-get install bind. Unfortunately, without DNS resolution working, apt-get is not able to download and install the package using the Web sites currently specified in the /etc/apt/sources.list file. To work around this, manually look up the IP address for ftp.debian.org and use it in the following command to download and install the BIND package:

      # wget
      # dpkg -i bind_8.4.6-1_i386.deb

Installing PVM

You now are ready to install PVM on the slave. From the debian_slave1 console, perform the following steps:

1. Install the packages necessary for compiling PVM:

      # apt-get update
      # apt-get install make build-essential m4

2. Compile and install PVM using the same process described above for the master node (in the section "Installing PVM on the Master Node").

3. Install an SSH client and sshd server:

      # apt-get install ssh

The SSH configuration asks the following questions. Answer them as indicated below:

  • Allow SSH Protocol 2 only? NO

  • Do you want /usr/lib/ssh-keysign to be installed SUID root? YES

  • Do you want to run the sshd server? YES

4. Set up SSH to accept connections from the master without a password:

  • On the debian_slave1 console:

          # cd $HOME
          # mkdir .ssh
  • On the master console:

          # cd $HOME
          # ssh-keygen -t dsa -f .ssh/id_dsa
          Leave the password blank (just press Enter)
          # cd .ssh
          There should be 2 new files in this directory: id_dsa and id_dsa.pub
          Copy the public key to the slave:
          # scp id_dsa.pub root@debian_slave1:~/.ssh/id_dsa.pub
          Enter the root password for the slave
          Connect to the slave using ssh:
          # ssh root@debian_slave1
          Using the remote ssh connection, add the master's public key to the known public keys on the slave:
          (remote)# cd .ssh
          (remote)# cat id_dsa.pub >> authorized_keys2
          (remote)# chmod 640 authorized_keys2
          (remote)# rm id_dsa.pub
          (remote)# exit

    Now, SSH will not prompt for a password when logging into the slave from the master. This will be useful later on.

At this point, you have completed the construction of your first unprivileged domain. Congratulations! In the next installment of this article, we'll look at how we easily can create additional unprivileged domains using the one we just created as a template. We'll then move on to configuring the PVM cluster and testing it using an open-source parallel ray tracer.

Ryan Mauer is a Computer Science graduate student at Eastern Washington University. In addition to Xen virtualization, he also dabbles in 3-D computer graphics programming as he attempts to finish his Master's thesis.



Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

how can i do virtualization on a cluster of 3 physical computer

Ajay vijayan's picture

i want to know weather this is possible for more than 2 physical cpu or a cluster of 2 to 3 computers

How Tos for Xen-Cluster online in German!

Anonymous's picture

If you're looking for a German Howto for a Xen-Cluster virtualization, take a look at http://www.thomas-krenn.com/de/wiki/Kategorie:Xen

application in SAN storage environment

Anonymous's picture

Whats the possible use of virtualisation in a storage environment?

application in SAN storage environment

Anonymous's picture

Whats the possible use of virtualisation in a storage environment?

High availability

Corey's picture

If you have a highly available cluster running over a shared filesystem such as GFS or Lustre etc. You can virtualize your nodes so that when a node needs to come down for maintainence or whatever, you can simply migrate the "virtual" node to other hardware, possible one that already has an instance running. XEN domain migration is very fast, less than 100ms from what I understand. It becomes a lot easier to attain "Five 9's" of uptime or better.

less than 100ms downtime,

miguelon's picture

less than 100ms downtime, that's awesome, I've done it and its true, if you continuosly ping a domain and migrate it, you only lose one of them.

nice howto, thank you!

Anonymous's picture

I have but one gripe: what is it with the page width? I'm at 1280x1024, and the page is still wider than my screen! Like, crazy, man.


Anonymous's picture

I wouldn't worry too much about missing some advertisements ;-)

page width

michael's picture

I have the same resolution, Displays fine.

ah, I should have tried more

Anonymous's picture

ah, I should have tried more browsers. It's a Konqueror problem, firefox and opera look fine.

Looks bad in Safari too

Anonymous's picture

You guys should test your pages in a few browsers. It comes out super wide for me, very irritating.

Thanks for the article though.

Safari and Konqueror

Albert Peschar's picture

Yeah, Safari and Konqueror both use about the same engine.