Xen Virtualization and Linux Clustering, Part 2

We ended last time after configuring our first unprivileged Xen domain. In this article, we complete our cluster and then test it using an open-source parallel ray tracer. The first thing we need to do is create additional slave nodes to be used with the cluster. So, let's get down to business.

Once you have created and configured an unprivileged domain using the instructions from the previous section, you easily can duplicate this domain as described below. Start by making a backup image of the Debian_Slave1_Root filesystem. You only need to do this step once. Note that the tar command probably will take a few minutes to complete, as it is archiving an entire filesystem.

   # mkdir /data/xen-images
   # mount /dev/VG/Debian_Slave1_Root /mnt/xen
   # cd /mnt/xen
   # tar jpcf /data/xen-images/debian-slave-root.tar.bz2 *
   # ls -sh /data/xen-images/debian-slave-root.tar.bz2
   # cd /
   # umount /mnt/xen

You now have a working root filesystem archive that can be used to create additional domains with minimal effort. Let's use this archive to create our second unprivileged domain, which we call debian_slave2. Start by creating the logical volumes for the additional domain:

   # lvcreate -L1024M -n Debian_Slave2_Root VG
   # lvcreate -L64M -n Debian_Slave2_Swap VG
   # mke2fs -j /dev/VG/Debian_Slave2_Root
   # mkswap /dev/VG/Debian_Slave2_Swap

   # mount /dev/VG/Debian_Slave2_Root /mnt/xen
   # cd /mnt/xen
   # tar jpxf /data/xen-images/debian-slave-root.tar.bz2

We now must modify two configuration files in the new domain root. This includes changing the IP address of eth0 in /mnt/xen/etc/network/interfaces to match the IP address we chose for the new domain--remember the Domain-0 /etc/hosts file we created earlier?. We also need to change the hostname in /mnt/xen/etc/hostname to debian_slave2.

Finally, create a Xen config file for the new domain and save it as debian_slave2.conf. Don't forget to give this domain a unique MAC address:

name="debian_slave2"
memory=64
kernel="/boot/vmlinuz-2.6.11-xenU"
nics=1
vif=[ 'mac=aa:00:00:00:00:02, bridge=xen-br0' ]
disk=[ 'phy:VG/Debian_Slave2_Root,sda1,w',
'phy:VG/Debian_Slave2_Swap,sda2,w' ]
root="/dev/sda1 ro"

You now are ready to boot the additional domain.

   # umount /mnt/xen
   # xm create /etc/xen/debian_slave2.conf -c

The archive technique presented above can be repeated to create as many Debian Sarge unprivileged domains as you like. In addition, this technique also can be used in a more general sense to create complete backups of an OS. This might be useful, for example, before performing software installations/upgrades or other tasks that potentially may harm your system. If things don't work out, simply restore the domain to a previously working state. In fact, I completely erased all of the files on one of my unprivileged domains and was able to restore the entire filesystem within five minutes.

At this point, you should have PVM installed on Domain-0 and be able to boot at least one slave domain that is completely configured as described in the previous sections. Although one slave is sufficient, the next few sections definitely are more interesting if you have two or more slaves configured.

The first step to configuring the PVM cluster is to create a pvm.hosts configuration file, which lists the hostnames of the cluster nodes that you want to use. Note that the specified hostnames should match the hostnames as listed in the /etc/hosts file on Domain-0. In turn, those hostnames should match the hostname in the /etc/hostname file on each domain. An example pvm.hosts file is shown below:

   # Master PVM Host
   master
   # Slaves
   debian_slave1
   debian_slave2
   debian_slave3

Lines beginning with # are comments. You can read the man page for pvmd3 for more details on the PVM configuration file. Your pvm.hosts config file now can be used to start PVM daemon processes (pvmd) on the master and all slave nodes. The PVM daemon provides the message passing interface that we discussed earlier. To start the PVM daemons on all nodes listed in the pvm.hosts file, use the command:

   # $PVM_ROOT/lib/pvm pvm.hosts

Before running this command, be sure that all of your slaves are booted using the xm create command. You can get a list of currently booted domains by running xm list.