Virtualize a Server with Minimal Downtime

When it's time to convert a physical machine to a virtual one, use these steps to make the move safely and with a small maintenance window.

You don't have to look far in tech news to see that virtualization has become a big deal. After all, computers continue to become faster and more powerful, and as they do, the services that run on them often use fewer overall resources. On top of that, modern servers often need a fraction of their predecessors' power and cooling. With virtualization, you can get power savings and more efficient use of server resources, and you can create servers quickly without waiting on parts to arrive.

Although some people start from a clean slate and create brand-new virtual servers from scratch to replace old physical machines, you simply may not have the extra time to invest to make the clean break to new versions of Linux with all-new packages. In many cases, it makes more sense to create a virtual clone of a physical server and keep all the files and services identical. If you want to go this route, there are a number of different methods you can use if you can handle unlimited downtime, but if you want to keep downtime and overall disruption to a minimum (and who doesn't?), there is a fairly simple process I've used to migrate a large number of servers to virtual machines with around 30 minutes of average downtime—well within most acceptable maintenance windows.

Caveats and Limitations

Before I get into the actual steps, there are some limitations to this approach that I should mention. First, this method has been designed and tested to work with VMware virtualization, specifically with its enterprise server products (although it would also work fine with their free server product). VMware works well for this process because it doesn't require that I modify my current Linux kernel to virtualize it—something that isn't always possible when you want to virtualize an old server. Having said that, these steps also could work with other virtual machine technologies that can use an unmodified Linux kernel. Second, this procedure has been tested with and is aimed at Red Hat-like distributions (Red Hat, CentOS and so on), but with a few tweaks I discuss later, it also could work with other distribution flavors. Finally, the actual amount of downtime you will need for this process probably will vary from my results, especially as you first test out each of the steps. Servers with large or slow disks and, specifically, servers that change large amounts of data frequently possibly will take longer to sync.

Along with these disclaimers, it's only fair to point out some of the benefits to this method:

  • You can do a majority of the migration work against a live server.

  • Standard Linux tools are used for the synchronization and other changes.

  • The process protects your network from both servers showing up at the same time.

  • You safely can leave the old server on the network and access its files while users use the new virtual machine.

Now that the disclaimers are out of the way, let me summarize the process in a few general steps. First, create a fresh virtual machine to replace the physical host. Then, boot in to the virtual machine with a rescue CD and partition the disk. Next, perform the main synchronization of files from the live physical server to the new virtual machine. Take down your physical machine and reboot in to a rescue CD, and perform a final synchronization from the off-line server. After that, change the boot settings on the virtual machine to suit its new environment, and then reboot in to your new virtual machine.

Create the Virtual Machine

To get started, first you must create a virtual machine container to replace your physical machine. Specific steps are different if you use VMware Server versus Virtual Infrastructure 3, but ultimately, what you want to do is to create a machine that mostly matches your physical machine's specifications. The specifications don't have to match exactly, and there actually are good reasons why you might want to tweak the settings a bit. For instance, if your server has 2GB of RAM, but you notice that it really needs only 1GB, now would be a good opportunity to change it. If your server is starting to run out of storage, this is a good time to increase it. If your physical server has a 32-bit or 64-bit processor, however, make sure the virtual machine matches. Also, be sure that you match the operating system version you report to VMware with your actual OS if possible. For instance, if your server runs RHEL 3, don't tell VMware that it runs RHEL 4. You want to ensure that the OS will have drivers for the virtual devices that VMware presents, specifically for the disk subsystem. For instance, I've had numerous headaches due to RHEL 4's removal of the BusLogic SCSI module from the base OS (a virtual SCSI device that is a commonly used by VMware along with an LSI Logic virtual SCSI device).

After you set the specs for the virtual machine, edit the CD-ROM device so that it points either to an actual rescue CD in the VMware server or to an ISO. I prefer Knoppix for this procedure, but any live CD should work as long as it has the rsync and chroot tools, an SSH server and enough module support to access the disks on both the physical machine and the virtual machine. Now, boot the virtual machine into the rescue CD. Everything you need to do is done via the command line, so under Knoppix, type knoppix 2 at the boot: prompt to bypass the GUI and go straight to a command line.

______________________

Kyle Rankin is a director of engineering operations in the San Francisco Bay Area, the author of a number of books including DevOps Troubleshooting and The Official Ubuntu Server Book, and is a columnist for Linux Journal.

Comments

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

Older Linux ext2/3 filesystem

Paul Schilling's picture

I was able to virtualize two older RedHat system using this method though I did run into ext2/3 filesystem issues. One system was an old RedHat 5.2 (circa 1998). I was booting the VM with Knoppix V6.0 live CD and then creating the partitions and ext2 filesystem. After rsync'ing files I found I couldn't boot. This turned out to be ext2 filesystem features. The Knoppix OS created an ext2 with all the latest features but the old RedHat 5.2 predated many of these features so it didn't know how to handle them.

I rebuilt the ext2 FS with options to turn off just about every feature.
mkfs -t ext2 -O none -r0 -O ^dir_index /dev/sdxx

Then I was able to rsync and complete all steps ending with a bootable RedHat 5.2 guest VM

Thanks for the great article.

Ubuntu 8 2.6.24 kernel I had

Pkilpo's picture

Ubuntu 8 2.6.24 kernel

I had to use "mount --bind /dev /mnt/sda1"
then chroot /mnt/sda1
then mount /proc /proc -t proc
and used mkinitramfs -o /boot/initrd.img

Filesystem has unsupported features

Len Kranendonk's picture

After migrating an old RedHat 9 server to VMWare using this howto, I got an error saying fsck.ext3: Filesystem has unsupported features.

I've posted a description on how to solve this issue on my blog: fsck: Filesystem has unsupported features.

I hope this might be helpful for people encountering the same issue.

nodev

Johannes Wolter's picture

I ran into problems because Knoppix mounted my root partition with the nodev option, which made the grub-install and mkinitrd calls fail, because the devices were not accessible. I had to change the mount-options with mount -oremount,dev,setuid /dev/sda1.
Furthermore I had a problem because /var is on a different partition (/dev/sda3)in my setup. I had to unmount this partition before chroot-ing and had to (re-)mount in the chroot-environment.

grub install issue

gc's picture

I'm having some issue with grub-install, mainly the fact that the system I'm virtualizing has a raid device for the hard drive. I've tried the grub install using some soft links to map the device names and the setup (hd0) command appears to complete but the virt-manager states the disk is not bootable

any suggestions

I had similar issues because

spingary's picture

I had similar issues because my set up has a /boot partition:

/dev/sda1 -> /boot
/dev/sda2 -> /
/dev/sda3 -> (swap)

so chroot'ing into my root, /dev/sda2, won't exactly work right since /boot is not there. Here's what I did to get it to work, using manual grub instead of grub-install:

Before you chroot into anything (while in Knoppix shell):

$ grub
grub> root (hd0,0)
grub> setup (hd0)
grub> exit

The 1st line points root to my first partition (/dev/sda1)
2nd line sets it up. What this does is set up grub on the MBR of the virtual disk.

Then, to rebuild the initrd correctly, I had to do this:

$ umount /mnt/sda1

$ mount -o remount,dev /mnt/sda2
(assuming sda2 was already mounted)
This mount command helps eliminate /dev/null problems when using the mkinitrd command later.

$ mount /dev/sda1 /mnt/sda2/boot
(This puts the boot partition into my sda2 root mount)

Now we can chroot:
$ chroot /dev/sda2
$ ls /boot
(should see the /boot partition stuff)

I had to add these to /etc/modprobe.conf in my install (I am using VMWare server 2.0.0 RC1):

alias scsi_hostadapter mptbase
alias scsi_hostadapter1 mptscsih

Now I rebuilt initrd:

Now we do the equivalent of this:

# mv /boot/initrd-2.4.21-32.0.1.ELsmp.img /boot/initrd-2.4.21-32.0.1.ELsmp.img.bak
# mkinitrd -v /boot/initrd-2.4.21-32-0.1.ELsmp.img 2.4.21-32-0.1.ELsmp
(Notice that the orig article was missing the .img in the second line)
(Also use the -v flag to see any useful errors)

I had one last problem - when I rebooted it complained the my e2fsck was too old and keep dumping me into repair mode. My install is a real old FC2, and when I used Knoppix to create the ext3 file systems, there are "new features" that the older e2fsck didn't support. So I followed these instructions and all is good:

http://www.mail-archive.com/trilug%40trilug.org/msg26479.html

Good luck everyone. Thank you for a great guide!

I had a problem...

Enrique Garcia's picture

Hi,

Before the second and final rsync, I wanted to test if the machines goes alive, but when I do the chroot /mnt/sda, and then
# grub-install /dev/sda
it gives:
/usr/share/grub/i386-redhat/stage1: Not found

And tried:
fdisk -l (without result)

It seems that I can't have access to /var /usr /home which are other partitions

What I tried next, was to cp the files in /mnt/sda3 (/usr), to /mnt/sda1 (/) just the files in the corresponding /usr/share/grub/i386-redhat/

And the output of that command was:
/sbin/grub-install: line 501: uniq: command not found
/dev/sda: Not found or not a block device.

any advice will be greatly appreciated.

Thanks in advance.

Even though

Enrique Garcia's picture

Even though the missing -H option, this is a great HOWTO, include I will be using this to migrate from one server to other and another, obviously I had never figure it out by myself... Thank you very much, And if possible, I would like to have your agreement to translate it, (and include fdisk steps) to Spanish.

Great!!!

rsync and hardlinks

Thomas Mueller's picture

the rsync options "-avx --numeric-ids --progress" IMHO doesn't include "-H" (hardlinks). So if a systems uses many hardlinks there will be many duplicated files.

White Paper
Linux Management with Red Hat Satellite: Measuring Business Impact and ROI

Linux has become a key foundation for supporting today's rapidly growing IT environments. Linux is being used to deploy business applications and databases, trading on its reputation as a low-cost operating environment. For many IT organizations, Linux is a mainstay for deploying Web servers and has evolved from handling basic file, print, and utility workloads to running mission-critical applications and databases, physically, virtually, and in the cloud. As Linux grows in importance in terms of value to the business, managing Linux environments to high standards of service quality — availability, security, and performance — becomes an essential requirement for business success.

Learn More

Sponsored by Red Hat

White Paper
Private PaaS for the Agile Enterprise

If you already use virtualized infrastructure, you are well on your way to leveraging the power of the cloud. Virtualization offers the promise of limitless resources, but how do you manage that scalability when your DevOps team doesn’t scale? In today’s hypercompetitive markets, fast results can make a difference between leading the pack vs. obsolescence. Organizations need more benefits from cloud computing than just raw resources. They need agility, flexibility, convenience, ROI, and control.

Stackato private Platform-as-a-Service technology from ActiveState extends your private cloud infrastructure by creating a private PaaS to provide on-demand availability, flexibility, control, and ultimately, faster time-to-market for your enterprise.

Learn More

Sponsored by ActiveState