Virtualize a Server with Minimal Downtime

When it's time to convert a physical machine to a virtual one, use these steps to make the move safely and with a small maintenance window.
Partition the Virtual Machine's Disk

After Knoppix boots, you need to partition, format and mount the new partitions for this virtual machine. Use fdisk or cfdisk from the command line to create your partitions to match your physical server. Again, you don't have to match the partition sizes exactly, as long as there is plenty of room to store all the files from the physical server. For this example, I will have a physical server with a single SCSI drive (/dev/sda) with three partitions: /dev/sda1 for root, /dev/sda2 for swap and /dev/sda3 for /home. After you create the same partitions on the virtual machine, format them with the same filesystems you use on the physical machine, create mountpoints for them and then mount them:

$ sudo mkfs -t ext3 /dev/sda1
$ sudo mkfs -t ext3 /dev/sda3
$ sudo mkswap /dev/sda2
$ sudo mkdir -p /mnt/sda1 /mnt/sda3
$ sudo mount /dev/sda1 /mnt/sda1
$ sudo mount /dev/sda3 /mnt/sda3

First Sync

Now that you have created and mounted the partitions, you are ready for the first synchronization. For this to work, your virtual machine must have network access, and specifically, it needs to be able to access SSH on the physical machine. By default, Knoppix will attempt to get a DHCP lease if available, but otherwise, if your rescue disc is not able to get on the network, you need to make the necessary changes so that it can. This virtualization procedure reduces downtime by synchronizing the files twice—once while the physical server is running and once after it is off-line. The idea here is that a majority of files on most servers stay the same, at least over one or two days. If you perform the bulk of the file synchronization while the server is on-line, when you take it off-line, the final synchronization can occur much faster.

I use rsync for the synchronization, and for it to work, you need to allow (at least temporarily) for root SSH logins to occur on the physical machine. If it is disabled, edit /etc/ssh/sshd_config and change PermitRootLogin no to PermitRootLogin yes, and restart sshd. Otherwise, it will be difficult for rsync to copy all the files on the system. You will run an rsync command for each partition on the physical server, so in this example, that makes two rsync commands:

$ sudo rsync -avx --numeric-ids 
 ↪--progress physicalhost:/ /mnt/sda1/
$ sudo rsync -avx --numeric-ids 
 ↪--progress physicalhost:/home/ /mnt/sda3/

The rsync options I use here are chosen very deliberately, so it's worth understanding what each of them does. The -a option sets “archive mode”, which essentially turns on a number of rsync options that preserve file ownership and permissions and other settings. The -v option makes rsync provide more output about what it is doing, and the --progress argument displays a progress meter so you can keep up with how long rsync will take. The other two arguments are rather important, and if you don't use rsync regularly, you might not come across them much. The -x argument tells rsync to stick to one filesystem. This is important particularly when you back up the / partition; otherwise, rsync happily will traverse into /home or any other partitions you have and copy them all into your local /mnt/sda1 mountpoint, which probably will not have enough space to hold everything. The --numeric-ids argument sets file permissions on the destination files based on their numeric ID and not the matching user or group name. This is important as the Knoppix CD very likely has different user and group ID mappings than your server.

After these rsync commands complete, you are ready to take your physical server off-line. If you did need to schedule a maintenance window for the physical server, just leave the virtual machine running in its current state, and proceed to the next step when you are ready to take the physical machine off-line. If a number of days will pass until your maintenance window, you might want to run the above rsync commands again once you are close to the maintenance window, just so the final off-line rsync will happen more quickly.

Second and Final Sync

On the Physical Server:

The last synchronization happens when the physical server is completely off-line, so you can make sure that no other files change on you. To do this, simply take a Knoppix CD (or your preferred rescue CD) to the physical machine and boot from it. All the commands you run will be from the command line, so you can boot in to Knoppix's terminal-only mode here as well. As Knoppix boots, it should detect your partitions automatically and create mountpoints under /mnt for them, but if it doesn't, just use the mkdir command to create them manually. Knoppix will not mount partitions automatically at boot, so you need to do that manually. In the case of this example, my physical server has two partitions to mount:

$ sudo mount /dev/sda1 /mnt/sda1
$ sudo mount /dev/sda3 /mnt/sda3

Now I need to set a password for the root Knoppix user and then start the SSH server on this machine so I can run the rsync:

$ sudo passwd
$ sudo /etc/init.d/ssh start

Keep in mind that because I booted this machine into Knoppix, it most likely has gotten a different IP address via DHCP. Type /sbin/ifconfig to check which IP address the machine currently has, as you will need it for the final rsync.

On the Virtual Server:

You now can start the final synchronization from the virtual server. The commands are very similar to what you used before, except this time, I add the --delete option so that rsync will remove any files on the virtual machine that were deleted from the physical machine since the last time I synced. Also notice that because the physical server is now booted in to Knoppix, I have to change the directory paths and the IP address for the remote host, as they changed since I booted in to Knoppix:

$ sudo rsync -avx --numeric-ids --progress 
 ↪--delete 192.168.1.150:/mnt/sda1/ /mnt/sda1/
$ sudo rsync -avx --numeric-ids --progress 
 ↪--delete 192.168.1.150:/mnt/sda3/ /mnt/sda3/

These commands could take a long time or a short time, depending on how many files have changed since the last time you ran rsync. Once it completes, you are ready to perform the final finishing touches on your virtual machine before bringing it into service.

______________________

Kyle Rankin is a director of engineering operations in the San Francisco Bay Area, the author of a number of books including DevOps Troubleshooting and The Official Ubuntu Server Book, and is a columnist for Linux Journal.

Comments

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

Older Linux ext2/3 filesystem

Paul Schilling's picture

I was able to virtualize two older RedHat system using this method though I did run into ext2/3 filesystem issues. One system was an old RedHat 5.2 (circa 1998). I was booting the VM with Knoppix V6.0 live CD and then creating the partitions and ext2 filesystem. After rsync'ing files I found I couldn't boot. This turned out to be ext2 filesystem features. The Knoppix OS created an ext2 with all the latest features but the old RedHat 5.2 predated many of these features so it didn't know how to handle them.

I rebuilt the ext2 FS with options to turn off just about every feature.
mkfs -t ext2 -O none -r0 -O ^dir_index /dev/sdxx

Then I was able to rsync and complete all steps ending with a bootable RedHat 5.2 guest VM

Thanks for the great article.

Ubuntu 8 2.6.24 kernel I had

Pkilpo's picture

Ubuntu 8 2.6.24 kernel

I had to use "mount --bind /dev /mnt/sda1"
then chroot /mnt/sda1
then mount /proc /proc -t proc
and used mkinitramfs -o /boot/initrd.img

Filesystem has unsupported features

Len Kranendonk's picture

After migrating an old RedHat 9 server to VMWare using this howto, I got an error saying fsck.ext3: Filesystem has unsupported features.

I've posted a description on how to solve this issue on my blog: fsck: Filesystem has unsupported features.

I hope this might be helpful for people encountering the same issue.

nodev

Johannes Wolter's picture

I ran into problems because Knoppix mounted my root partition with the nodev option, which made the grub-install and mkinitrd calls fail, because the devices were not accessible. I had to change the mount-options with mount -oremount,dev,setuid /dev/sda1.
Furthermore I had a problem because /var is on a different partition (/dev/sda3)in my setup. I had to unmount this partition before chroot-ing and had to (re-)mount in the chroot-environment.

grub install issue

gc's picture

I'm having some issue with grub-install, mainly the fact that the system I'm virtualizing has a raid device for the hard drive. I've tried the grub install using some soft links to map the device names and the setup (hd0) command appears to complete but the virt-manager states the disk is not bootable

any suggestions

I had similar issues because

spingary's picture

I had similar issues because my set up has a /boot partition:

/dev/sda1 -> /boot
/dev/sda2 -> /
/dev/sda3 -> (swap)

so chroot'ing into my root, /dev/sda2, won't exactly work right since /boot is not there. Here's what I did to get it to work, using manual grub instead of grub-install:

Before you chroot into anything (while in Knoppix shell):

$ grub
grub> root (hd0,0)
grub> setup (hd0)
grub> exit

The 1st line points root to my first partition (/dev/sda1)
2nd line sets it up. What this does is set up grub on the MBR of the virtual disk.

Then, to rebuild the initrd correctly, I had to do this:

$ umount /mnt/sda1

$ mount -o remount,dev /mnt/sda2
(assuming sda2 was already mounted)
This mount command helps eliminate /dev/null problems when using the mkinitrd command later.

$ mount /dev/sda1 /mnt/sda2/boot
(This puts the boot partition into my sda2 root mount)

Now we can chroot:
$ chroot /dev/sda2
$ ls /boot
(should see the /boot partition stuff)

I had to add these to /etc/modprobe.conf in my install (I am using VMWare server 2.0.0 RC1):

alias scsi_hostadapter mptbase
alias scsi_hostadapter1 mptscsih

Now I rebuilt initrd:

Now we do the equivalent of this:

# mv /boot/initrd-2.4.21-32.0.1.ELsmp.img /boot/initrd-2.4.21-32.0.1.ELsmp.img.bak
# mkinitrd -v /boot/initrd-2.4.21-32-0.1.ELsmp.img 2.4.21-32-0.1.ELsmp
(Notice that the orig article was missing the .img in the second line)
(Also use the -v flag to see any useful errors)

I had one last problem - when I rebooted it complained the my e2fsck was too old and keep dumping me into repair mode. My install is a real old FC2, and when I used Knoppix to create the ext3 file systems, there are "new features" that the older e2fsck didn't support. So I followed these instructions and all is good:

http://www.mail-archive.com/trilug%40trilug.org/msg26479.html

Good luck everyone. Thank you for a great guide!

I had a problem...

Enrique Garcia's picture

Hi,

Before the second and final rsync, I wanted to test if the machines goes alive, but when I do the chroot /mnt/sda, and then
# grub-install /dev/sda
it gives:
/usr/share/grub/i386-redhat/stage1: Not found

And tried:
fdisk -l (without result)

It seems that I can't have access to /var /usr /home which are other partitions

What I tried next, was to cp the files in /mnt/sda3 (/usr), to /mnt/sda1 (/) just the files in the corresponding /usr/share/grub/i386-redhat/

And the output of that command was:
/sbin/grub-install: line 501: uniq: command not found
/dev/sda: Not found or not a block device.

any advice will be greatly appreciated.

Thanks in advance.

Even though

Enrique Garcia's picture

Even though the missing -H option, this is a great HOWTO, include I will be using this to migrate from one server to other and another, obviously I had never figure it out by myself... Thank you very much, And if possible, I would like to have your agreement to translate it, (and include fdisk steps) to Spanish.

Great!!!

rsync and hardlinks

Thomas Mueller's picture

the rsync options "-avx --numeric-ids --progress" IMHO doesn't include "-H" (hardlinks). So if a systems uses many hardlinks there will be many duplicated files.

Webinar
One Click, Universal Protection: Implementing Centralized Security Policies on Linux Systems

As Linux continues to play an ever increasing role in corporate data centers and institutions, ensuring the integrity and protection of these systems must be a priority. With 60% of the world's websites and an increasing share of organization's mission-critical workloads running on Linux, failing to stop malware and other advanced threats on Linux can increasingly impact an organization's reputation and bottom line.

Learn More

Sponsored by Bit9

Webinar
Linux Backup and Recovery Webinar

Most companies incorporate backup procedures for critical data, which can be restored quickly if a loss occurs. However, fewer companies are prepared for catastrophic system failures, in which they lose all data, the entire operating system, applications, settings, patches and more, reducing their system(s) to “bare metal.” After all, before data can be restored to a system, there must be a system to restore it to.

In this one hour webinar, learn how to enhance your existing backup strategies for better disaster recovery preparedness using Storix System Backup Administrator (SBAdmin), a highly flexible bare-metal recovery solution for UNIX and Linux systems.

Learn More

Sponsored by Storix