Introduction to the Xen Virtual Machine

Everyone's talking about Xen, but the code is complex. Here's a starting point.

This article is intended mainly for developers who are new to Xen and who want to know more about it. The first two sections, however, are general and do not deal with code.

The Xen VMM (virtual machine monitor) is an open-source project that is being developed in the computer laboratory of the University of Cambridge, UK. It enables us to create many virtual machines, each of which runs an instance of an operating system.

These guest operating systems can be a patched Linux kernel, version 2.4 or 2.6, or a patched NetBSD/FreeBSD kernel. User applications can run on guest OSes as they are, without any change in code. Sun also is working on a Solaris-on-Xen port.

I have been following the Xen project closely for more than a year. My interest in Xen began after I read about it in the OLS (Ottawa Linux Symposium) 2004 proceedings. It increased after hearing an interesting lecture on the subject at a local UNIX group meeting.

Full virtualization has been done with some hardware emulators; one of the popular open-source projects is the Bochs IA-32 Emulator. Another known project is qemu. The disadvantage of hardware emulators is their performance.

The idea behind the Xen Project (para-virtualization) is not new. The performance metrics and the high efficiency it achieves, however, can be seen as a breakthrough. The overhead of running Xen is very small indeed, about 3%.

As was said in the beginning, currently Xen patches the kernel. But, future processors will support virtualization so that the kernel can run on it unpatched. For example, both Intel VT and AMD Pacifica processors will include such support.

In August 2005, XenSource, a commercial company that develops virtualization solutions based on Xen, announced in Intel Developer Forum (IDF) that it has used Intel VT-Enabled Platforms with Xen to virtualize both Linux and Microsoft Windows XP SP2.

Xen with Intel VT or Xen with AMD Pacifica would be competitive with if not superior to other virtualization methods, as well as to native operation.

In the same arena, VMware is a commercial company that develops the ESX server, a virtualization solution not based on Xen. VMware announced in early August 2005 that it will be providing its partners with access to VMware ESX Server source code and interfaces under a new program called VMware Community Source.

A clear advantage of VMware is that it does not require a patch on the guest OS. The VMware solution also enables the guest OS to be Windows. VMware solution is probably slower than Xen, though, because it uses shadow page tables whereas Xen uses both direct and shadow page tables.

Xen already is bundled in some distributions, including Fedora Core 4, Debian and SuSE Professional 9.3, and it will be included in RHEL5. The Fedora Project has RPMs for installing Xen, and other Linux distros have prepared installation packages for Xen as well.

In addition, there is a port of Xen to IA-64. Plus, an interesting Master's Thesis already has been written on the topic, "HPC Virtualization with Xen on Itanium".

Support for other processors is in progress. The Xen team is working on an x86_64 port, while IBM is working on Power5 support.

The Xen Web site has some versions available for download, both the 2.0.* version and the xen-unstable version, also termed xen-3.0-devel. You also can use the Mercurial source code management system to download the latest version.

I installed the xen-3.0-devel, because at the time, the 2.0.* version did not have the AGP support I had needed. This may have changed since my installation. I found the installation process to be quite simple. You should run make world and make install, update the bootloader conf file and that's it--you're ready to boot into Xen. You should follow the instructions in the user manual for best results.

The Return of the Ring

The protection model of the Intel x386 CPU is built from four rings: ring 0 is for the OS and ring 3 is for user applications. Rings 1 and 2 are not used except in rare cases, such as OS/2; see the IA-32 Intel Architecture Software Developer's Manual, Volume 1: Basic Architecture, section 4.5 (privilege levels).

In Xen, a "hypervisor" runs in ring 0, while guest OSes run in ring 1 and applications run in ring 3. The x64/64 is a little different in this respect: both guest kernel and applications run in ring 3 (see Xen 3.0 and the Art of Virtualization, section 4.1 in OLS 2005 proceedings).

Xen itself is called a hypervisor because it operates at a higher privilege level than the supervisor code of the guest operating systems that it hosts.

At boot time, Xen is loaded into memory in ring 0. It starts a patched kernel in ring 1; this is called domain 0. From this domain you can create other domains, destroy them, perform migrations of domains, set parameters to a domain and more. The domains you create also run their kernels in ring 1. User applications run in ring 3. See Figure 1, illustrating the x86 protection rings in Xen.

Figure 1

Currently, domain 0 can be a patched 2.4 or 2.6 Linux kernel. According to the Xen developer mailing list, however, it seems that in the future, domain 0 will support only a 2.6 kernel patch. Much of the work of building domain0 is done in construct_dom0() method, in xen/arch/x86/domain_build.c.

The physical device drivers run only in the privileged domain, domain 0. Xen relies on Linux or another patched OS kernel to provide virtually all of its device support. The advantage of this is it liberates the Xen development team from having to write its own device drivers.

Using Xen on a processor that has a tagged TLB improves performance. A tagged TLB enables attaching address space identifier (ASID) to the TLB entries. With this feature, there is no need to flush the TLB when the processor switches between the hypervisor and the guest OSes, and this reduces the cost of memory operations.

Some manufacturers offer this tagged TLB feature. For example, a document titled "AMD64 Virtualization Codenamed 'Pacifica' Technology Secure Virtual Machine Architecture Reference Manual" was published in May 2005. According to it, this architecture uses a tagged TLB.

Next up is an overview of the Xend and XCS layers. These layers are the management layers that enable users to manage and control both the domains and Xen. Following it is a discussion of the communication mechanism between domains and of virtual devices. The Xen Project source code is quite complex, and I hope this may be a starting point for delving into it.

______________________

Comments

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

Hele me please

Anonymous's picture

This is my course work in School.
1. Example applied of Xen & VM (Virtual Machine)
2. Description of relation between VM and OS
3. Example applied of VM now
Hele me please.

Some performace test result

ericzqma's picture

I installed Xen on two of our server. The kernel and xen are the modern versions (The platform information can be found here ). I have done some test on this platform, the result is that the CPU performance is nearly 100% while the memory performance is only 90% compared to the physical machine. Details can also be found here . I am satisfied by the cpu performance. Maybe the 10% memory overhead is a bit large. I am wondering whether there are some mistake in my configuration or how to improve it. -- Eric

Good

Laks's picture

Good article - shown few concepts behind Xen and it's useful for beginners.

is it possible to do the memory copy operation between two VM's?

Anonymous's picture

is it possible to do the memory copy operation between two guest VM's directly through through XEN without involving dom 0?

Inter-Domain Comms

Mr. B's picture

Hello,

It is a little confusing on how you have described the interaction between XenStore and a domain. How exactly does a Domain interact with Xenstore i.e. TCP ports, sockets, etc...? Since XenStore resides in ring 3 how does it access the hypervisor itself? Thanks.

Mr. B

xen against qemu/bochs

Anonymous's picture

With Xen on x86(_32) running guest OS kernel in ring 1 and guest OS applications in ring 3 carefuly exploited guest OS is wide open door to hijack host OS root applications in ring 3 and this way compromise host OS.

That's something I guess can't happen with qemu/bochs etc. Other words: you trade that for speed.

And at first guess the enhanced CPU architecture will have just tags at descriptors and more complicated descriptor access rules to enable more page tables separated and being loaded simultaneously switched/selected on demand and privileges. But then how can it provide applications/OSes existing in different page tables with similar amount of cpu time to run? Maybe someone can summarize the tech a bit and publish it?

carefuly exploited guest OS

Anonymous's picture

Xen does validation of memory accesses

does xen overhead include OS overhead?

undefined's picture

one clarification that i need is if the 3% overhead of xen includes the overhead of running multiple identical guest kernels. yes, xen adds 3% overhead, but is there also some duplication when running 3 linux kernels, whether in memory or in processing?

i recently investigated viritualization for the purposes of consolidating, yet keeping partitioned, a linux server & desktop. as there is very little difference between my current linux server & desktop kernels, i would prefer not to duplicate the linux kernel, but merely have different userlands. i am currently testing linux vserver as it allows me to run a single linux kernel, but maintain multiple userland "instances", each "instance" with its own ip address and other features.

granted vserver, chroot, etc does not help when a user wants to run different operating systems (linux & windows), and if full separation between userland images, even down to the kernel level (kernel-level exploits, user-visible features like nfsd, etc), is desired, then xen is the proper tool for that job. heck, give the xen livecd a test drive and marvel at xen's accomplishment.

just wanted to share my holiday weekend's research to help save someone else some time.

we tested it recently. yup,

Anonymous's picture

we tested it recently. yup, it involves 3% overhead on simple operations, but overhead is more than 20-30% on disk I/O, network etc.
And sure, memory pressure/requirements you mentioned are rather big.

I would recommend you to take a look at OpenVZ project as well. It is more mature, than vserver. We successfully run 30-50 VPSs on 1GB of RAM with it.

disk/network io

Anonymous's picture

Why not use separate drives for each server slice instead of a file system on a file? Perhaps separate network cards also?

This might mitigate the slow down but perhaps satiating the buses.

Anyone doing that?

Cheers,
-b

disk i/o

Luke Crawford's picture

things run much faster if you give each domU it's own partition. LVM helps a lot here, both to run many small domains on one disk, and to keep track of who owns what partition.

anyone care to write proof of concept exploit?

Anonymous's picture

> (kernel-level exploits,

I guess this may be still issue with xen compared to qemu/bochs. It's not that straight forward but have a look at the access privileges model behind ring. Once you gain ring 1 privileges then the userland of host OS is toast.

windows applications

Anonymous's picture

I am curios, after the VT and Pacifica gets in and you can then run windows on xen directly, could you run games, graphics, etc...
I guess it depends what kind of drivers xen would provide or allow access too. Anyone?
I.e. work on linux and windows in tandem.
For example, applications that can't or have not yet been ported to linux will work on windows (such as games, proprietery...) and the rest would be linux.

Sadly the SMP support is rath

Anonymous's picture

Sadly the SMP support is rather unstable (and therefore currently only in xen unstable. :-) ).

VMware Community Source is nonsense

Anonymous's picture

VMware's "Community Source" program is exactly like open source, only they don't share their software with anyone except their corporate partners, and don't share the contributed code.

Agreed: VMware Community Source is a load...

Anonymous's picture

I've been reading VMware press releases for the last few weeks with zero substance except how they were going to "open" something up. I went to Intel's Developer Forum and spoke with numerous developers from IBM, HP and Intel at and asked them straight up what the deal was. I asked, "Where is the "open code"? They all kind of (quietly) said the same thing. VMware is getting freaked out by Xen and wanted some press. In reality, they may document a few more APIs, but this is just a load...

Author Response

Rami's picture

Hello,

I had written in this article about the advantages and disadvantages of the Xen and VMWare virtualization solutions.
One of the Xen advantages I pointed out was it was
free and open source project.

I felt it will be unfair not to mention that VMware
started that Community Source program in the beginning
of this August.

In the article I wrote aboout this Community Source program : "..it will be providing its partners with access to VMware ESX Server source code"; also VMWare news release (to which I gave a link) talks about giving source to ***partners***.

I think your comment should be read considering this and in this light.

Regards,
Rami Rosen

Where can i read your article?

Anonymous's picture

Hi,

I was wondering if there is a link or web page where I can read your article about the advantages and disadvantages of Xen and Vmware.

Where can I find it? is it online?

Thanks.

Gabriel

NetBSD doesn't need to get patched

hubertf's picture

NetBSD has native support for Xen for some time in the official releases now, and does NOT need to be patched. See www.NetBSD.org/Ports/xen for more information.

- Hubert

no POWER5 support

Hollis Blanchard's picture

I am one of the developers working on the PowerPC port of Xen, and we are supporting the PowerPC 970, not POWER5.

Xen in IBM

Anonymous's picture

Hello,

Please look here:

http://lwn.net/Articles/139964/

It says:
...
IBM is working on Power5 support...
...

Are you shure your team is the only one in IBM working on
Xen ?

Yes, I'm positive. The LWN pa

Hollis Blanchard's picture

Yes, I'm positive. The LWN page is also incorrect, though it cites its source so you can see where the information comes from.

Why bother with POWER5 support?

Anonymous's picture

Why would IBM waste resources on POWER5 support? They already have a rock-solid micropartitioning and virtualization environment on the POWER5 that supports Linux, and one that appears to provide even greater protection across partitions then xen does with domains. I'm running my own distribution on one as I write this, and I'm sold. I'd rather manage a SAN-backed POWER5 installation over a blade server any day.

I can see a big advantage for the PPC970, though, given that you can get JS20 blades for their blade center, and the HS20 already.

More on VM and Emulators

moma's picture
Webinar
One Click, Universal Protection: Implementing Centralized Security Policies on Linux Systems

As Linux continues to play an ever increasing role in corporate data centers and institutions, ensuring the integrity and protection of these systems must be a priority. With 60% of the world's websites and an increasing share of organization's mission-critical workloads running on Linux, failing to stop malware and other advanced threats on Linux can increasingly impact an organization's reputation and bottom line.

Learn More

Sponsored by Bit9

Webinar
Linux Backup and Recovery Webinar

Most companies incorporate backup procedures for critical data, which can be restored quickly if a loss occurs. However, fewer companies are prepared for catastrophic system failures, in which they lose all data, the entire operating system, applications, settings, patches and more, reducing their system(s) to “bare metal.” After all, before data can be restored to a system, there must be a system to restore it to.

In this one hour webinar, learn how to enhance your existing backup strategies for better disaster recovery preparedness using Storix System Backup Administrator (SBAdmin), a highly flexible bare-metal recovery solution for UNIX and Linux systems.

Learn More

Sponsored by Storix