Virtualization has long been the realm of proprietary software, starting with IBM mainframes and more recently VMware becoming a $1.3 billion-a-year powerhouse around x86 computers. In the coming year, look for open source code to play a larger role in virtualization.

This market shows how open source code can become a central part of proprietary vendors' strategies.

At a time when VMware had the x86 virtualization market to itself, Xen became an open source alternative with the backing of vendors from IBM to Intel to Microsoft. Xen is distinguished by the A-list vendors that have taken the core hypervisor and built their own product sets around it, including Oracle VM, Sun's xVM line, and Virtual Iron. Amazon.com runs its Elastic Compute Cloud on a tweaked version of Xen, and Red Hat and Novell have added Xen to enterprise Linux, operating outside the kernel.

But now that the Xen project's parent company, XenSource, was acquired by Citrix Systems for $500 million, many open source developers are shifting their interest to a lesser-known alternative: Kernel Virtual Machine, or KVM, the x86 virtualization engine embedded in the Linux kernel since February 2007. Among developers, KVM will benefit from being seen as less proprietary than Xen, and also from its position in the heart of the operating system, making use of the kernel scheduler, memory manager, and other functions.

Linux kernel developer Anthony Liguori makes the case in his Tales Of A Code Monkey blog that, with KVM in the kernel, Linux can host virtualization as an efficient, built-in feature. Red Hat must see that potential as well, having bought Qumranet, the company behind KVM, for $107 million. It plans to add KVM to Version 6 of Red Hat Enterprise Linux, which doesn't yet have a release date.

Adding KVM to Red Hat Enterprise Linux "will reach new customers who might not otherwise have considered Red Hat as their virtualization vendor," says Michael Ferris, Red Hat's director of product strategy. Companies considering desktop virtualization today probably are thinking of virtual machines running Windows. With KVM, they could consider desktops under Linux VMs and still access Active Directory and other key Windows apps, he says.

Xen, meanwhile, will likely continue to build momentum, thanks to its powerful vendor backers. Neither Xen nor KVM has dented sales of VMware, and VMWare executives are almost certainly more worried about Microsoft's competing virtualization products than those from open source. Still, this market shows how open source code can change the dynamics of a fast-growth market, and keep the market leaders from getting too comfortable.

Photo illustrations by Sek Leung