Greg Scherer, CTO of the I/O device maker Neterion, explains a weakness buried in virtualization's hypervisor. When it comes to virtual machine I/O, the hypervisor has to deal with it through a software switch, and lots of I/O means frequent interruption of the hypervisor's main job, processing guest application needs.If network traffic coming into a virtual machine, the 10 Gb Ethernet adapter on a server has to "tap the hypervisor on the shoulder and say, you need to process this," explains Scherer.

The Hypervisor set asides what it's doing, which means flushing out the deeply pipelined data and instructions that the application was working with and loading the Ethernet packets, reading their headers and moving them via the hypervisor's software switch--in VMware's ESX Server's case, the vSwitch--to get them started toward their destination.

On a CPU, this is known as context switching and it imposes one of the chip's severest latencies. The logic unit is dropping the instructions it's processing and taking up the networking task. And if it's expensive time-wise in hardware, it's equally expensive in the hypervisor. A 10 Gigabit Ethernet adapter on modern servers can quickly move one million 1,500-byte Ethernet packets. If the hypervisor gets "one million interruptions per second, you won't get very much done," says Scherer.

As he explained this, a little light will go off in the head of anyone who's sat through Cisco's recent Unified Computing System briefings.

I was down at VMware's vSphere 4 launch April 21 in Palo Alto and listened intently to John Chambers, Cisco CEO, as he joined in praising vSphere. Why not? Cisco's Unified Computing System works with VMware's ESX hypervisor to off-load virtual machine traffic from the vSwitch. Cisco can handle the traffic, whether it's headed for Infiniband storage or a high speed, Ethernet network, in a 10 Gb Ethernet fabric switch outside the blade server.

That makes a Cisco blade theoretically able host many virtual machines without getting bogged down in virtual machine network packet processing.

There are other ways to accomplish the same goal. Scherer's Neterion has its own converged network adapters that can do the vSwitch type processing. The venerable I/O firm, Emulex, is busy in this field, as is its 1994 spin-off, QLogic. A startup, called Xsigo, has built hardware that captures the converged network traffic coming from a virtual machine and processes it in a high powered box sitting next to the virtualized server.

The ability to offload from the hypervisor the virtual machine's network traffic is going to boost the number of virtual machines that a standard Intel or AMD server can host.

While most IT organizations are happy to get a 10 to 1 ratio of virtual machines to host, InformationWeek will next feature how Accenture's outsourcing unit has been able to run 30 virtual machines per host. And it's about to move from 30 to 60 per physical server and I doubt it's found its upper limit yet.

When it comes to tapping the power of virtualization, most IT organizations are still scratching the surface, with many gains still to come.