look at the contents of the memory on the host server. The controller was meant to be restricted to a tiny portion of the memory concerned with its own operation, but an exploit could order it to report data being used by other virtual machines or even the host's hypervisor itself. The hypervisor's operations themselves were not breached, but visibility into them is a serious security flaw.
The bug only affected the 4.1 and later versions of Xen, which have been out since March 25, 2011, and would be in place in most public cloud services relying on Xen. Amazon Web Services uses its own Amazon Machine Images, but they are a close variant of Xen open source code and were affected as well.
In other words, "an 'evil' virtual machine could essentially read over the shoulder of other virtual machines running on the same server, bypassing security," wrote James Gallagher on Ars Technica.
Amazon's Barr said the bug had affected "less than 10% of its servers."
Rackspace's Rhodes said, "We don't want to advertise the vulnerability before it's fixed -- lest we, in effect, ring a dinner bell for the world's cyber criminals." Rackspace was so cautious it didn't mention that the problem was buried in the Xen hypervisor. Amazon didn't either in its first mention of the problem, but cited a Xen exposure in its second, as work got underway 7 p.m. Pacific time on Sept. 25.
Rhodes said: "That's the dilemma that we faced over the Xen bug. Such vulnerabilities are regularly found in software, whether proprietary or open source. The key, once a bug is identified, is to fix it swiftly and quietly. This particular vulnerability could have allowed bad actors who followed a certain series of memory commands to read snippets of data belonging to other customers, or to crash the host server."
SoftLayer said it took the time it needed to make sure the problem was fixed: "As soon as the risk was identified, our systems engineers and technology partners have been working nonstop to prepare the update," it said.
The Xen bug is both a good example of collective security and a warning of what can happen as IT shifts toward a greater reliance on cloud computing. The bug was discovered and interested parties notified before the full nature of the exploit was disclosed. Collective security action followed, apparently (at this early date) in time before any malicious code writers could act on the disclosure.
At the same time, the bug illustrates the cloud's dependence on one hypervisor or another and how a major hypervisor bug will affect more than one supplier. The growing, more uniform nature of x86 cloud environments represent a fatter target for highly skilled intruders to aim for, and a richer environment for manipulation if they succeed at getting inside.
Considering how prevalent third-party attacks are, we need to ask hard questions about how partners and suppliers are safeguarding systems and data. In the Partners' Role In Perimeter Security report, we'll discuss concrete strategies such as setting standards that third-party providers must meet to keep your business, conducting in-depth risk assessments -- and ensuring that your network has controls in place to protect data in case these defenses fail. (Free registration required.)