Contrast the years of jostling around the scheduler with the experience of Avi Kivity, an Israeli developer who submitted a large, 12,000-line batch of code called the KVM virtualization engine. It helps to be known to kernel developers and maintainers when submitting a patch, but "KVM came out of the blue," says Morton. "I had never heard of him or his company [Qumranet] before."
Kivity describes himself as a "longtime lurker" on the Linux kernel mailing list, reading it avidly and noting its expert personalities and debates without submitting much code himself. He designed KVM to what he considered kernel standards, kept the kernel's file system expert abreast of progress on the code, and responded immediately to questions and comments from kernel maintainers. KVM addressed an important need in Linux given the rush of interest in virtualization, giving the kernel its first features to exploit the latest virtualization hooks in the Intel and AMD chips. It also artfully made use of the kernel's scheduler and memory manager and affected little else in the operating system. The result was that KVM sailed into the kernel in less than three months after its submission last fall.
Adding code from a little-known author and a fledgling company was a risk, Morton says, since both could fade away, leaving no one with expertise in the code. But given the code's standalone approach, developers could just as simply remove it if it withered.
Even when code like KVM zips into the kernel, there can be a lag of a year or two before it's picked up by one of the top two enterprise distributions, Red Hat Enterprise Linux and SUSE Linux Enterprise Server. ("Community releases" such as Red Hat Fedora and Novell OpenSUSE are updated quickly.) That's to allow for extensive testing and support materials. Many businesses are happy to have that stability and hold off on having the latest and greatest kernel.
And yet Linux races ahead, with developers pouring new features into the kernel in the name of fame, curiosity, and sometimes salary. Over a recent 28-month period in which 11 new kernels appeared, the number of identifiable individual contributors went from 479 to 838. And for every person who gets his or her name on a block of code, there are probably three or four people who helped that person, goes the common estimate. That means 3,000 or so people are involved in each iteration of the kernel.
It's that volunteer community that the Linux movement's still counting on, even as the kernel maintainers, the skilled developers who lead Linux subsystems, are paid by companies such as Google, Hewlett-Packard, IBM, Novell, and Red Hat. That community is why Morton says there's not a "direct trade-off" between speed of development and reliability, since getting features out sooner gets them hammered on long before they'd show up in a business.
Still, there's a drawback, compared with commercial code. "I don't want to call it unpredictability, but you can't guarantee a delivery date," Intel's Hohndel says. "Linux code is delivered when it's ready."
In another two or three months, Torvalds will issue kernel 2.6.24, with a dozen or so new features produced and tested with the help of hundreds of developers who weren't involved in this month's release, with no way to know how much if any of it will eventually make it into business-tested versions. It's not really what anyone would call a product "road map." But so far, it hasn't steered businesses wrong.
Illustration by Dale Stephanos
Seven Areas Linux Could Get Better