Linux Kernel Maintainers: Accountable To All, Beholden To None?

The Linux kernel is surrounded by hundreds of interested parties. How is it that none of them gains a commanding influence over the kernel's development priorities? HP, IBM, Oracle. Google, eBay and Intel each has a primary stake in Linux and employs kernel developers. Does this mean money talks when it comes to Linux? If not, why not?Linux is "part of HP's mainstream business compared to just a few years ago," acknowledges Randy Hergett, director of HP engineering for open source and Linux.

HP developers have been involved in the kernel development process and contributed expertise on scalability and performance through HP's Nashua, N.H., development facility for the 2.4 - 2.6 kernel releases. It's also contributed changes in file system audit, so events executed by the kernel can be recreated and tracked.

And what does HP get in return for this work?

Well, a large HP telecommunications customer ran into a problem with its Linux operations. HP came up with a patch that fixed the customer's problem, without waiting for a new release of the kernel.

"Our experts thought the patch would benefit the entire community," says Hergett, so in addition to fixing the customer's problem, it submitted the patch to its Linux distribution partner. That could be either Red Hat or Novell, Hergett won't say which. But the point is, the Linux distributor, was also the employer of the kernel maintainer responsible for the part of the kernel to which the patch applied. The patch went through the maintainer's review and was submitted to higher level review, probably Andrew Morton. It eventually was inspected by Linus Torvalds and merged into the kernel.

Does this mean HP got special consideration?

Without knowing all the details, it's not possible to say. But the way the kernel process normally works is that a patch has to be determined to be a benefit to many, not just one vendor's customer. A problem that gets fixed for telecommunications may represent, not special pleading, but an improvement to the overall operating system.

Does the kernel process work this way in every instance? Without knowing about every change, it's not possible for an outsider to say. But again, many eyes are watching what does and does not happen with the kernel; many developers are reading the kernel mailing list and there's a consistent atmosphere of challenging short-term fixes. If nothing else, one vendor would protest changes that tended to benefit only another vendor.

The stronger safeguard is that open source developers by nature are jealous of vendors exerting sway over their work. Many of them are employed by Google, eBay, IBM, HP, Red Hat, Novell or Intel. But they gain the kernel maintainer jobs because they were good developers in the first place. They have their own principles of what constitutes sound software design. They consider the overall design scheme of Linux paramount. One doesn't get to be a kernel maintainer by showing agreeableness in the face of pressure. On the contrary, consistent thinking under pressure is what's required.

IBM has made Linux central to its long term strategy by featuring it on all its servers. Even so, it's in trouble if an enthusiastic employee shows up in a mailing list discussion as announcing himself as from a really big Linux supporter.

"What matters is how good your code is. The code talks, not who you work for," says Dan Frye, VP of open source development at IBM.

Dirk Hohndel, chief Linux and open source technologist at Intel, says Intel is a big contributor of drivers and other code and a key tester of Linux kernels. It's important that changes made don't break the kernel's compatibility with the latest Intel chipset. However, when it comes to getting Intel contributions accepted, "it's as hard for us as for anyone else. No one gets special treatment," Hohndel says. "The fact the code is open allows a thorough review. The whole development process is discussed in a very open fashion." If Linux were just the window dressing of another vendor consortium, one vendor might instruct a kernel maintainer to do something for the benefit of another vendor in a horse trading arrangement. But kernel maintainers would resent such instruction and are free to move to a new host, if they choose. Jeremy Allison, a prestige employee at Novell as lead developer of Samba, left that company after the Novell/Microsoft agreement to go to work at Google, where the atmosphere was more to his liking. Kernel maintainers are free to do the same.

Few, if any, of them are subject to instructions from their employers. Their relationship is more like the artist/patron relationship of the Renaissance than a master/slave relationship.

When it comes to Intel's interest in the kernel development, Hohndel says he and many other observers have come to realize that "my purpose is better served by having the best possible person as a subsystem maintainer, than keeping an Intel person in the role." With strong, independent maintainers, Linux keeps its design principles intact and keeps forwarding fresh code that fits into the kernel to Andrew Morton and Linus Torvalds. When the process is working, the kernel advances rapidly. And that appears to be the case now. IBM's Frye concedes that "the majority of the people who work on the kernel are paid to do so" by companies like IBM. But IBM can't hire a developer and lobby for his appointment as a maintainer of a particular Linux subsystem. "The people who write the best code rise to the top. It is, roughly speaking, a meritocracy," he says. It's not a pure meritocracy. The kernel process has a few characteristics of the old "who knows who, who's trusted by who" of the proprietary world that came before it. If you have trampled on the sensibilities of a key kernel expert, it may not matter that your latest lines of code are your best.

But, says HP's Hergett, "I think we're getting the best of both worlds."