With each release, the Linux kernel is resolving more bugs, thanks to a new staging version of the operating system that identifies problems before the kernel's next proposed release reaches its final review by Linus Torvalds.

The Linux-next code "tree," or reassembly of old code with new contributions, is built five times a week by kernel developer Stephen Rothwell, an IBM employee, and that was one of the topics discussed at the Linux Foundation's Collaboration Summit on Wednesday in San Francisco. The Linux-next tree was launched a year ago after the Linux Collaboration Summit in Austin, Texas. The tenor of the discussion at Wednesday's event made it clear that the kernel development process has adopted it as a long-term change.

With each release of the kernel, dozens of new drivers, functions, bug fixes, and other changes appear in the kernel from a network of about two dozen subsystem maintainers. These developers in turn are processing the work of several hundred other contributors and developers. The influx of code occurs at such a rate -- more than 2,000 lines per day -- that it's difficult to predict when a change to one part of the kernel will conflict with either existing operations or a new change added elsewhere.

It used to be the task of Linus Torvalds, founder and lead developer of the Linux project, to referee the changes, perform integrations, or reject a code submission as too disruptive to the kernel. In recent years, much of that work devolved to Andrew Morton, the "colonel of the kernel," who checked code and performed many integrations himself.

Wednesday Morton said the creation of Rothwell's staging code tree, Linux-next, allowed many changes to be meshed together early and submitted for a compile, where problems often first appear. Code that "breaks the build" in the process of a fresh compile is then identified and its submitters are notified to do more work on it.

"The testing done in the Linux-next tree represents 85% to 90% of the work I used to do," Morton said as a member of panel, "The Linux Kernel, What's Next," Wednesday morning.

"Linux-next is finding a lot of problems that would be found in Linus' tree [in final review before release]. When something goes bad at that stage, quite often it was never tried in Linux-next," Morton added.

Kernel subsystem maintainers, who hold their own powerful positions of building a code tree for specific sections of the kernel, such as James Bottomley's SCSI subsystem area, didn't all warm to the idea of an additional step in their workload. When Linux-next was still being fledged, the following exchange took place between Rothwell and Russell King, maintainer of the kernel's port to the Arm chip architecture. Arm is an important platform for Linux because it's used in many embedded systems, including digital cameras and other portable devices. On April 23, 2008, Rothwell pointed out that the Linux-next build had been broken by a previously untested Arm addition. "This is why I would have liked you to participate in the next Linux-next tree," Rothwell told him in a message.

King answered, "Linux-next will not give me anything" that Morton's efforts didn't already provide and he preferred the established procedure, he wrote back. "Linux-next's value is very small for me. Sorry, but true."

Then Morton weighed in: "Putting Arm into Linux-next means that Stephen handles the merges rather than having me do it. Which helps me." Morton added that Rothwell's builds tested all contributions to the kernel, rather than just the ones Morton was working on, giving Arm "a lot more cross-compilation testing. ... Which helps you."

Ted Ts'o, CTO of the Linux Foundation and kernel file subsystem maintainer, said the update to Linux's most popular file system, EXT, "has been pronounced stable." EXT4 was added to the kernel several release ago with the experimental label on its module. That label was removed in the 2.6.29 release on March 24. He said he has been using EXT4 on his own laptop since last July without mishap.

The Collaboration Summit is in its third year and is sponsored by the Linux Foundation, a consortium of vendors who use Linux in their product lines and pay the salary of Torvalds.

Oracle chief architect Edward Screven said Oracle would like to see another file system added to the Linux kernel, the btrfs, which he pronounced "butter... eff-ess." He said btrfs has enhancements that aid file repair and reliability. He said Oracle developers have looked at Sun Microsystems' ZFS, an advanced file system capable of handling a very large number of data, and concluded, "We like btrfs better than ZFS."

ZFS is part of Sun's OpenSolaris project and is now open source code but isn't a candidate to be added to Linux because of license incompatibility, said Ts'o.

Corruption of data "can occur at any step" in the path from when it's created to when it's used in a transaction and stored in a hard disk file. "We need a more enhanced file system for end to end data integrity," he said. Btrfs is not yet a candidate to be merged into the kernel. But Screven said Oracle is a strong contributor of code and bug fixes to Linux kernels and will lobby on its behalf.

Ts'o said the use of another specialized file system already in the kernel, the Reiser system, was likely to drop off among because its inventor and chief maintainer, Hans Reiser, has been convicted of and imprisoned for the murder of his wife.

InformationWeek Analytics has published an independent analysis of the challenges around virtualization management. Download the report here (registration required).