Linux's Colonel Of The Kernel Andrew Morton: 'Fix More Bugs'

Andrew Morton, sometimes referred to as the colonel of the kernel, is Linus Torvalds' right hand man when it comes to getting out new kernel releases. Morton screens patches that are candidates for being merged into the kernel. He distributes them to kernel maintainers, watches discussions and feedback from key kernel developers and in general applies a layer of organization to a sometimes chaotic process. In this interview with InformationWeek editor at large, Charles Babcock, he talks about recent kernel development including an assessment of recent patches and tools.

INFORMATIONWEEK: Which is the goal these days in developing the kernel... rapid development of new features or minimizing possible bugs and defects?

Morton: We try to do both, of course. It's not a direct trade-off. If you apply patches faster, you can find bugs and fix things faster... There are many layers of back-up behind us. There are hundreds and thousands of testers who help us find new bugs. As the kernel goes into the leading-edge products, [Red Hat] Fedora and [Novell] OpenSuse, it goes out to many thousands of additional testers. It's a year old or more by the time it gets added to an enterprise version of Linux.

INFORMATIONWEEK: Do you emphasize one side or the other in the debate over new features?

Morton: I would like to see people spend more time fixing bugs and less time on new features. That's my personal opinion. People don't have much incentive to fix bugs on old hardware... [where few users are affected]. Natalie Protasevich is now administering the Bugzilla database and trying to keep it in shape and call attention to more bugs. Many bugs are reported by e-mail and corrected that are never entered into the database. The 1,400 [in the database] are a tiny subset of all those that get worked on. Many of those bugs only occur on a particular combination of hardware and can only be reproduced on that combination.

INFORMATIONWEEK: Is there a delay in getting bug fixes into the kernel of the enterprise editions, Red Hat Enterprise Edition and Suse Linux Enterprise Server, which have older kernels?

Morton: Those kernels are irrelevant to the kernel development process. That code is so old that it's up to the distributors to take care of it. It's entirely up to the distributors. They wouldn't have the problems they're having if they updated their kernels more frequently.

INFORMATIONWEEK: What's the prospect of the utrace diagnostic tool making it into the kernel?

Morton: It was in my kernel tree for a while but it fell out. They just need to do the work [to make it kernel ready].

INFORMATIONWEEK: Where does the application security patch AppArmor stand at this point?

Morton: There's a fundamental disagreement with the pathname basis for application security. There's a feeling that it's just not the way to go. But I can't help them. I am not a security developer. I don't know how to get that one unstuck.

INFORMATIONWEEK: When did you get started working on the kernel?

Morton: I started kernel development in 2000. I had been doing a lot of integration work and running the memory management tree. Then Linus tapped me on the shoulder and asked me to please do this [act as lead maintainer]. I guess I had demonstrated an ability to play well with others and maintain my sanity.

INFORMATIONWEEK: How much time do you spend on kernel development a week?

Morton: Oh, I spend about 50-60 hours a week. I spend a lot of time working in low power mode at the computer on e-mail, responding quickly to people so I'm not blocking their activity. I don't do that much coding. I spend a lot of time straightening out other people's work, but only once every two or three months will I code for a day on something that's really irritating me.

INFORMATIONWEEK: What was your impression of the KVM virtualization software when you first saw the patch?

Morton: KVM came out of the blue. I have never heard of Avi Kivity [its author] or his company [Qumranet]. The patch looked good. It fit into the conventions of the kernel and didn't affect anything else. It looked kernel-ready on day one.

INFORMATIONWEEK: What were the pros and cons of merging it quickly into the kernel?

Morton: It added functionality that we hadn't had before. It exploited the virtualization features that the hardware manufacturers were giving us. If there was any risk, it was the chance that Kivity or his company wouldn't be there in a year. But it was a minimal risk because KVM didn't disrupt anything else. It would have been easy enough to remove it, if we had to. It was a no-brainer, really... It was an exceptional patch in many ways.