Oracle's Linux: Unbreakable? Or Just A Necessary Adjustment?

As I talked to Wim Coekaerts, VP of Linux engineering at Oracle, about "Unbreakable Linux," a gap emerged between what he was saying and what Red Hat's product management director, Joel Berman, was saying. Riders of Linux's impressive upsurge are advised to "mind the gap" and try not to fall into it.Oracle produces Unbreakable Linux, but it's not in the business of competing with Red Hat as a Linux distributor, says Coekaerts, a kernel developer himself.

As I asked Coekaerts and other kernel developers about the kernel development process, the subject of so-called Unbreakable Linux inevitably came up. Oracle's year-old Unbreakable Linux is more about enterprise Linux technical support and keeping Oracle customers up and running than a separate distribution, Coekaerts said in an interview.

In fact, Coekaerts has to say this often because Oracle is widely viewed as an opportunistic supporter of Linux, taking Red Hat's product, stripping out its trademarks, and offering it as its own. Coekaerts says what's more important is that Oracle is a contributor to Linux. It contributed the cluster file system and hasn't really generated a competing distribution.

Yet, in some cases, there is an Oracle distribution. Most customers Coekaerts deals with get their Linux from Red Hat and then ask for Oracle's technical support in connection with the Oracle database. But Oracle has been asked often enough to supply Linux with its applications or database that it makes available a version of Red Hat Enterprise Linux, with the Red Hat logos and labels stripped out. Oracle's version of Linux has a "cute" penguin inserted and is optimized to work with Oracle database applications. It may also have a few Oracle-added "bug fixes," Coekaerts says.

The bug fixes, however, lead to confusion about Coekaert's relatively simple formulation of Oracle enterprise support, not an Oracle fork. And that confusion stems from Oracle CEO Larry Ellison's attention-getting way of introducing Unbreakable Linux at the October 2006 Oracle OpenWorld.

When enterprise customers call with a problem, Oracle's technical support finds the problem and supplies a fix. If it's a change in the Linux kernel, the customer would normally have to wait for the fix to be submitted to kernel maintainers for review, get merged into the kernel, and then get included in an updated version of an enterprise edition from Red Hat or Novell. Such a process can take up to two years, observers inside and outside the kernel process say.

The pace of bug fixes "is the most serious problem facing the Linux community today," Ellison explained during an Oracle OpenWorld keynote a year ago.

When Oracle's Linux technical support team has a fix, it gives that fix to the customer without waiting for Red Hat's uptake or the kernel process itself, Ellison said.

Red Hat's Berman argues that when it comes to the size of the problem, Oracle makes too much of too little.

When Red Hat learns of bugs, it retrofits the fixes into its current and older versions of Red Hat Enterprise Linux. That's one of Red Hat's main engineering investments in Linux, Berman said in an interview.

Coekaerts responds, "There are disagreements on what is considered critical by the distribution vendors and us or our customers."

Berman acknowledges that several judgment calls are involved. Some bugs affect only a few enterprise customers. They may apply to an old RHEL version. "Three or four times a year" a proposed fix may not be deemed important enough to undergo this retrofit, he says.

But Coekaerts told InformationWeek: "Oracle customers encounter this problem more than three or four times a year. I cannot give a number, it tends to vary. But it does happen rather frequently."

Berman counters that when Oracle changes Red Hat's tested code with its own bug fixes, it breaks the certification that Red Hat offers on its distribution, so it's no longer guaranteed to work with other software. "Oracle claims they will patch things for a customer. That's a fork," he says.

What Red Hat calls a fork is what Oracle calls a "one-off fix to customers at the time of the problem. … If the customer runs version 5 but Red Hat is at version 8, and the customer runs into a bug, does he want to go into [the next release with a fix] version 9? Likely not. He wants to minimize the amount of change. Oracle will fix the customer's problem in version 5…" Coekaerts says.

I think it's fair to characterize what Oracle does as technical support, not a fork. There's no attempt to sustain the aberration through a succession of Linux kernels offered to the general public as an alternative to the mainstream kernel.

But the Oracle/Red Hat debate defines a gray area in a fast-moving kernel development process. Bugs that affect many users get addressed through the kernel process or the Red Hat and Novell retrofits. That still may not always cover a problem for an individual user or a set of users sitting on a particular piece of aging hardware or caught in a specific hardware/software configuration.

If Oracle fixes some of these problems, I say more power to it.

But if they are problems that are isolated in nature or limited in scope, as I suspect they are, that makes them something less than Ellison's "most serious problem facing the Linux community today." Ellison needed air cover to take Red Hat's product and do what he wanted with it. In the long run, he's probably increasing the use of Linux in the enterprise and keeping Red Hat on its toes as a support organization. That's less benefit than claimed, but still something.

Linux users should stay aware of how Linux bug fixes occur and when they are likely to be done or not done -- and try not to fall into the gap.