Do Linux Benchmarks Have A Leg To Stand On? - InformationWeek

InformationWeek is part of the Informa Tech Division of Informa PLC

This site is operated by a business or businesses owned by Informa PLC and all copyright resides with them.Informa PLC's registered office is 5 Howick Place, London SW1P 1WG. Registered in England and Wales. Number 8860726.

Government // Mobile & Wireless
03:42 AM

Do Linux Benchmarks Have A Leg To Stand On?

A recent set of Linux distro benchmarks appears to show some surprising results. Yet it also shows the dangers of relying too heavily upon such benchmarks to make real-world technology decisions.

A recent set of Linux distro benchmarks appears to show some surprising results. Yet it also shows the dangers of relying too heavily upon such benchmarks to make real-world technology decisions.Phoronix is one of the few Web sites with a heavy focus on Linux hardware testing. Its editors have also developed a comprehensive, and highly regarded, testing and benchmarking suite for Linux distros. As a result, Phoronix is a notable source for comparative data on Linux distro performance, involving both multiple distros and different releases of the same distro.

Recently, Phoronix offered up comparative benchmarks of four leading Linux distros: "With it being a while since we last compared many Linux distributions when it comes to their measurable desktop performance, we decided to run a new round of tests atop four of the most popular Linux distributions: OpenSuSE, Ubuntu, Fedora, and Mandriva. To see where these Linux distributions are at, we used their latest development releases and then performed all package updates as of 2009-07-15. Following that, we ran an arsenal of tests using the Phoronix Test Suite." Different distros typically run different versions of key software, and this case was typical. While three of the distros installed the EXT4 file system by default, for example, a fourth -- Mandriva -- uses EXT3. To some extent, these differences make exact comparisons very difficult, although benchmarking can still provide useful information about general performance trends.

Or can they? Perhaps that's a debatable point.

On one hand, the Phoronix benchmarks indicate the performance benefits of the new EXT4 file system. The Dbench test, which simulates filesystem load, clearly showed Mandriva lagging behind the other distros. Even though Dbench, like most such tests, does a poor job of simulating real-world server loads, such results are still noteworthy when one takes them in context.

On the other hand, a few of the Phoronix Test Suite results showed problems with the Fedora development release. The Apache benchmark, for example, made Fedora look downright anemic compared to its competitors.

The Phoronix author took a tentative stab at explaining these results: He speculated that it could be due to "SELinux on Fedora, debugging options, or a combination of other factors."

In fact, it turns out that Rawhide, the Fedora development code base, is compiled by default with its debugging options enabled. These options impose a significant performance handicap, since they require the operating system to perform a variety of running, often very resource-intensive, checks.

In defense of Phoronix, the benchmarks were apparently run with each distro's default, "out of the box" installation options. Yet such options in a development release are predictably very different than they would be in a production release, as Fedora's default use of a debugging mode indicates.

While this particular test appears to be seriously flawed, other Phoronix benchmarks are far more useful, in my opinion. Always evaluate such results, however, with a grain of salt: No matter how carefully it is done, benchmarking remains as much an art form as it is a science.

We welcome your comments on this topic on our social media channels, or [contact us directly] with questions about the site.
Comment  | 
Print  | 
More Insights
10 Ways to Transition Traditional IT Talent to Cloud Talent
Lisa Morgan, Freelance Writer,  11/23/2020
Top 10 Data and Analytics Trends for 2021
Jessica Davis, Senior Editor, Enterprise Apps,  11/13/2020
Can Low Code Measure Up to Tomorrow's Programming Demands?
Joao-Pierre S. Ruth, Senior Writer,  11/16/2020
White Papers
Register for InformationWeek Newsletters
Current Issue
Why Chatbots Are So Popular Right Now
In this IT Trend Report, you will learn more about why chatbots are gaining traction within businesses, particularly while a pandemic is impacting the world.
Flash Poll