In addition, Thompson admitted that the vulnerability counts lumped together the vulnerabilities found in Linux, as well as add-on open source software for the Apache web server, PHP scripting platform, and MySQL database. The report mentioned, though, that MySQL had five vulnerabilities that took more than 90 days to fix.
One critic of the report said it's difficult to measure the relative severity of vulnerabilities.
"There are so many ways to rate vulnerabilities and severities," said Johannes Ullrich, chief technology officer of the SANS Internet Storm Center, a service that reports on security vulnerabilities. "It's hard to come to up with an objective measure."
He also noted that a complete Linux distribution comes with a greater variety of software than Windows, making it larger, more complex, and more prone to vulnerabilities.
And the skills of the person running the system is extremely important to measuring how secure that system is, Ullrich added, "No operating system is secure unless you know how to apply the patches, configure the passwords, and disable services you don't need. You can't rely on a single security measure. You have to use firewalls and such to build up layered defenses. If you don't do that right, any operating system is vulnerable," he said.
Thompson expects he and his co-researcher will face charges of bias on behalf of Microsoft due to the company's funding of the study. "One of the big issues was to get the methodology out there. We knew people would question the results because of Microsoft's involvement in funding," he said.
He and Ford submitted their research proposal to Microsoft, Microsoft evaluated the proposal, and decided to fund it. Thompson said the researchers also sent the methodology to various analysts, including Charles Kolodgy of IDC, and had it vetted by various academics as well as people at the RSA Conference. ."
Asked if the study would have been published if the results had come out in favor of Linux, Thompson responded, "They certainly gave us input but I'm sure the results would ultimately have been published no matter what the outcome was."
In the report, the researchers cited an earlier study by Forrester Research that also attracted a fair amount of criticism from Linux proponents. Thompson expects to hear reaction from them again. "I'm sure we'll get a fair amount of creative input based on who funded this study," he said. He pointed out, however, that Security Innovation has a wide range of clients, including Hewlett-Packard, Cisco, and IBM, and his aim was to encourage feedback from the technology community about how the methodology can be optimized for future studies. "Certainly I hope that when the criticism comes, it comes on the methodology and our acts instead of loud commentary on who funded this particular study," he said.
While the current study examines Windows Server and Red Hat Enterprise Linux in Web server configurations, Thompson and Ford plan to conduct future comparisons of database server and workstation roles.