InformationWeek.com

Welcome Guest. | Log In| Register | Membership Benefits

July 5, 1999

TechView:
Benchmarks: The Best Test Is Your Own

ecently, another publication (which will remain nameless here) sponsored a benchmark showdown between Microsoft Windows NT and Linux, and then retested them after questions were raised about Microsoft's sponsorship of the benchmarks. Not surprisingly, NT won again on the retest.

I'm not questioning the validity of the scoring of either of these benchmarks--especially since I didn't participate in them and I don't have access to the methodologies or source code used. What I do question is the validity of using any benchmark as the sole method of selecting a development platform or operating system, particularly when they're someone else's proprietary benchmarks.

Benchmarking is a tricky business to begin with. It's a problem common to any sort of quantitative analysis. In quantum mechanics, Heisenberg's Uncertainty Principle states that the simple act of measuring in itself affects what's being measured. Taking pains to measure one metric with great accuracy reduces the accuracy with which you can measure related metrics. Bottom line: Benchmarks are voodoo.

It's always been my philosophy--and the philosophy of InformationWeek Labs--that performance is only a small part of the equation when it comes to evaluating any technology. The real performance benchmark is the performance of your own applications in your own environment, as well as the productivity of your own developers and IT staff.

On top of that, a lab benchmark can't begin to measure how well a technology will work in the hands of your staff, or how well it will leverage their previous experience and training. While your application and environment may be similar to someone else's, none of those elements can be accurately reproduced in a lab environment. There is no substitute for your own testing.

In the end, that's the main flaw with any external source of information about any technology. Those providing the information may have similar ideas about what's important about a certain technology, but they most likely apply different weights to them. That goes double or triple for analysts and consultants who charge both vendors (to participate in evaluations) and customers (to receive thick binders filled with what they say about the vendors).

So, if all external information sources are suspect, where does that leave you? There's only one sure way to make sure you get good information, and that's to get it yourself--either through your own testing or by holding those who do the testing accountable to your own metrics. The best purpose that external sources can serve is to narrow down the field to the technologies that apply to your needs.

Back to Labs