Software // Information Management
Commentary
9/3/2010
12:42 PM
Doug Henschen
Doug Henschen
Commentary
Connect Directly
Google+
LinkedIn
Twitter
RSS
E-Mail
50%
50%
Repost This

Reconsidering TPC Database Performance Benchmarks

Some say they are abstract if not downright misleading. But TPC benchmarks remain the only available standardized test of total-system performance and cost. Use them with caution.

Benchmarks can be incredibly useful. But the trust and value placed in such measures tends to increase with their independence from the products measured.

Take, for example, Energy Star efficiency specifications. Developed by the Environmental Protection Agency in 1992, Energy Star measures have since been heartily embraced by electronic equipment manufacturers and consumers alike.

Ditto Insurance Institute for Highway Safety (IIHS) vehicle tests. Some manufacturers initially complained about IIHS ratings, but now these ratings routinely show up in advertising.

Back in the mid 1980s, computer hardware manufacturers and database vendors recognized the need for standardized performance benchmarks for systems rather than individual components, such as disk drives and CPUs. Thus the Transaction Processing Performance Council was born.

Created and funded by vendors, the TPC formed committees to agree on standardized benchmark tests and procedures. And to ensure validity and consistency, the TPC established bylaws, policies, auditing procedures, test documentation requirements and more.

Today there are three core TPC benchmarks. TPC-C, the original and still most widely used benchmark, measures online transaction processing performance. TPC-H measures ad-hoc decision-support (analytic data mart/data warehouse) systems. The relatively new TPC-E benchmark simulates the OLTP workloads of a brokerage firm.

TPC also publishes standardized pricing and energy-consumption specifications that cut across all three benchmarks. The pricing specification details three-year total cost of ownership including hardware, software and maintenance. The energy-consumption specification was created in 2009 and shows up in the most recently published tests.

All recent benchmark test results are available online without restriction and at no cost. What's more, you can download executive summary reports with complete cost and energy-consumption detail as well as full reports with audited documentation on the complete test procedure.

This all sounds like a pretty good deal for would-be database buyers seeking unbiased insight on products. But the TPC benchmarks aren't without critics. Database expert and Intelligent Enterprise contributor Curt Monash wrote last year that most TPC benchmarks are "run on absurdly unrealistic hardware configurations."

Monash questioned one highly publicized TCP-H test result that was based on 32-to-1 disk-to-data ratio. He pointed out that production systems with good compression would likely run at closer to a 1-to-1 ratio.

Indeed, one of the weaknesses of the TPC program is that database and computer hardware vendors -- the members of the organization -- are free to test and publish results (or not) when and how they please.

"This is a voluntary organization, so we can't force people to publish benchmarks," explains Michael Majdalany, the TPC Administrator (equivalent to Executive Director). "The majority of vendors in the market do participate and publish benchmarks... [but] some established vendors don't feel the need to publish benchmarks because their attitude is that their customers known them already."

Previous
1 of 2
Next
Comment  | 
Print  | 
More Insights
The Agile Archive
The Agile Archive
When it comes to managing data, donít look at backup and archiving systems as burdens and cost centers. A well-designed archive can enhance data protection and restores, ease search and e-discovery efforts, and save money by intelligently moving data from expensive primary storage systems.
Register for InformationWeek Newsletters
White Papers
Current Issue
InformationWeek Elite 100 - 2014
Our InformationWeek Elite 100 issue -- our 26th ranking of technology innovators -- shines a spotlight on businesses that are succeeding because of their digital strategies. We take a close at look at the top five companies in this year's ranking and the eight winners of our Business Innovation awards, and offer 20 great ideas that you can use in your company. We also provide a ranked list of our Elite 100 innovators.
Video
Slideshows
Twitter Feed
Audio Interviews
Archived Audio Interviews
GE is a leader in combining connected devices and advanced analytics in pursuit of practical goals like less downtime, lower operating costs, and higher throughput. At GIO Power & Water, CIO Jim Fowler is part of the team exploring how to apply these techniques to some of the world's essential infrastructure, from power plants to water treatment systems. Join us, and bring your questions, as we talk about what's ahead.