IBM SoftLayer Performance Less Impressive Than Claimed

8 Hot Tech Jobs Getting Big Salary Bumps In 2016

A benchmark purported to illustrate the speed of the IBM SoftLayer cloud actually tells an opposite story. At best it shows that bare-metal servers run faster than virtual machines in the cloud, as they do in the data center, but most experienced IT managers already knew that.

What it also illustrates is that Google, Microsoft, and Amazon run faster than IBM when it comes to running normal, virtual machine workloads. It's the second year that IBM and VoltDB have played up the results of a benchmark using the Yahoo Cloud Serving Benchmark, an open source standard out of Yahoo Labs that tests a variety of workloads.

VoltDB is an in-memory database system designed by Michael Stonebreaker and used for the cloud benchmarking test. An experienced benchmarking third party, Tim Callaghan of Wakefield, Mass., conducted the test under the name of Acme Benchmarking. He is noted for having done a number of MongoDB and database vs. database benchmarks.

Damion Heredia, VP of platform services for IBM Cloud, called attention to the benchmark in an interview with InformationWeek November 18 at IBM offices in San Francisco.

He cited IBM's advantages in cloud computing, including its commitment to the open source Apache Spark Project as a flexible system for handling big data. He cited Watson analytical services that can be applied "to a particular domain" to come up with new, vertical industry insights. He also cited the many tools and capabilities of the Bluemix platform-as-a-service, "the largest implementation of Cloud Foundry in the world."

[Want to learn more about VoltDB? Meet VoltDB: Relational Database With NoSQL Speed.]

But the linchpin for these attributes was SoftLayer's underlying speed of execution. That speed was established in the Callaghan benchmark, touted by IBM spokesmen before the interview as illustrating SoftLayer "to be nearly three times faster than AWS."

Heredia didn't repeat that claim, but other parties have. VoltDB did a similar benchmark test a year ago, citing favorable results when its system ran on IBM SoftLayer. Forbes writer Ben Kepes picked up on the results in August 2014 and said they belie Amazon Web Services' claims for competitive performance. "VoltDB is coming out today saying that customers who use its product on IBM SoftLayer cloud infrastructure see five times faster application performance than those running it on Amazon Web Services," wrote Kepes, principal of the technology investment advisory firm Diversity Limited.

If nothing else, the claim reflects the nervousness of database vendors as Amazon launches competing database services on its own cloud. Amazon's Kinesis in-memory system could be construed as a competitor with VoltDB, a point that Kepes noted at the time.

This year, the tests were done again, this time by Callaghan (a third party) rather than VoltDB tester Alex Rogers, who conducted the 2014 tests. Contacted by InformationWeek, a VoltDB spokesman said IBM played no role in financing the tests. Callaghan explained in a disclaimer note on his Acme site: "Someone from VoltDB contacted me in June asking if I'd be interested in performing a cloud vs. cloud benchmark, offering to compensate me for my time and to cover any cloud vendor bills." Callaghan agreed to do the benchmark.

So did it really show SoftLayer is 3X (scaled back from last year's 5X) faster than AWS? Only if you like comparing apples to oranges. The part of the benchmark where SoftLayer outstrips Amazon compares VoltDB performance on bare-metal SoftLayer servers to AWS EC2 instances. EC2 has no bare-metal cloud service and must spin up its version of a virtual machine, an Amazon Machine Image, as a server. Once running, that AMI will have the overhead of a hypervisor and its virtual switch with which to complete the tasks assigned to it.

The benchmark showed that SoftLayer, with its bare-metal servers, could conduct 1,543,000 operations per second using VoltDB, compared to AWS's 585,000 ops/second, or 2.6 times the performance of AWS. But once it became a VM-to-VM comparison, which went unmentioned in the press release, SoftLayer's performance was worse that AWS's. No numbers were cited for the latter benchmark, but a graph published with the press release shows SoftLayer virtual at about 410,000 or 420,000 ops/second.

How did Microsoft and Google fare in this benchmark? They too were slower than IBM's bare-metal servers. Google Compute Engine produced the highest virtual machine performance, at 1,038,586 ops/second. Microsoft was second with 902,017 ops/sec.

The benchmark also produced a price performance measure, which came out at 4.630 billion operations per dollar for bare-metal SoftLayer, 3.338 billion operations per dollar for Google, 1.608 billion operations per dollar for AWS, and 0.415 billion operations per dollar for Microsoft, with an explanatory note: "due to limited configuration options for the benchmark." There was no price performance figure provided for SoftLayer running virtual machines.

The VoltDB benchmark therefore should be taken with a large grain of salt. A bare-metal server in the cloud is a single-tenant server. It's not safe for either privacy or data integrity to crowd multiple tenants on bare-metal servers, and no cloud providers do it. Rackspace and IBM are the primary cloud providers who supply bare-metal servers. Amazon, Google, and Microsoft do not.

Using typical cloud workloads spun up in a virtual machine turns the claimed results of the benchmark on its head. Google shows up with cost-efficient operations, followed by Amazon and Microsoft, with SoftLayer a distant fourth.

New deadline of Dec. 18, 2015 Be a part of the prestigious InformationWeek Elite 100! Time is running out to submit your company's application by Dec. 18, 2015. Go to our 2016 registration page: InformationWeek's Elite 100 list for 2016.