IBM SoftLayer Performance Less Impressive Than Claimed - InformationWeek
IoT
IoT
Cloud // Infrastructure as a Service
News
11/23/2015
10:29 AM
Connect Directly
Twitter
RSS
E-Mail
50%
50%

IBM SoftLayer Performance Less Impressive Than Claimed

VoltDB's latest cloud performance benchmark claims IBM SoftLayer is 3X faster than Amazon, but that's only true if you compare apples to oranges.

8 Hot Tech Jobs Getting Big Salary Bumps In 2016
8 Hot Tech Jobs Getting Big Salary Bumps In 2016
(Click image for larger view and slideshow.)

A benchmark purported to illustrate the speed of the IBM SoftLayer cloud actually tells an opposite story. At best it shows that bare-metal servers run faster than virtual machines in the cloud, as they do in the data center, but most experienced IT managers already knew that.

What it also illustrates is that Google, Microsoft, and Amazon run faster than IBM when it comes to running normal, virtual machine workloads. It's the second year that IBM and VoltDB have played up the results of a benchmark using the Yahoo Cloud Serving Benchmark, an open source standard out of Yahoo Labs that tests a variety of workloads.

VoltDB is an in-memory database system designed by Michael Stonebreaker and used for the cloud benchmarking test. An experienced benchmarking third party, Tim Callaghan of Wakefield, Mass., conducted the test under the name of Acme Benchmarking. He is noted for having done a number of MongoDB and database vs. database benchmarks.

Damion Heredia, VP of platform services for IBM Cloud, called attention to the benchmark in an interview with InformationWeek November 18 at IBM offices in San Francisco.

He cited IBM's advantages in cloud computing, including its commitment to the open source Apache Spark Project as a flexible system for handling big data. He cited Watson analytical services that can be applied "to a particular domain" to come up with new, vertical industry insights. He also cited the many tools and capabilities of the Bluemix platform-as-a-service, "the largest implementation of Cloud Foundry in the world."

[Want to learn more about VoltDB? Meet VoltDB: Relational Database With NoSQL Speed.]

But the linchpin for these attributes was SoftLayer's underlying speed of execution. That speed was established in the Callaghan benchmark, touted by IBM spokesmen before the interview as illustrating SoftLayer "to be nearly three times faster than AWS."

Heredia didn't repeat that claim, but other parties have. VoltDB did a similar benchmark test a year ago, citing favorable results when its system ran on IBM SoftLayer. Forbes writer Ben Kepes picked up on the results in August 2014 and said they belie Amazon Web Services' claims for competitive performance. "VoltDB is coming out today saying that customers who use its product on IBM SoftLayer cloud infrastructure see five times faster application performance than those running it on Amazon Web Services," wrote Kepes, principal of the technology investment advisory firm Diversity Limited.

If nothing else, the claim reflects the nervousness of database vendors as Amazon launches competing database services on its own cloud. Amazon's Kinesis in-memory system could be construed as a competitor with VoltDB, a point that Kepes noted at the time.

This year, the tests were done again, this time by Callaghan (a third party) rather than VoltDB tester Alex Rogers, who conducted the 2014 tests. Contacted by InformationWeek, a VoltDB spokesman said IBM played no role in financing the tests. Callaghan explained in a disclaimer note on his Acme site: "Someone from VoltDB contacted me in June asking if I'd be interested in performing a cloud vs. cloud benchmark, offering to compensate me for my time and to cover any cloud vendor bills." Callaghan agreed to do the benchmark.

(Image: DKart/iStockphoto)

(Image: DKart/iStockphoto)

So did it really show SoftLayer is 3X (scaled back from last year's 5X) faster than AWS? Only if you like comparing apples to oranges. The part of the benchmark where SoftLayer outstrips Amazon compares VoltDB performance on bare-metal SoftLayer servers to AWS EC2 instances. EC2 has no bare-metal cloud service and must spin up its version of a virtual machine, an Amazon Machine Image, as a server. Once running, that AMI will have the overhead of a hypervisor and its virtual switch with which to complete the tasks assigned to it.

The benchmark showed that SoftLayer, with its bare-metal servers, could conduct 1,543,000 operations per second using VoltDB, compared to AWS's 585,000 ops/second, or 2.6 times the performance of AWS. But once it became a VM-to-VM comparison, which went unmentioned in the press release, SoftLayer's performance was worse that AWS's. No numbers were cited for the latter benchmark, but a graph published with the press release shows SoftLayer virtual at about 410,000 or 420,000 ops/second.

How did Microsoft and Google fare in this benchmark? They too were slower than IBM's bare-metal servers. Google Compute Engine produced the highest virtual machine performance, at 1,038,586 ops/second. Microsoft was second with 902,017 ops/sec.

The benchmark also produced a price performance measure, which came out at 4.630 billion operations per dollar for bare-metal SoftLayer, 3.338 billion operations per dollar for Google, 1.608 billion operations per dollar for AWS, and 0.415 billion operations per dollar for Microsoft, with an explanatory note: "due to limited configuration options for the benchmark." There was no price performance figure provided for SoftLayer running virtual machines.

The VoltDB benchmark therefore should be taken with a large grain of salt. A bare-metal server in the cloud is a single-tenant server. It's not safe for either privacy or data integrity to crowd multiple tenants on bare-metal servers, and no cloud providers do it. Rackspace and IBM are the primary cloud providers who supply bare-metal servers. Amazon, Google, and Microsoft do not.

Using typical cloud workloads spun up in a virtual machine turns the claimed results of the benchmark on its head. Google shows up with cost-efficient operations, followed by Amazon and Microsoft, with SoftLayer a distant fourth.

New deadline of Dec. 18, 2015 Be a part of the prestigious InformationWeek Elite 100! Time is running out to submit your company's application by Dec. 18, 2015. Go to our 2016 registration page: InformationWeek's Elite 100 list for 2016.

Charles Babcock is an editor-at-large for InformationWeek and author of Management Strategies for the Cloud Revolution, a McGraw-Hill book. He is the former editor-in-chief of Digital News, former software editor of Computerworld and former technology editor of Interactive ... View Full Bio

We welcome your comments on this topic on our social media channels, or [contact us directly] with questions about the site.
Comment  | 
Print  | 
More Insights
Comments
Newest First  |  Oldest First  |  Threaded View
batye
50%
50%
batye,
User Rank: Ninja
12/6/2015 | 12:38:27 AM
Re: Apples to Oranges?
@acmebenchmarking, I trust you are right... but this days with educated consumer you never know....
MichaelS401
100%
0%
MichaelS401,
User Rank: Apprentice
12/4/2015 | 5:01:41 PM
Make sure you look at the underlying numbers in that study.
Charles, I think you should delve a bit deeper.  The virtual softlayer instance had 50% less cores than the 32-way Amazon VM, yet was only 30% slower.  In other words, processor to processor it had better performance and better price performance.  The bare metal server, as you astutely put it, blew Amazon out of the water. That said, I disagree with you stating Apples and Oranges.  Both are cloud offerings, both offer consumptive billing - why would anyone pay more for a slower, less performant, less secure offering?  Especially one where the performance varies but the price doesn't.  

 

As one of my mentors says, I'm just sayin'; I'm not shoutin'.   
tzubair
50%
50%
tzubair,
User Rank: Ninja
11/29/2015 | 5:44:38 PM
Re: How it is Apples and Oranges?
@georgeacosta: While calculating the number of operations per dollar spent is a good measure, it may not be the best measure. It may not always truly reflect the outcome of a cloud solution. A cloud solution also yields certain long term benefits which aren't being taken into account here. Further, the quality and reliability matter too, not just the number of operations.
tzubair
50%
50%
tzubair,
User Rank: Ninja
11/29/2015 | 5:37:46 PM
Re: Open, fair, and consistent benchmarking - PerfKit Benchmarker
"PerfKit Benchmaker is designed to help customers and the industry answer the vary apples to apples (or oranges) questions posed here.  What is does is make cross-cloud comparison easy and uses a consistent methodology when running popular benchmarks"

@P3rfguy: What is the criteria being used to benchmark? On what factors is the comparison being drawn to compare two cloud services? How reliable are the results? Has the app been endorsed by an independent authority to certify that it gives the right results?
acmebenchmarking
50%
50%
acmebenchmarking,
User Rank: Apprentice
11/26/2015 | 6:42:46 PM
Apples to Oranges?
I don't get the use of "apples to oranges" here. If the apples and oranges cost the same, the consumer gets to choose whichever one they want.
P3rfguy
50%
50%
P3rfguy,
User Rank: Apprentice
11/24/2015 | 10:41:35 AM
Open, fair, and consistent benchmarking - PerfKit Benchmarker
PerfKit Benchmaker is designed to help customers and the industry answer the vary apples to apples (or oranges) questions posed here.  What is does is make cross-cloud comparison easy and uses a consistent methodology when running popular benchmarks.  Check it out here - https://github.com/GoogleCloudPlatform/PerfKitBenchmarker/wiki
georgeacosta
50%
50%
georgeacosta,
User Rank: Apprentice
11/23/2015 | 3:58:33 PM
How it is Apples and Oranges?
I think you're missing the point on what cloud computing actually is.  Cloud implies on-demand compute with consumptive billing, and if IBM SoftLayer can provision hourly bare metal compute at a lower cost and higher performance than other provisers virtual compute, what's the difference?

You quote the price to performance ratios in your piece:  "4.630 billion operations per dollar for bare-metal SoftLayer, 3.338 billion operations per dollar for Google, 1.608 billion operations per dollar for AWS, and 0.415 billion operations per dollar for Microsoft".  These numbers speak for themselves.  

 
Register for InformationWeek Newsletters
White Papers
Current Issue
The Next Generation of IT Support
The workforce is changing as businesses become global and technology erodes geographical and physical barriers.IT organizations are critical to enabling this transition and can utilize next-generation tools and strategies to provide world-class support regardless of location, platform or device
Video
Slideshows
Twitter Feed
Sponsored Live Streaming Video
Everything You've Been Told About Mobility Is Wrong
Attend this video symposium with Sean Wisdom, Global Director of Mobility Solutions, and learn about how you can harness powerful new products to mobilize your business potential.
Flash Poll