IBM SoftLayer Performance Less Impressive Than Claimed - InformationWeek

InformationWeek is part of the Informa Tech Division of Informa PLC

This site is operated by a business or businesses owned by Informa PLC and all copyright resides with them.Informa PLC's registered office is 5 Howick Place, London SW1P 1WG. Registered in England and Wales. Number 8860726.

IoT
IoT
Cloud // Infrastructure as a Service
News
11/23/2015
10:29 AM
Connect Directly
Twitter
RSS
E-Mail

IBM SoftLayer Performance Less Impressive Than Claimed

VoltDB's latest cloud performance benchmark claims IBM SoftLayer is 3X faster than Amazon, but that's only true if you compare apples to oranges.

(Image: DKart/iStockphoto)

(Image: DKart/iStockphoto)

Comment  | 
Print  | 
Comments
Oldest First  |  Newest First  |  Threaded View
georgeacosta
50%
50%
georgeacosta,
User Rank: Apprentice
11/23/2015 | 3:58:33 PM
How it is Apples and Oranges?
I think you're missing the point on what cloud computing actually is.  Cloud implies on-demand compute with consumptive billing, and if IBM SoftLayer can provision hourly bare metal compute at a lower cost and higher performance than other provisers virtual compute, what's the difference?

You quote the price to performance ratios in your piece:  "4.630 billion operations per dollar for bare-metal SoftLayer, 3.338 billion operations per dollar for Google, 1.608 billion operations per dollar for AWS, and 0.415 billion operations per dollar for Microsoft".  These numbers speak for themselves.  

 
P3rfguy
50%
50%
P3rfguy,
User Rank: Apprentice
11/24/2015 | 10:41:35 AM
Open, fair, and consistent benchmarking - PerfKit Benchmarker
PerfKit Benchmaker is designed to help customers and the industry answer the vary apples to apples (or oranges) questions posed here.  What is does is make cross-cloud comparison easy and uses a consistent methodology when running popular benchmarks.  Check it out here - https://github.com/GoogleCloudPlatform/PerfKitBenchmarker/wiki
acmebenchmarking
50%
50%
acmebenchmarking,
User Rank: Apprentice
11/26/2015 | 6:42:46 PM
Apples to Oranges?
I don't get the use of "apples to oranges" here. If the apples and oranges cost the same, the consumer gets to choose whichever one they want.
tzubair
50%
50%
tzubair,
User Rank: Ninja
11/29/2015 | 5:37:46 PM
Re: Open, fair, and consistent benchmarking - PerfKit Benchmarker
"PerfKit Benchmaker is designed to help customers and the industry answer the vary apples to apples (or oranges) questions posed here.  What is does is make cross-cloud comparison easy and uses a consistent methodology when running popular benchmarks"

@P3rfguy: What is the criteria being used to benchmark? On what factors is the comparison being drawn to compare two cloud services? How reliable are the results? Has the app been endorsed by an independent authority to certify that it gives the right results?
tzubair
50%
50%
tzubair,
User Rank: Ninja
11/29/2015 | 5:44:38 PM
Re: How it is Apples and Oranges?
@georgeacosta: While calculating the number of operations per dollar spent is a good measure, it may not be the best measure. It may not always truly reflect the outcome of a cloud solution. A cloud solution also yields certain long term benefits which aren't being taken into account here. Further, the quality and reliability matter too, not just the number of operations.
MichaelS401
100%
0%
MichaelS401,
User Rank: Apprentice
12/4/2015 | 5:01:41 PM
Make sure you look at the underlying numbers in that study.
Charles, I think you should delve a bit deeper.  The virtual softlayer instance had 50% less cores than the 32-way Amazon VM, yet was only 30% slower.  In other words, processor to processor it had better performance and better price performance.  The bare metal server, as you astutely put it, blew Amazon out of the water. That said, I disagree with you stating Apples and Oranges.  Both are cloud offerings, both offer consumptive billing - why would anyone pay more for a slower, less performant, less secure offering?  Especially one where the performance varies but the price doesn't.  

 

As one of my mentors says, I'm just sayin'; I'm not shoutin'.   
batye
50%
50%
batye,
User Rank: Ninja
12/6/2015 | 12:38:27 AM
Re: Apples to Oranges?
@acmebenchmarking, I trust you are right... but this days with educated consumer you never know....
Slideshows
What Digital Transformation Is (And Isn't)
Cynthia Harvey, Freelance Journalist, InformationWeek,  12/4/2019
Commentary
Watch Out for New Barriers to Faster Software Development
Lisa Morgan, Freelance Writer,  12/3/2019
Commentary
If DevOps Is So Awesome, Why Is Your Initiative Failing?
Guest Commentary, Guest Commentary,  12/2/2019
White Papers
Register for InformationWeek Newsletters
Video
Current Issue
Getting Started With Emerging Technologies
Looking to help your enterprise IT team ease the stress of putting new/emerging technologies such as AI, machine learning and IoT to work for their organizations? There are a few ways to get off on the right foot. In this report we share some expert advice on how to approach some of these seemingly daunting tech challenges.
Slideshows
Flash Poll