Big Data // Big Data Analytics
News
3/3/2014
09:06 AM
Connect Directly
Google+
LinkedIn
Twitter
RSS
E-Mail
50%
50%

In-Memory Databases: Do You Need The Speed?

IBM, Microsoft, Oracle, and SAP are ramping up the fight to become your in-memory technology provider.

Download the entire
March 3 issue of InformationWeek
,
In-Memory Databases, distributed in an all-digital format.

Here are three surefire facts you need to consider about in-memory databases, along with one hard question those facts should lead you to ask.

First, databases that take advantage of in-memory processing really do deliver the fastest data-retrieval speeds available today, which is enticing to companies struggling with high-scale online transactions or timely forecasting and planning.

Second, though disk-based storage is still the enterprise standard, the price of RAM has been declining steadily, so memory-intensive architectures will eventually replace slow, mechanical spinning disks.

Third, a vendor war is breaking out to convert companies to these in-memory systems. That war pits SAP -- application giant but database newbie -- against database incumbents IBM, Microsoft, and Oracle, which have big in-memory plans of their own.

Which leads to the question for would-be customers: So what if you can run a query or process transactions 10 times, 20 times, even 100 times faster than before? What will that difference do for your business? The answer will determine whether in-memory databases are a specialty tool for only a few unique use cases, or a platform upon which all of your enterprise IT runs.

Pioneer companies are finding compelling use cases. In-memory capabilities have let the online gaming company Bwin.party go from supporting 12,000 bets per second to supporting up to 150,000. That's money in the bank. For the retail services company Edgenet, in-memory technology has brought near-real-time insight into product availability for customers of AutoZone, Home Depot, and Lowe's. That translates into fewer wasted trips and higher customer satisfaction.

[The software giants might have the inside track, but there's no shortage of innovative, alternative providers. Read In-Memory Technology: The Options Abound.]

SAP's Hana in-memory database lets ConAgra test new approaches for material forecasting, planning, and pricing. "Accelerating something that exists isn't very interesting," Mindy Simon, ConAgra's vice president of IT, said at a recent SAP event. "We're trying to transform what we're doing with integrated forecasting."

ConAgra, an $18 billion-a-year consumer packaged goods company, must quickly respond to the fluctuating costs of 4,000 raw materials that go into more than 20,000 products, from Swiss Miss cocoa to Chef Boyardee pasta. What's more, if it could make its promotions timelier by using faster analysis, ConAgra and its retailer customers could command higher prices in a business known for razor-thin profit margins.

"If there's a big storm hitting the Northeast this weekend, for example, we want to make sure that Swiss Miss cocoa is available in that end-cap display," Simon said. But a company can't do that quickly if it has to wait for overnight data loads and hours-long planning runs to get a clear picture of in-store inventories, available product, and the profitability tied to various pricing scenarios.

With its Hana platform, SAP has been an early adopter and outspoken champion of in-memory technology, but it certainly doesn't have a monopoly on using RAM for processing transactions or analyzing data. Database incumbents IBM, Microsoft, and Oracle have introduced or announced their own in-memory capabilities and plans, as have other established vendors and startups. The key difference is that the big three incumbents are looking to defend legacy deployments and preserve license revenue by adding in-memory features to conventional databases. SAP is promising "radical simplification," with an entirely in-memory database that it says will eliminate layers of data management infrastructure and cut the overall tab for databases and information management.

As the in-memory war of words breaks out in 2014, the market is sure to get confusing. To cut through the hyperbole, here's a close look at what leading vendors are promising and, more importantly, what early adopters are saying about the business benefits.

SAP's four promises
If you're one of the 230,000 customers of SAP, you've been hearing about in-memory technology for more than three years, as it's at the heart of Hana. SAP laid out a grand plan for Hana in 2010 and has been trumpeting every advancement as a milestone ever since. It's now much more than a database management system, packing analytics, an application server, and other data management components into what the vendor calls the SAP Hana Platform. At the core of SAP's plans are four big claims about what Hana will do for customers.

First, SAP says Hana will deliver dramatically faster performance for applications. Second, SAP insists Hana can deliver this performance "without disruption" -- meaning customers don't have to rip out and replace their applications. Third, SAP says Hana can run transactional and analytical applications simultaneously, letting companies eliminate "redundant" infrastructure and "unnecessary" aggregates, materialized views, and other copies of data. Fourth, SAP says Hana will provide the foundation for new applications that weren't possible on conventional database technologies. ConAgra's idea for real-time pricing-and-profitability analysis is a case in point.

(Source: InformationWeek 2014 State of Database Technology Survey of 955 business technology professionals)
(Source: InformationWeek 2014 State of Database Technology Survey of 955 business technology professionals)

The first and biggest challenge for SAP is persuading customers to use Hana in place of the proven databases they've been using for years. Roughly half of SAP customers run their enterprise-grade applications on Oracle Database. Most other SAP customers use IBM DB2 and Microsoft SQL Server, in that order. SAP says it has more than 3,000 Hana customers, but that's just over 1% of its total customer base. IBM, Microsoft, and Oracle combined have upward of 1 million database customers. The dominance of the big three is confirmed by our recent InformationWeek 2014 State of Database Technology survey (see chart).

The value of faster performance
Nobody questions that in-memory performance beats disk-based performance. Estimates vary depending on disk speed and available input/output (I/O) bandwidth, but one expert puts RAM latency at 83 nanoseconds and disk latency at 13 milliseconds. With 1 million nanoseconds

Next Page

Doug Henschen is Executive Editor of InformationWeek, where he covers the intersection of enterprise applications with information management, business intelligence, big data and analytics. He previously served as editor in chief of Intelligent Enterprise, editor in chief of ... View Full Bio

Previous
1 of 3
Next
Comment  | 
Print  | 
More Insights
Comments
Newest First  |  Oldest First  |  Threaded View
Page 1 / 2   >   >>
D. Henschen
50%
50%
D. Henschen,
User Rank: Author
3/4/2014 | 12:37:47 PM
Re: In-memory DBMS vs. In-memory option
Paul,

Great post with deep insight. I believe SAP is addressing some of these application write needs on its own end, whereby its updates of SAP BusinessSuite and certainly its promised Suite On Hana deploymnets (particularly on SAP's cloud) will address these transactional requirements on SAP's end. SAP can rely on ASE, SQL Anywhere and IQ as part of the Hana "Platform," but you are right, SAP doesn't like to talk about anything but Hana.

I, for one, am hoping Hana does not take up another day of Keynotes. Let that be a wonky track. Do we need another session with Hasso Plattner and Vishal Sikka congratulating each other on their technical brilliance and the "never-before-possible applications... radical simplification... incredible performance improvement... yadda, yadda, yadda. Let the customers using Hana take over. It's year four for Hana, so it's time for the real-world case examples to come to the forefront.

If there are 100+ live with Suite on Hana, it should be no problem finding a few talkers.
PV21
IW Pick
50%
50%
PV21,
User Rank: Apprentice
3/4/2014 | 1:06:07 AM
Re: In-memory DBMS vs. In-memory option
Hi Doug, good article.


However I take issue with one of SAP's "4 promises" for HANA, specifically the claim that it runs applications faster.  I should be clear that except where stated otherwise I am talking about HANA as a database here, not "HANA the platform".

HANA is designed for very fast analytics plus acceptable (for some customers) OLTP performance.  Its columnar store puts it at huge disadvantage to row stores when running high volume, high concurrency OLTP workloads.  However, in SAP-land there are a number of mitigations:

(1) Rewrite all code better, to eliminate SELECT * and other unnecessarily long column lists that will trash any column store database.  This is fine, and good practice, just try getting everyone to do it with the non-SAP code that you don't control!

(2) Use stored procedures and SQL functions to massively reduce the hideous chattiness of the SAP application.  In other words, do what you should have done in the first place and ship functions not data.  Again this is good practice and an idea as old as the hills.  It will make your month-end processing run like lightning.  This is also where the boundaries between HANA as a database and HANA as a platform start to blur. SAP have decided that HANA will not use standard procedural SQL but its own dialect that they call SQL Script.  They give some justifications for this decision but they are spurious.  The real reason is competitive advantage.  By refusing to implement the stored procedure and functions approach for other databases, they ensure that HANA the platform is faster for such tasks than SAP using any other database.

(3) For what SAP term "extreme transaction processing" but other vendors like IBM and Oracle would view as no big deal, HANA cannot cope.  For these scenarios SAP refer to HANA in the platform sense, and bundle it with, ahem , Sybase ASE.  (See http://www.sap.com/pc/tech/real-time-data-platform/software/extreme-transaction-oltp/index.html for details of this bundle.)  The idea is to use Sybase to cope with the volumes that HANA can't, leaving HANA to run analytics - its true strength.

SAP are also pulling a trick or two on the analytics side, but this post is long enough already.

Paul
Li Tan
50%
50%
Li Tan,
User Rank: Ninja
3/3/2014 | 10:35:45 PM
Re: For some, a switch to SSDs might be enough
I think moving to SSD and deploying traditional DB on it may be a more feasible solution for most of enterprises. Migrating data to another DB such as Mongo may not be cost effectivite. Furthermore, it will create some disturbance to the ongoing business. The in-memory DB looks good and it's really fast, but the reliability is always a concern - how frequent should we make the checkpoint and take snapshot for the DB?
Charlie Babcock
50%
50%
Charlie Babcock,
User Rank: Author
3/3/2014 | 10:08:38 PM
For some, a switch to SSDs might be enough
SomeDude8: I too wonder about the SSD option alongside the in-memory database. For many people, switching from disk to SSD would be a good investment and gives applications better response times. But it's doesn't give the extra big yields in latency savings that the in-memory database operation does. Calls for data still have to exit the database server to an external device and return the data, which yields a 3X or 4X improvement, but not a 10X. But I suspect that a switch to SSDs makes a lot of sense for a lot of users in how it would decrease wait times and increase output in a highly non-disruptive manner.
D. Henschen
50%
50%
D. Henschen,
User Rank: Author
3/3/2014 | 9:01:28 PM
Teradata Covered Here... Re: Vendors
Teradata is an analytical database, so it's not in the transactional (OLTP) fray. That said, its "Intelligent Memory" feature, introduced last year, is covered in "In-Memory Technology: Options Abound."
Brian.Dean
50%
50%
Brian.Dean,
User Rank: Ninja
3/3/2014 | 6:43:23 PM
Re: Disruption
Yes, the Internet of things connections are predicted to reach a number close to 50 billion and much of these connections will only be able to deliver value if real-time analysis capability is present. And the slim profit margins that banks work upon indicates that banks will have to be increasingly online (electronic) in developing countries (where margins are slimmer) to provide service, bringing the unbanked population of the world down. This creates a need for faster and efficient hardware on which their current staff is comfortable, so that once expansion occurs -- demand does not overwhelm their systems.

 
ChrisMurphy
50%
50%
ChrisMurphy,
User Rank: Author
3/3/2014 | 6:34:20 PM
Re: Vendors
andrewise, Teradata is noted in this accompanying article:

http://www.informationweek.com/big-data/big-data-analytics/in-memory-technology-the-options-abound/d/d-id/1114082
Brian.Dean
50%
50%
Brian.Dean,
User Rank: Ninja
3/3/2014 | 6:24:12 PM
Re: Whoa...
Good point, if the economics of in-memory is not making sense for a firm at the moment than SSD should definitely be further investigated, not just become of the faster processing aspect but also because SSD has much lower failure rate. And SSD also provides that path into a hybrid in-momery and Flash solution.
ryanbetts
50%
50%
ryanbetts,
User Rank: Apprentice
3/3/2014 | 1:56:54 PM
Disruption
"Avoiding disruption is crucial for the three incumbents, because they want to keep and extend their business with database customers."

This is the key. Certainly all the macro trends are deeply disruptive. Fast mobile networks, broadly deployed sensors, real time electric grids ... all of the components of "Internet of Things," "Machine to Machine," and "Smart Planet" initiatives that are producing data that requires fast, real time processing at scales not possible in legacy database systems.

This disruption is driving multiple avenues of change in the in-memory database space. Most interesting to us, at VoltDB, is the use of in-memory decisioning and analytics to ingest, analyze, organize, respond to and archive incoming high velocity inputs. Combining analytics with transactional decision making on a per-event basis is necessary in a large class of use cases: real time alerting, alarming, authorization, complex policy enforcement, fraud, security and DDoS attack detection, micro-personalization and real time user segmentation.

Viable solutions must be virtualizable for cloud deployment, must offer integrated HA, must run on commodity cloud servers. Adding in-memory without eliminating the expense and complexity of shared storage is insufficient.

 

Ryan.
andrewise
50%
50%
andrewise,
User Rank: Apprentice
3/3/2014 | 12:03:53 PM
Vendors
I just dont understand why you didnt mention Teradata in-database systems...
Page 1 / 2   >   >>
6 Tools to Protect Big Data
6 Tools to Protect Big Data
Most IT teams have their conventional databases covered in terms of security and business continuity. But as we enter the era of big data, Hadoop, and NoSQL, protection schemes need to evolve. In fact, big data could drive the next big security strategy shift.
Register for InformationWeek Newsletters
White Papers
Current Issue
InformationWeek Must Reads Oct. 21, 2014
InformationWeek's new Must Reads is a compendium of our best recent coverage of digital strategy. Learn why you should learn to embrace DevOps, how to avoid roadblocks for digital projects, what the five steps to API management are, and more.
Video
Slideshows
Twitter Feed
InformationWeek Radio
Archived InformationWeek Radio
A roundup of the top stories and trends on InformationWeek.com
Sponsored Live Streaming Video
Everything You've Been Told About Mobility Is Wrong
Attend this video symposium with Sean Wisdom, Global Director of Mobility Solutions, and learn about how you can harness powerful new products to mobilize your business potential.