Software // Information Management
News
2/25/2013
01:34 PM
Connect Directly
LinkedIn
Twitter
Google+
RSS
E-Mail
50%
50%

EMC Brings Data Analysis Breakthrough To Hadoop

EMC's new Pivotal HD Hadoop distribution brings direct SQL querying to data stored on a big data cluster.

EMC announced on Monday that it has resolved one of the big limitations of the Apache Hadoop platform by finding a way to use its Greenplum massively parallel processing (MPP) database to directly query the data in the Hadoop Distributed File System (HDFS).

Hadoop is catching on as a big data platform because it's super scalable, cost-effective and can flexibly handle a range of data types (also known as multi-structured data) without the data-modeling and transformation delays associated with relational databases. The big drawback, however, is that the options for analysis of data in Hadoop range from limited (through Hive, for example) to downright slow and complicated (through batch-oriented MapReduce processing). Plenty of vendors are working on fixes -- notably Cloudera with Impala and Hortonworks with HCatalog -- but EMC says its new Pivotal HD distribution has the problem nailed.

"It's well proven that people want SQL-based querying of the data in Hadoop, so we decoupled the file-system and SQL kernel dependencies of the Greenplum database and made it a default part of HDFS," explained Scott Yara, co-founder and VP of Products for EMC Greenplum.

[ Want more on Cloudera's rival Impala project? Read Cloudera Debuts Real-Time Hadoop Query. ]

EMC calls its integration of the Greenplum database into Hadoop HAWQ, and a key advantage of the combination is that it brings standard SQL querying to Hadoop. That's a contrast with the Hive component of Hadoop, which uses a SQL-like approach to support only a limited subset -- roughly 30%, by some estimates -- of standard SQL queries. What's more, HAWQ is 100 times to 600 times faster than Hive, according to EMC, because it doesn't require the SQL to be converted and executed as MapReduce jobs. Query response times are said to be in line with current BI and data warehousing service levels, and the distribution is compatible with both conventional BI and analytics platforms and emerging big-data analytics platforms such as Datameer, Karmasphere and Platfora.

"It's really cool to start seeing folks using a multi-structured data store as the storage layer for SQL-based analysis," said John Myers, senior analyst at Enterprise Management Associates, in an interview with InformationWeek. The combination will enable companies to use Hadoop as a single platform for both structured and multi-structured data, essentially combining data warehouses and Hadoop, Myers said. With HAWQ, business users and analysts can use conventional SQL querying and BI tools for their work while data scientist can continue to access date directly using programming APIs and Hadoop-related tools such as MapReduce, Pig, Hive, Sqoop and Mahout.

"We're going to see more single-platform approaches in the future, but Pivotal HD is one of the first emerging steps," Myers said.

Until today's announcement, Cloudera Impala has been the leading candidate to solve the gap in Hadoop analysis, but Impala is still in preview release, and Yara said Pivotal HD, which will be released by the end of March, will put EMC ahead of the market. "Cloudera is making bold claims, but I think Impala is five years away from being able to match the depth of capabilities that are built into Greenplum," he said.

A number of customers have beta tested Pivotal HD, according to EMC, but only one customer, Steven Hirsch, chief data officer, SVP Global Data Services, NYSE Euronext, was quoted in EMC's press release. "Pivotal HD provides true SQL query interfaces for data workers and tools -- not a superficial implementation of the kind that's so common today, but a native implementation that delivers the capability of real and true SQL processing and optimization," Hirsch said.

With Pivotal HD, EMC is "going all in" on Hadoop, Yara said, throwing down a gauntlet not only to Hadoop distribution leader Cloudera, but also to the likes of Oracle, IBM, Teradata and Microsoft. Yara declined to detail the number of customers who have deployed on EMC's Hadoop distributions to date. Pivotal HD will replace the company's existing Greenplum HD distribution and will include the Greenplum database.

Database rivals won't be able to quickly match HAWQ because the Greenplum database is the only one with a "distributed pipeline execution model for query processing," according to Yara. While older databases rely on materialized execution, Yara said Greenplum doesn't have to write data to disk with every intermediate query step. "We stream data to the next stage of a query plan in memory, and we never have to materialize the data to disk, so we're much faster than what anybody has demonstrated on Hadoop" Yara said.

Describing Hadoop as a once-in-a-generation transformation in information management, Yara said EMC is aiming to be "the leading Hadoop platform provider in the industry." That's a direct swipe at Cloudera, but it's also statement that IBM, Oracle and Teradata will likely feel compelled to respond to in the days, weeks or months ahead. EMC is clearly saying we're moving toward a convergence, rather than a coexistence, of database and Hadoop capabilities.

Attend Interop Las Vegas, May 6-10, and attend the most thorough training on Apple Deployment at the NEW Mac & iOS IT Conference. Use Priority Code DIPR02 by March 2 to save up to $500 off the price of Conference Passes. Join us in Las Vegas for access to 125+ workshops and conference classes, 350+ exhibiting companies, and the latest technology. Register for Interop today!

Comment  | 
Print  | 
More Insights
Comments
Newest First  |  Oldest First  |  Threaded View
D. Henschen
50%
50%
D. Henschen,
User Rank: Author
2/25/2013 | 10:33:18 PM
re: EMC Brings Data Analysis Breakthrough To Hadoop
The cost savings discussions are quite often around the per-terabyte cost of Hadoop vs. more conventional relational platforms. That's not really the issue here. It's assumed that Hadoop is the scale champion and that it's capable of handling multistructured data -- something relational databases don't do well.

With this announcement EMC is addressing the same Hadoop flaw that Cloudera is addressing with Impala -- supporting SQL querying that is faster and broader than the primitive, SQL-like querying supported by Hive. Yes, speed of query is important, but just getting access to standard-SQL querying is a big win. This is something for people who are already sold on Hadoop... not for those who are still tire kicking and wondering if they should even get into big data.
ChrisMurphy
50%
50%
ChrisMurphy,
User Rank: Author
2/25/2013 | 10:15:00 PM
re: EMC Brings Data Analysis Breakthrough To Hadoop
We definitely see a lot of tire-kicking around Hadoop, a lot of pilot tests and small-scale projects. But the appeal often leans heavily on potential cost savings, not just the speed. Is cost savings a big part of the pitch from EMC, or just a performance focus?
D. Henschen
50%
50%
D. Henschen,
User Rank: Author
2/25/2013 | 9:32:08 PM
re: EMC Brings Data Analysis Breakthrough To Hadoop
Cloudera is trying to solve the speedy data-analysis problem with Impala. EMC says database & Hadoop should converge, not coexist... and they say they'll support this by the end of March. Bold promise, but there's only one reference cusotmer and it's not clear if they have beta-tested the capability.
The Agile Archive
The Agile Archive
When it comes to managing data, donít look at backup and archiving systems as burdens and cost centers. A well-designed archive can enhance data protection and restores, ease search and e-discovery efforts, and save money by intelligently moving data from expensive primary storage systems.
Register for InformationWeek Newsletters
White Papers
Current Issue
InformationWeek Tech Digest, Nov. 10, 2014
Just 30% of respondents to our new survey say their companies are very or extremely effective at identifying critical data and analyzing it to make decisions, down from 42% in 2013. What gives?
Video
Slideshows
Twitter Feed
InformationWeek Radio
Archived InformationWeek Radio
Join us for a roundup of the top stories on InformationWeek.com for the week of November 16, 2014.
Sponsored Live Streaming Video
Everything You've Been Told About Mobility Is Wrong
Attend this video symposium with Sean Wisdom, Global Director of Mobility Solutions, and learn about how you can harness powerful new products to mobilize your business potential.