EMC Brings Data Analysis Breakthrough To Hadoop

EMC announced on Monday that it has resolved one of the big limitations of the Apache Hadoop platform by finding a way to use its Greenplum massively parallel processing (MPP) database to directly query the data in the Hadoop Distributed File System (HDFS).

Hadoop is catching on as a big data platform because it's super scalable, cost-effective and can flexibly handle a range of data types (also known as multi-structured data) without the data-modeling and transformation delays associated with relational databases. The big drawback, however, is that the options for analysis of data in Hadoop range from limited (through Hive, for example) to downright slow and complicated (through batch-oriented MapReduce processing). Plenty of vendors are working on fixes -- notably Cloudera with Impala and Hortonworks with HCatalog -- but EMC says its new Pivotal HD distribution has the problem nailed.

"It's well proven that people want SQL-based querying of the data in Hadoop, so we decoupled the file-system and SQL kernel dependencies of the Greenplum database and made it a default part of HDFS," explained Scott Yara, co-founder and VP of Products for EMC Greenplum.

[ Want more on Cloudera's rival Impala project? Read Cloudera Debuts Real-Time Hadoop Query. ]

EMC calls its integration of the Greenplum database into Hadoop HAWQ, and a key advantage of the combination is that it brings standard SQL querying to Hadoop. That's a contrast with the Hive component of Hadoop, which uses a SQL-like approach to support only a limited subset -- roughly 30%, by some estimates -- of standard SQL queries. What's more, HAWQ is 100 times to 600 times faster than Hive, according to EMC, because it doesn't require the SQL to be converted and executed as MapReduce jobs. Query response times are said to be in line with current BI and data warehousing service levels, and the distribution is compatible with both conventional BI and analytics platforms and emerging big-data analytics platforms such as Datameer, Karmasphere and Platfora.

"It's really cool to start seeing folks using a multi-structured data store as the storage layer for SQL-based analysis," said John Myers, senior analyst at Enterprise Management Associates, in an interview with InformationWeek. The combination will enable companies to use Hadoop as a single platform for both structured and multi-structured data, essentially combining data warehouses and Hadoop, Myers said. With HAWQ, business users and analysts can use conventional SQL querying and BI tools for their work while data scientist can continue to access date directly using programming APIs and Hadoop-related tools such as MapReduce, Pig, Hive, Sqoop and Mahout.

"We're going to see more single-platform approaches in the future, but Pivotal HD is one of the first emerging steps," Myers said.

Until today's announcement, Cloudera Impala has been the leading candidate to solve the gap in Hadoop analysis, but Impala is still in preview release, and Yara said Pivotal HD, which will be released by the end of March, will put EMC ahead of the market. "Cloudera is making bold claims, but I think Impala is five years away from being able to match the depth of capabilities that are built into Greenplum," he said.

A number of customers have beta tested Pivotal HD, according to EMC, but only one customer, Steven Hirsch, chief data officer, SVP Global Data Services, NYSE Euronext, was quoted in EMC's press release. "Pivotal HD provides true SQL query interfaces for data workers and tools -- not a superficial implementation of the kind that's so common today, but a native implementation that delivers the capability of real and true SQL processing and optimization," Hirsch said.

With Pivotal HD, EMC is "going all in" on Hadoop, Yara said, throwing down a gauntlet not only to Hadoop distribution leader Cloudera, but also to the likes of Oracle, IBM, Teradata and Microsoft. Yara declined to detail the number of customers who have deployed on EMC's Hadoop distributions to date. Pivotal HD will replace the company's existing Greenplum HD distribution and will include the Greenplum database.

Database rivals won't be able to quickly match HAWQ because the Greenplum database is the only one with a "distributed pipeline execution model for query processing," according to Yara. While older databases rely on materialized execution, Yara said Greenplum doesn't have to write data to disk with every intermediate query step. "We stream data to the next stage of a query plan in memory, and we never have to materialize the data to disk, so we're much faster than what anybody has demonstrated on Hadoop" Yara said.

Describing Hadoop as a once-in-a-generation transformation in information management, Yara said EMC is aiming to be "the leading Hadoop platform provider in the industry." That's a direct swipe at Cloudera, but it's also statement that IBM, Oracle and Teradata will likely feel compelled to respond to in the days, weeks or months ahead. EMC is clearly saying we're moving toward a convergence, rather than a coexistence, of database and Hadoop capabilities.

Attend Interop Las Vegas, May 6-10, and attend the most thorough training on Apple Deployment at the NEW Mac & iOS IT Conference. Use Priority Code DIPR02 by March 2 to save up to $500 off the price of Conference Passes. Join us in Las Vegas for access to 125+ workshops and conference classes, 350+ exhibiting companies, and the latest technology. Register for Interop today!