Cloudera Debuts Real-Time Hadoop Query

Adding a new component for real-time querying to its Hadoop software distribution, Cloudera introduced Cloudera Impala on Wednesday at the Strata Conference in New York. Developed in stealth mode and now in public beta, the software takes on one of Hadoop's biggest flaws: batch-oriented processing delays and poor access to data.

Impala is an interactive-speed SQL query engine that runs on existing Hadoop infrastructure. It makes all the data in the Hadoop Distributed File System (HDFS) and Apache HBase database tables accessible for real-time querying. With that it promises to open up Hadoop to relational databases and conventional business intelligence (BI) tools that rely on SQL querying.

Hadoop already offers a degree of structured-data access via Apache Hive, but that data warehousing infrastructure lacks support and suffers from poor performance because it turns SQL queries into MapReduce jobs. Hive also supports less than 30% of the analyses supported in SQL, according to critics. With Impala, Cloudera is promising a much better answer, Olson says.

[ Want more on this week's Hadoop announcements? Read Microsoft Releases Hadoop On Windows. ]

"Our view is that, long-term, this will supplant Hive, but right now Hive has a huge installed base and lots of applications running against it, so I don't imaging that [Impala] will replace Hive quickly," Cloudera CEO Mike Olson told InformationWeek. "Because it's real time, Impala is going to be much more attractive."

Impala is a two-part product. At the core is the Impala real-time query engine, which is shipping under Apache open source license. Cloudera says it intends to submit it to the Apache Software Foundation once it's proven in the field and hardened for production use.

Hadoop users will be able to use the engine in stand-alone fashion, according to Olson, but a new Cloudera Enterprise RTQ component of the vendor's proprietary Cloudera Manager administrative console will manage the Impala server. Dashboards in the management console will show the queries that are running, how long they're taking, how many users are active, how many tables exist, and so on.

Cloudera has yet to run benchmark tests on Impala, but Olson said users will "start seeing results immediately" upon issuing a query. That's not to say it will deliver relational-database-like performance, Olson granted, but a 3X to 30X improvement over Hive will lead Hadoop users away from that interface, he said.

Impala sounds like a godsend to legacy business intelligence vendors, which have been stuck with the compromises of moving data out of Hadoop through slow manual processes or putting up with the delays and limitations of Hive. In ad hoc query-and-analysis scenarios, Impala will give BI tools direct access to data in Hadoop, but production, mission-critical workloads will likely remain in relational databases, Olson said.

"Some workloads will clearly move off of the high-powered data warehouses into Hadoop with Impala, but if you need to fly through a deep, [multidimensional] cube at high speed, that's going to stay in the data warehouse," he said. "The OLAP engines that run in big data warehouses have special-purpose interfaces and data-summarization and aggregation support that's not a part of the SQL language and not a part of Impala." (Interestingly, OLAP is threatened by in-memory technology, so one wonders if that, too, could be implemented in Hadoop.)

Cloudera has a number of customers beta testing Impala, two of which are going public. Monsanto has research scientists all over the world who want to collaborate on disease- and weed-resistant genomes, but research data has been locked in siloed systems. By consolidating all that information in Hadoop and using Impala to offer speedy SQL query access, Monsanto is building out a collaborative, interactive environment spanning all of Monsanto's research.

The travel booking site Expedia manages more than 4 petabytes of data using Cloudera, and it is experimenting with Impala as a way to quickly understand which users are booking which trips, and what destinations and air, car, and hotel options are winning and losing.

"This evolution of Hadoop has enabled us to reduce our latency by 50% and produce a new business insight service not previously viable," said Jeff Prather, Expedia's director of global business intelligence and data warehousing platforms, in a statement issued by Cloudera.

Olson encouraged the Hadoop Community to download the Cloudera Impala beta during the kickoff keynote at Strata on Wednesday, as adoption is the key to testing, proving, and bringing the technology into production.

Whether the core real-time engine gains broader adoption than Hive--and an embrace from Cloudera competitors like Hortonworks and MapR--has yet to be seen. But with the largest Hadoop software distributor and support provider backing it, and with so many vendors and users of legacy tools seeking quick SQL access to Hadoop data, the deck is stacked in Impala's favor.