Big Data Meets BI: Beyond The Hype

Big Data was the big news in New York last week at the sold out Strata Conference. I was lured to Strata by both the traditional vendors I cover, such as SAP, SAS, and Tableau exhibiting there, as well as by big data analytics startups such as Datameer and Karmasphere.

There's a high degree of hype around big data, but there's also a high degree of innovation, tangible benefits, and venture capital backing. Coming from the perspective of the established business intelligence world, here's the skinny on where big data meets BI.

First, big data is more than Hadoop, the open source distributed file system capable of scaling to handle petabytes of data. Scalability is not the only appeal of Hadoop; it can also handle multi-structured data such as clickstreams, tweets, video, Facebook comments, sensor data, and so on. Such data is a challenge to model and store in a traditional relational database and data warehouse schema. When I first suggested to a fellow attendee that big data is more than Hadoop, he warned me such a comment might be considered blasphemous at Strata.

[ Want more of Cindi Howson's expert BI analysis? Read Oracle Exalytics: Is It A Must-Have for BI?. ]

In the traditional BI world, technologies such as analytic appliances, columnar databases, and in-memory engines can also handle big data. It all depends on whether the challenge is volume and performance, variety and complexity, or combinations thereof.

That brings me to the first big announcement at the conference: Cloudera Impala, a new real-time query engine for Hadoop. Cloudera is the leading provider of Hadoop software and support. It competes with Hortonworks and MapR, among others.

In the last year, many BI vendors have announced support for Hadoop. Access to data in Hadoop has been through Hive, a virtual data warehouse for Hadoop that has its own query language, HiveQL. The thing is, HiveQL generates MapReduce jobs to get to the data in Hadoop. MapReduce is batch-oriented and slow, in contrast to BI which is supposed to be fast. But hey, we might be talking petabytes here, so maybe a little patience is reasonable.

Impala, in contrast to Hive, is real-time. With Cloudera leading Impala (stealth) development for nearly two years, early adopters wondered if the technology would be kept proprietary to Cloudera. Instead, Cloudera says it will release Impala to the open source community. The first BI vendors to support Impala are MicroStrategy, Tableau, Pentaho, and QlikView. Pentaho cited a 10x query performance improvement s using Impala over Hive, but acknowledge it's still not as fast as what users are used to in the BI world.

So what's an impatient BI user to do? This is where the rest of the big data architecture comes into play. Querying vast volumes of granular data in Hadoop via Hive may be slow, so once an initial exploration is done, BI vendors will cache the results in their technology to ensure speed-of-thought analysis. This is where solutions like SAP Hana, MicroStrategy OLAP Services (for in-memory), SAS LASR server, the Tableau Data Engine, or the QlikView in-memory engine all come into play. Startup vendor SiSense was also at Strata, showcasing its combination columnar and in-memory engine, exploring 1 terabyte of data on a simple laptop with only 8 gigabytes of RAM.

With Hadoop gaining traction, there are a number of new data visualization and exploration vendors that offer access to and exploration of data in Hadoop. Isn't that what BI vendors with Hadoop connectors do? Yes, but those BI vendors can also access data in a data warehouse, analytic appliance, or spreadsheet. Hadoop may or may not be part of the picture.

With the new big-data-oriented vendors, such as Datameer, Karmasphere, and Platfora, Hadoop is definitely in the picture and may be the whole picture. Datameer, for example, generates its own MapReduce jobs (not relying on Hive) and provides 40+ connectors to sources such as Salesforce, Twitter, and Google Ads that allow you to load data into Hadoop. Initial data is explored through a spreadsheet-style interface and then cached in Datameer's in-memory engine. Datameer announced a new app market with 30 initial free apps. Karmasphere takes a different approach, focusing on data access and query generation through Hive. It partners with Tableau for visualization.

Platfora, currently in beta, has an appealing front end that reminds me of MicroStrategy's new Visual Insight, so it queries Hadoop through MapReduce jobs to create what it refers to as a lens. Lenses are in-memory caches of the data sets that users can then visually explore.

It was refreshing for me to see a mixture of data scientists and technologists at the Strata conference. Both camps seem to recognize that the explosion of data along with the shortage of talent in this space is creating a perfect storm in which only the smartest will survive.

Survival of the smartest has been prevalent theme in the recession. Some companies are still struggling to analyze sales and who's buying what. In the big data economy, the analysis extends to who's interested in your products, who's influencing buying decisions, and who's not even engaged, but should be!