Software // Information Management
Commentary
10/29/2012
11:28 AM
Cindi Howson
Cindi Howson
Commentary
Connect Directly
Twitter
RSS
E-Mail
50%
50%

Big Data Meets BI: Beyond The Hype

With Hadoop quickly gaining adoption, Cloudera, Platfora, SiSense and others are introducing new options for gaining business intelligence from this big data platform.

Big Data was the big news in New York last week at the sold out Strata Conference. I was lured to Strata by both the traditional vendors I cover, such as SAP, SAS, and Tableau exhibiting there, as well as by big data analytics startups such as Datameer and Karmasphere.

There's a high degree of hype around big data, but there's also a high degree of innovation, tangible benefits, and venture capital backing. Coming from the perspective of the established business intelligence world, here's the skinny on where big data meets BI.

First, big data is more than Hadoop, the open source distributed file system capable of scaling to handle petabytes of data. Scalability is not the only appeal of Hadoop; it can also handle multi-structured data such as clickstreams, tweets, video, Facebook comments, sensor data, and so on. Such data is a challenge to model and store in a traditional relational database and data warehouse schema. When I first suggested to a fellow attendee that big data is more than Hadoop, he warned me such a comment might be considered blasphemous at Strata.

[ Want more of Cindi Howson's expert BI analysis? Read Oracle Exalytics: Is It A Must-Have for BI?. ]

In the traditional BI world, technologies such as analytic appliances, columnar databases, and in-memory engines can also handle big data. It all depends on whether the challenge is volume and performance, variety and complexity, or combinations thereof.

That brings me to the first big announcement at the conference: Cloudera Impala, a new real-time query engine for Hadoop. Cloudera is the leading provider of Hadoop software and support. It competes with Hortonworks and MapR, among others.

In the last year, many BI vendors have announced support for Hadoop. Access to data in Hadoop has been through Hive, a virtual data warehouse for Hadoop that has its own query language, HiveQL. The thing is, HiveQL generates MapReduce jobs to get to the data in Hadoop. MapReduce is batch-oriented and slow, in contrast to BI which is supposed to be fast. But hey, we might be talking petabytes here, so maybe a little patience is reasonable.

Impala, in contrast to Hive, is real-time. With Cloudera leading Impala (stealth) development for nearly two years, early adopters wondered if the technology would be kept proprietary to Cloudera. Instead, Cloudera says it will release Impala to the open source community. The first BI vendors to support Impala are MicroStrategy, Tableau, Pentaho, and QlikView. Pentaho cited a 10x query performance improvement s using Impala over Hive, but acknowledge it's still not as fast as what users are used to in the BI world.

So what's an impatient BI user to do? This is where the rest of the big data architecture comes into play. Querying vast volumes of granular data in Hadoop via Hive may be slow, so once an initial exploration is done, BI vendors will cache the results in their technology to ensure speed-of-thought analysis. This is where solutions like SAP Hana, MicroStrategy OLAP Services (for in-memory), SAS LASR server, the Tableau Data Engine, or the QlikView in-memory engine all come into play. Startup vendor SiSense was also at Strata, showcasing its combination columnar and in-memory engine, exploring 1 terabyte of data on a simple laptop with only 8 gigabytes of RAM.

With Hadoop gaining traction, there are a number of new data visualization and exploration vendors that offer access to and exploration of data in Hadoop. Isn't that what BI vendors with Hadoop connectors do? Yes, but those BI vendors can also access data in a data warehouse, analytic appliance, or spreadsheet. Hadoop may or may not be part of the picture.

With the new big-data-oriented vendors, such as Datameer, Karmasphere, and Platfora, Hadoop is definitely in the picture and may be the whole picture. Datameer, for example, generates its own MapReduce jobs (not relying on Hive) and provides 40+ connectors to sources such as Salesforce, Twitter, and Google Ads that allow you to load data into Hadoop. Initial data is explored through a spreadsheet-style interface and then cached in Datameer's in-memory engine. Datameer announced a new app market with 30 initial free apps. Karmasphere takes a different approach, focusing on data access and query generation through Hive. It partners with Tableau for visualization.

Platfora, currently in beta, has an appealing front end that reminds me of MicroStrategy's new Visual Insight, so it queries Hadoop through MapReduce jobs to create what it refers to as a lens. Lenses are in-memory caches of the data sets that users can then visually explore.

It was refreshing for me to see a mixture of data scientists and technologists at the Strata conference. Both camps seem to recognize that the explosion of data along with the shortage of talent in this space is creating a perfect storm in which only the smartest will survive.

Survival of the smartest has been prevalent theme in the recession. Some companies are still struggling to analyze sales and who's buying what. In the big data economy, the analysis extends to who's interested in your products, who's influencing buying decisions, and who's not even engaged, but should be!

Comment  | 
Print  | 
More Insights
Comments
Oldest First  |  Newest First  |  Threaded View
SFEW000
50%
50%
SFEW000,
User Rank: Apprentice
10/31/2012 | 2:34:51 PM
re: Big Data Meets BI: Beyond The Hype
Cindi,

You stated that Big Data is "more than Hadoop" but that's the closest you came to providing a definition. What is Big Data? In what sense is it different from data of the past? Why are features such as in-memory databases suddenly being called Big Data solutions? We've been querying "vast volumes of granular data" for many years, so what's changed to make this into something new called Big Data? We've been adding new sources of data constantly since the advent of computers, so what about today's new sources requires the name Big Data? How is what BI companies are doing today a qualitative departure from the past?

Stephen Few
HM
50%
50%
HM,
User Rank: Strategist
11/1/2012 | 1:43:07 PM
re: Big Data Meets BI: Beyond The Hype
Cindi, nice article. We are seeing an increase in businesses seeking specialized skills to help address challenges that arose with the era of big data. The HPCC Systems platform from LexisNexis helps to fill this gap by allowing data analysts themselves to own the complete data lifecycle. Designed by data scientists, ECL is a declarative programming language used to express data algorithms across the entire HPCC platform. Their built-in analytics libraries for Machine Learning and BI integration provide a complete integrated solution from data ingestion and data processing to data delivery. More at http://hpccsystems.com
RaviGarg
50%
50%
RaviGarg,
User Rank: Apprentice
11/2/2012 | 3:34:32 PM
re: Big Data Meets BI: Beyond The Hype
Stephen makes an interesting point; Industry has to constantly innovate and sell new stuff; .. and the latest is to capture unstructured data (from FB, Twitter and other social interaction) that can be correlated with products and services (big data analytics) and hopefully this correlation (interest) to drive the sales (yet to be done); I am hoping to see someone establishing this multichannel advertising spending model that can help me maximize the actual sales. Everything else is fun and games!
The Agile Archive
The Agile Archive
When it comes to managing data, donít look at backup and archiving systems as burdens and cost centers. A well-designed archive can enhance data protection and restores, ease search and e-discovery efforts, and save money by intelligently moving data from expensive primary storage systems.
Register for InformationWeek Newsletters
White Papers
Current Issue
InformationWeek Government Tech Digest Oct. 27, 2014
To meet obligations -- and avoid accusations of cover-up and incompetence -- federal agencies must get serious about digitizing records.
Video
Slideshows
Twitter Feed
InformationWeek Radio
Archived InformationWeek Radio
A roundup of the top stories and community news at InformationWeek.com.
Sponsored Live Streaming Video
Everything You've Been Told About Mobility Is Wrong
Attend this video symposium with Sean Wisdom, Global Director of Mobility Solutions, and learn about how you can harness powerful new products to mobilize your business potential.