With Hadoop, Big Data Analytics Challenges Old-School Business Intelligence

Datameer and Karmasphere, two competing upstart vendors offering reporting, data-visualization, and data-analysis capabilities on top of Hadoop, released new versions of their software on Monday. Both talked up the need for next-generation tools.

It's not that old-school business intelligence software tools are going away, these upstarts grant. But both portray batch-oriented extract-transform-load (ETL) data integration, relational data warehousing, and old-school analytics as too slow, rigid, and expensive to keep up in the big-data era.

Hadoop is the future, these vendor's contend, because it's a massively scalable data-management and analysis environment that can handle variably structure data from many sources--log files, clickstreams, sensor data, social media sources and so on--without the delays inherent in dealing with the static schemas of relational databases.

If companies want to look at recent point-of-sale transactions alongside Web site clickstreams, recent online enrollments, email campaign results, and social media chatter, for example, it would be difficult if not impossible to quickly put all that data into a relational data warehouse and look for correlations.

[ Want more on Apache's big-data platform? Read Why Hadoop Crowd Is Hearing Much About Hortonworks. ]

Datameer and Karmasphere offer integration, data-analysis, and data visualization products that run on Hadoop. They're two of the better-established companies hoping to provide what they describe as next-generation platforms. Founded in 2009 and 2005, respectively, Datameer and Karmasphere are private companies that each have a handful of nameable customers. Datameer points to blue chip Visa, among others, while Karmasphere says Intel and Microsoft are customers.

In his company's work with a global bank on meeting Basel II capital requirements, Datameer CEO Stefan Groschupf reports that more than 250 data sources are required.

"Finding a perfect star schema and building the required ETL for that is impossible," Groschupf told InformationWeek.

In another example, a big retailer wanted to get data that helped it better understand all its interactions with customers, rather than looking at isolated transactions.

"ETL and data warehousing and BI are just fine for the problem of looking at transactions here and there, but there's no chance of bringing it all together to look at the interactions across all of the islands of information," Groschupf says.

Hadoop also scales in linear fashion to solve the data-volume challenge, and it's built on commodity hardware, so it's less expensive, terabyte for terabyte, than conventional relational systems, Datameer and Karmasphere contend.

What about the scarcity of Hadoop talent, the platform's own batch-related delays and the alternative of using "old-school" BI systems to analyse data either inside or moved out of Hadoop? Datameer and Karmasphere have responses to some of these questions, but more on that later.

What's Different?

Datameer's platform for analytics on Hadoop provides modules for data-integration (including connectors to mainframes, databases, social sources like Facebook and Twitter, and more), a spreadsheet-driven data-analysis environment, and a dashboarding and data-visualization environment, the last part being new in version 2.0. The upgrade also introduces two lower-priced editions designed for fewer users and lower data volumes than the enterprise version.

Datameer's key appeal is its spreadsheet-driven data-analysis environment, which the vendor says provides more than 200 analytic functions from simple joins to predictive analytics. Most importantly, it's said to eliminate the need for Hadoop wonks and IT people to support data analysis. The interface is designed to let business analysts--and certainly any SQL-savvy BI or data-warehousing power user--use a point-and-click interface to analyze data within Hadoop. In the background, Datameer's software turns the selections into Hadoop-based MapReduce and data-manipulation jobs without users having to write code.

Datameer Enterprise starts at $100,000 and supports Hadoop deployments with hundreds or even thousands of nodes. Datameer Workgroup is subscription based software that creates a small-scale Hadoop deployment on a single server. It starts at $3,000 per year for up to 10 users, but data volumes are limited to one terabyte. Exceed that ballpark and you'll have to upgrade to the enterprise edition, though the reports and analyses are portable.