With Hadoop, Big Data Analytics Challenges Old-School Business Intelligence
Datameer and Karmasphere say their Hadoop-based platforms are what's needed for the next era of data analysis. Are you buying it?
12 Hadoop Vendors To Watch In 2012
(click image for larger view and for slideshow)
Datameer and Karmasphere, two competing upstart vendors offering reporting, data-visualization, and data-analysis capabilities on top of Hadoop, released new versions of their software on Monday. Both talked up the need for next-generation tools.
It's not that old-school business intelligence software tools are going away, these upstarts grant. But both portray batch-oriented extract-transform-load (ETL) data integration, relational data warehousing, and old-school analytics as too slow, rigid, and expensive to keep up in the big-data era.
Hadoop is the future, these vendor's contend, because it's a massively scalable data-management and analysis environment that can handle variably structure data from many sources--log files, clickstreams, sensor data, social media sources and so on--without the delays inherent in dealing with the static schemas of relational databases.
If companies want to look at recent point-of-sale transactions alongside Web site clickstreams, recent online enrollments, email campaign results, and social media chatter, for example, it would be difficult if not impossible to quickly put all that data into a relational data warehouse and look for correlations.
Datameer and Karmasphere offer integration, data-analysis, and data visualization products that run on Hadoop. They're two of the better-established companies hoping to provide what they describe as next-generation platforms. Founded in 2009 and 2005, respectively, Datameer and Karmasphere are private companies that each have a handful of nameable customers. Datameer points to blue chip Visa, among others, while Karmasphere says Intel and Microsoft are customers.
In his company's work with a global bank on meeting Basel II capital requirements, Datameer CEO Stefan Groschupf reports that more than 250 data sources are required.
"Finding a perfect star schema and building the required ETL for that is impossible," Groschupf told InformationWeek.
In another example, a big retailer wanted to get data that helped it better understand all its interactions with customers, rather than looking at isolated transactions.
"ETL and data warehousing and BI are just fine for the problem of looking at transactions here and there, but there's no chance of bringing it all together to look at the interactions across all of the islands of information," Groschupf says.
Hadoop also scales in linear fashion to solve the data-volume challenge, and it's built on commodity hardware, so it's less expensive, terabyte for terabyte, than conventional relational systems, Datameer and Karmasphere contend.
What about the scarcity of Hadoop talent, the platform's own batch-related delays and the alternative of using "old-school" BI systems to analyse data either inside or moved out of Hadoop? Datameer and Karmasphere have responses to some of these questions, but more on that later.
Datameer's platform for analytics on Hadoop provides modules for data-integration (including connectors to mainframes, databases, social sources like Facebook and Twitter, and more), a spreadsheet-driven data-analysis environment, and a dashboarding and data-visualization environment, the last part being new in version 2.0. The upgrade also introduces two lower-priced editions designed for fewer users and lower data volumes than the enterprise version.
Datameer's key appeal is its spreadsheet-driven data-analysis environment, which the vendor says provides more than 200 analytic functions from simple joins to predictive analytics. Most importantly, it's said to eliminate the need for Hadoop wonks and IT people to support data analysis. The interface is designed to let business analysts--and certainly any SQL-savvy BI or data-warehousing power user--use a point-and-click interface to analyze data within Hadoop. In the background, Datameer's software turns the selections into Hadoop-based MapReduce and data-manipulation jobs without users having to write code.
Datameer Enterprise starts at $100,000 and supports Hadoop deployments with hundreds or even thousands of nodes. Datameer Workgroup is subscription based software that creates a small-scale Hadoop deployment on a single server. It starts at $3,000 per year for up to 10 users, but data volumes are limited to one terabyte. Exceed that ballpark and you'll have to upgrade to the enterprise edition, though the reports and analyses are portable.
How Enterprises Are Attacking the IT Security EnterpriseTo learn more about what organizations are doing to tackle attacks and threats we surveyed a group of 300 IT and infosec professionals to find out what their biggest IT security challenges are and what they're doing to defend against today's threats. Download the report to see what they're saying.
IT Strategies to Conquer the CloudChances are your organization is adopting cloud computing in one way or another -- or in multiple ways. Understanding the skills you need and how cloud affects IT operations and networking will help you adapt.