Big Data Tempest In A Teapot

15 Hot Skill Sets For IT Pros In 2015

If you followed last week's big data news, it's likely you came away dazed and confused about the state of Hadoop. And if you follow this week's headlines about Hortonworks' latest, $12.7 million quarter, you might wonder why there's such a fuss about big data.

The fact is, the big data market is still very small, and it's full of green products, discord, and factionalism. One camp, led by Pivotal and Hortonworks, last week announced the Open Data Platform, describing it as "a structured way for vendors to agree on a fully integrated and validated core distribution of Apache Hadoop." That group's claim is that the Hadoop community is fragmented. By rallying ODP members (including Hortonworks, Pivotal, IBM, SAS, and others) around ODP Core-sanctified components of Hadoop, the group hopes to focus investment and foster continuity and compatibility.

[ Want more on data analysis? Read Gartner BI Magic Quadrant 2015 Spots Market Turmoil. ]

A second camp emerged when both Cloudera and MapR declined to join ODP. Cloudera co-founder and chief strategy officer Mike Olson reasoned that the open-source Apache Foundation process has ensured a stable Hadoop trunk that every Hadoop distributor builds upon. "There’s simply no fundamental incompatibility among the core Hadoop components shipped by the various vendors," Olson says.

I have to agree with Curt Monash, who sizes up ODP as way for Hortonworks to "minimize the importance of any technical advantages Cloudera or MapR might have," and a "face-saving way" for IBM and Pivotal to let go of (or at least reduce the cost of maintaining) their Hadoop distributions. On the other hand, it's not necessarily a bad thing -- or a danger to the health of the Hadoop Community, as Olson suggests -- to see commercial vendors like Pivotal, IBM, and SAS adapting their software to run on core components of Hadoop that are available to all.

Cloudera, Hortonworks, and MapR clearly have to differentiate their offerings in order to compete and win business, but this holier-than-thou posturing and infighting is a distraction. Hortonworks' $46 million in annual revenue reported Tuesday equals that of a typical car dealership. At somewhere over $100 million in annual revenue, Cloudera isn't that much bigger.

InformationWeek's latest research, based on interviews with 374 information-management decision makers, shows that only about 4% of mainstream enterprises use Hadoop "extensively." Another 18% use it on "a limited basis," and 20% are "considering" the technology. That leaves 58% with "no current or planned use" of Hadoop.

If Hadoop is going to grow into the multi-billion dollar market that many envision, it's that 78% just starting to consider or not even looking at Hadoop that will make the difference. The likes of $86-billion-annual-revenue Microsoft and $38-billion-annual-revenue Oracle are well along in training their eyes on this market. Microsoft put an emphasis on big data analysis last week, introducing an Apache Storm service on its Azure HDInsight Cloud offering while also adding more algorithms and language options (including Python) to its Azure Machine Learning service.

Oracle is focusing on data analysis and movement, because that's where the money is -- not in merely storing data in a lake. Oracle took the wraps off its previously announced Oracle Big Data Discovery tool, and it introduced an adaptation of Oracle GoldenGate for big data streaming into HDFS, HBase, Hive, Storm, and Spark.

What Microsoft and Oracle are doing is reassuring the other 78% that they can help them if and when they're ready for big data analysis. When that day arrives, maybe that will or maybe that won't require Hadoop. Perhaps by then it will be commoditized into operating systems, as Forrester has warned.

If adoption is going to happen sooner, rather than later, what the Hadoop market needs is a constant drumbeat of user success stories. It needs use-case examples in every major industry sector. Companies large and small need to see that this is a technology they can manage.

The infighting is counterproductive and won't reassure anyone that Hadoop is ready for adoption. Tell me your technology is somehow purer and I'll yawn. Show me that a company much like mine is getting breakthrough, money-making results and I might be interested.

Attend Interop Las Vegas, the leading independent technology conference and expo series designed to inspire, inform, and connect the world's IT community. In 2015, look for all new programs, networking opportunities, and classes that will help you set your organization’s IT action plan. It happens April 27 to May 1. Register with Discount Code MPOIWK for $200 off Total Access & Conference Passes.