Hortonworks Cofounder: Hadoop Rivals Can't Touch It

Hadoop is the best and most economical option for extracting value from massive data sets, according to Arun Murthy, Apache Hadoop project management committee chair. In addition, the up-and-coming data management platform has quickly become the "de facto standard" for big data, he claimed.

Given Murthy's background, it's not surprising he's a bit biased in favor of Hadoop. One of the cofounders of Hortonworks, an enterprise software company focused on the development and support of Hadoop, Murthy has been a major participant in the open source project since its beginning in 2006.

In a phone interview with InformationWeek, Murthy expressed confidence that Hadoop is in a strong position to become the dominant platform for big data management.

"The exciting part of Hadoop--and the way things have gone the last 12 months or so--is that a lot of really big and important enterprise software companies are starting to work with it," said Murthy.

[ Learn about new Hadoop features. See Hadoop Gets Elastic Like Amazon's Cloud. ]

Take Microsoft, for instance. At last week's Strata Conference in New York, Microsoft announced the preview release of HDInsight Server for Windows, the first beta version of the company's Hadoop software distribution.

HDInsight Server, in addition to core Hadoop components such as HDFS (Hadoop Distributed File System) and MapReduce, includes Hortonworks' HCatalog table management service for accessing Hadoop data.

Available on Windows Server or as a Windows Azure service, HDInsight connects to popular business intelligence (BI) tools, including Microsoft Excel, PowerPivot for Excel and Power View.

"Hadoop has to work with all the tools that people have already invested in," Murthy said. "People have invested in Excel, for example, and you've got to make sure that Hadoop works with it."

While Hadoop is gaining acceptance in the enterprise, the platform is still in "its early days," Murthy said. "A lot of people are beginning to understand not just the value of Hadoop, but also the value of data," he added.

And, of course, the sheer volume of digital information is growing dramatically. "Since the dawn of humanity until early 2000 or so, 5 exabytes of data had been produced," said Murthy. "And since then, we're producing about 5 exabytes of data every couple of years. That's incredible. Every one of us walks around with cellphones and other devices, and all that data we generate is pretty amazing."

Hadoop has open source competitors in the big data space, of course, including HPCC (High Performance Computing Cluster) from LexisNexis Risk Solutions. But Murthy believes Hadoop's challengers will have a tough time gaining adherents.

"It's going to be harder to build a competitor to Hadoop because there's such a big ecosystem and mindshare around it," Murthy said.

While Hadoop has many attributes for big data management, enterprises should proceed carefully before implementing it, cautioned Forrester Research analyst Boris Evelson. In a September 2012 blog post, "What Do BI Vendors Mean When They Say They Integrate With Hadoop," Evelson wrote:

"Hadoop certainly plays a key role in the big data revolution, so all business intelligence (BI) vendors are jumping on the bandwagon and saying that they integrate with Hadoop. But what does that really mean? First of all, Hadoop is not a single entity; it's a conglomeration of multiple projects, each addressing a certain niche within the Hadoop ecosystem, such as data access, data integration, DBMS, system management, reporting, analytics, data exploration and much, much more."

Before taking the Hadoop plunge, enterprises should grill their business intelligence vendors to see how well they really integrate with the big data platform, Everson warned.

At this hands-on Wall Street & Technology Virtual Event, Big Data On Wall Street, experts and solution providers will offer detailed insight into how risk management, financial reporting, trading analytics and financial modeling, along with a host of other opportunities, can all benefit from applying big data techniques and technologies to business processes. When you register, you will gain access to live and on-demand webcast presentations, as well as virtual booths packed with free resources. It happens Nov. 1.