Hortonworks' Hadoop Dilemma: Get Rich Giving Ideas Away

How do you stand out as an innovator when you share everything you do with an open source community? That's the challenge for Hortonworks as it marks its first year as an independent company and its first 90 days distributing an entirely open source distribution of Apache Hadoop software.

Spun out of Yahoo last year, Hortonworks built its company around nearly 50 of the Apache Hadoop community's earliest and most prolific contributors who joined Hortonworks from Yahoo. Hadoop was practically born at Yahoo in the middle of the last decade, and the search and Web portal company remains one of Hadoop's biggest users (and Hortonworks' largest customer).

Hortonworks has had a busy year, releasing its first software distribution, Hortonworks Data Platform (HDP) 1.0, at the company's successful Hadoop Summit in June--a San Jose, Calif., event that had more than 2,200 attendees. The company now has more than 120 employees and 70 partners (most notably Microsoft and Teradata).

Last week Hortonworks released HDP 1.1, an incremental upgrade with enhancements including high-availability support by way of Red Hat Enterprise Linux, real-time data capture based on Apache Flume, better APIs for systems management and monitoring, and improved processing speed.

[ Want more on Hortonworks? Read Hortonworks Releases Its (Conservative) Hadoop Platform. ]

Hadoop is in big demand for its combination of massive scalability, ability to handle highly variable and complex data without time-consuming data modeling, and its low infrastructure cost per terabyte when used for high volumes of data exceeding tens or hundreds of terabytes.

Like Cloudera and MapR, Hortonworks offers a Hadoop software distribution as well as consulting, training, and ongoing support. But unlike its rivals, Hortonworks' distribution is entirely made up of open source Apache Hadoop software. Cloudera, which has a four-year head start in the market and claims hundreds of paying customers (it won't say exactly how many), also licenses proprietary Cloudera Enterprise 4.0 software to handle Hadoop deployment, administration, and management tasks.

Rival MapR touts its M5 Hadoop distribution as the performance leader, replacing what the vendor views as flawed components of the open source platform like the Hadoop Distributed File Systems (HDFS) with proprietary components to ensure high-availability and deliver higher throughput and capacity than standard Hadoop clusters.

By taking the conservative approach of shipping only generally available Apache open source software and contributing all of its work back to the Hadoop community, Hortonworks lets competitors take advantage of its best work. In the case of HCatalog, for example, the table management service for unified access to Hadoop data was "99% done by Hortonworks employees," Hortonworks CEO Rob Bearden told InformationWeek.

HCatalog is important because it will enable database management systems and business intelligence tools to access Pig, Hive, and MapReduce data without having to move it, which is obviously difficult at high scale. No surprise, then, that "other vendors, including MapR, have embraced it," Bearden said.

Open source work by any contributor is, by definition, shared with the entire community, but Bearden said Hortonworks "holds nothing back." So how, then, will Hortonworks stand apart if it gives away what other vendors might reserve for proprietary components?

"We're very focused on making sure the right enterprise functionality gets into the core platform and that we are the more reliable, stable platform on the planet," Bearden said.

The "reliable, stable" spin is a not-so-veiled slam against Cloudera, which is shipping Hadoop 2.0 software as part of its CDH4 distribution. Hortonworks contends that software is alpha-stage componentry that isn't ready for prime time. "Our view is that [software] needs to bake a bit more before we unleash it on the next wave of enterprise adopters," Shaun Connolly, Hortonwork's VP of corporate strategy, told InformationWeek in June.

Cloudera apparently felt that the Hadoop's controlling namenode was a high-profile single point of failure that had to be addressed, so better to deliver an early version of Apache's high-availability software than to rely on options from third-party vendors. Hortonworks now has high-availability options through both VMWare and Red Hat, but some would-be customers might not be using software from either one of those vendors.

For those who don't care to pioneer new Hadoop features, Hortonworks' conservative approach is reassuring. And where customers of Microsoft and Teradata in particular are concerned, going with Hortonworks will be an obvious choice because those partners are connecting their software with HDP. Teradata will announce a Hadoop appliance in October that will feature HDP, according to Bearden, and Hortonworks has developed the software needed to run Hadoop on Windows.

"HDP is Microsoft's reference platform, and they're putting a lot of resources to make sure that HDP runs extraordinarily well not only in a Windows environment with deep integration to Active Directory, but also on the Azure cloud platform," Bearden says.

Don't feel too sorry for Hortonworks: the revenue it foregoes licensing software to end users it likely more than makes up collecting fees from partners like Microsoft to do development work such as making Hadoop clusters run on Windows. As for the end-user community, Hortonworks is doing proof-of-concept projects with "hundreds of companies" and its software is seeing steady download demand, Bearden said. Will Hortonworks reap all the benefits of its hard work on Hadoop?

"We're helping the Hadoop market to become very big, and ultimately, the market will return to its rightful owner," Bearden said. "The ecosystem recognizes who's innovating, and now that we have technology out there, we're seen the migration very rapidly."