Big Data Progress Requires Better Development Tools

16 Top Big Data Analytics Platforms

"Big data for the masses" is a popular rallying cry among people who see the complexity of Hadoop and related tools as an impediment to a data-driven nirvana. A recent Wikibon survey of big data vendors and service providers shows that companies' technical staff often lack the training to manage the notoriously complex Hadoop effectively.

But rather than training more Hadoop experts, enterprises should focus on getting better tools to help their developers build big data apps, says Jonathan Gray, CEO and cofounder of Continuuity, a two-year-old startup with an app development platform designed specifically for Hadoop. The Palo Alto, Calif., company has 30 employees, two-thirds of whom are engineers.

"We're a Hadoop-focused company. What makes us different from everyone else is that we're a developer-focused company," Gray told InformationWeek in a phone interview. "We've built middleware, or an application server, on top of Hadoop."

Before cofounding Continuuity with Nitin Motgi (formerly of Yahoo) in late 2011, Gray was a software engineer at Facebook, where he was an early adopter of Hadoop who steered the company's HBase engineering efforts. The pair's mission was to make it easier for developers to build and manage Hadoop apps.

[What does it take to land a Hadoop gig? Read Hadoop Jobs: 6 Recruiter Tips.]

As Gray sees it, a lot of businesses today -- most notably, telecom, financial services, retail, and insurance companies -- are swimming in data, but they're not sure how to profit from it. These firms are Hadoop's early adopters and are the ones that could benefit most from big data apps. "That's where we fit in. Enabling some bank with 10,000 developers -- but only three Hadoop developers -- to get those 10,000 people to build on Hadoop."

This legion of developers doesn't need vast expertise in Hadoop, nor do they need significant retraining to become full-fledged data scientists. Rather, it needs a powerful toolkit for developing big data apps -- a market that Continuuity hopes to exploit. "We separate the data scientist from the developer, and those are generally two different people. There's been a little bit too much emphasis on the science part of this equation and not enough on the developer side of it."

Emerging hardware categories such as wearables, connected cars, and eventually smart homes show great promise for data-driven apps. Telcos are in a particularly strong position to profit from the massive volumes of user data they amass, Gray said. "They have location data, demographic data, web browsing history. They have tons and tons of data. And the interesting thing is, all these companies are sitting on all this data and essentially saying, 'How do I turn this pile of bits into a pile of dollars?'"

The big data space is crowded, of course, with a lot of venture capital pouring into Hadoop companies. "I'd say we are differentiated in one particular way: Our emphasis on the developer and on what we call big data applications," Gray said. "If you look at what most of the traditional vendors are offering, and if you also look at what the Cloudera and Hortonworks of the world are offering, they're selling a data warehouse. They're selling a BI tool for analytics."

BI tools are valuable, of course, but there's less focus these days on products and services built on top of data. "It's really all about applications, and most of the market isn't focused on that. That presents a challenge because we're ahead of the market a little bit."

Still, the focus on big data apps appears promising. "We're looking past the data scientist at the next person in line, and that's the developer," Gray said. "And there's a huge problem there. There are millions of developers out there, but they don't have the right skills, and the infrastructure [of Hadoop] is too hard for them."

Engage with Oracle president Mark Hurd, NFL CIO Michelle McKenna-Doyle, General Motors CIO Randy Mott, Box founder Aaron Levie, UPMC CIO Dan Drawbaugh, GE Power CIO Jim Fowler, and other leaders of the Digital Business movement at the InformationWeek Conference and Elite 100 Awards Ceremony, to be held in conjunction with Interop in Las Vegas, March 31 to April 1, 2014. See the full agenda here.