How To Beat The Big Data Disconnect

Your organization has its shiny new Hadoop platform in place -- great. You've spent a sizable chuck of your budget on big data appliances, and your IT folks have spent countless hours figuring out how everything works.

Now what? Many organizations at this point "hit a bit of a roadblock," according to David Smith, vice president of marketing and community for Revolution Analytics, a software company and one of the guiding forces behind the open source R programming language. Forbes recently selected Smith as one of the top 20 influencers in the big data space.

In a phone interview with InformationWeek, Smith noted it's not uncommon for organizations to feel a twinge (or jolt) of disillusionment after jumping aboard the big data bandwagon, particularly if they're not exactly sure what they're doing.

"If you want to make use of all that data, you need some pretty serious expertise, especially with Hadoop systems, where a lot of the data is unstructured," said Smith. "It does take a lot of skill to tease out information from that unstructured data."

[ What's your perception of big data? Read Big Data Perceptions: Good, Bad And Ugly. ]

Not surprisingly, Smith touted the benefits of the R language, which is used by data scientists to develop predictive models. "It's taken off in academia over the last 10 years or so," he said. "And now we've made it scale to big data and run very quickly."

For programmers working with big data, R has two primary advantages, according to Smith. "It's designed to work with data and build models with data," he said. "[Programmers] can go from a concept to a working model in a fraction of the time it takes with legacy systems."

The second advantage is R's open source design. "You've got an entire community of statisticians and data scientists [who] are really pushing the envelope on data access, data platforms like Hadoop, data analysis techniques and also data visualization, which is an increasingly important part of the story," said Smith.

A potential problem with big data is that organizations run the risk of becoming big data hoarders, storing far more bits than they know what to do with.

However, storing petabytes of unstructured data isn't a bad thing as long as you have the tools and strategy to extract insights. The arrival of Hadoop a few years ago greatly helped organizations with this process, Smith noted. "Suddenly we had an inexpensive system, built on commodity hardware and open source technologies, which scales to the volumes of data these organizations were seeing, and they realized there's immense value in that data."

Chat logs, for instance, are an example of unstructured data that big data platforms have made relatively easy to store, process and analyze. A company can now quickly explore historical data to gauge whether its customers are pleased with its products or services. "[That's done] not by reading each chat log manually, but by using automated systems that can scan them and assign sentiment scores to them," Smith explained. "And suddenly you've got real, concrete, quantifiable information that was previously just locked up in messy text data."

Of course, data is a valuable resource -- if managed properly. "You can think about it in the same way that oil companies think about crude oil," Smith noted. "You dig this stuff out of the ground. It's dirty; it's got contaminants in it. It's not yet ready for putting into a vehicle. But with a distillation process, you get something that's very valuable."

Smith believes real-time applications are becoming increasingly important in the big data space as well. Several of Revolution Analytics' customers in online advertising, for instance, are making great use of real-time apps. "They have models that run a thousand times a second to choose which ad appears on a website, as you're visiting that website," Smith said.

InformationWeek is conducting a survey on the state of database technology in the enterprise. Take our InformationWeek 2013 Database Technology Survey now. Survey ends Feb. 15.