Big Data // Big Data Analytics
News
2/11/2013
10:31 AM
Connect Directly
Google+
RSS
E-Mail
50%
50%

How To Beat The Big Data Disconnect

Organizations often hit a roadblock after the initial setup of a big data project. But to get value from unstructured data, keep on driving.

 Big Data Analytics Masters Degrees: 20 Top Programs
Big Data Analytics Masters Degrees: 20 Top Programs
(click image for larger view and for slideshow)
Your organization has its shiny new Hadoop platform in place -- great. You've spent a sizable chuck of your budget on big data appliances, and your IT folks have spent countless hours figuring out how everything works.

Now what? Many organizations at this point "hit a bit of a roadblock," according to David Smith, vice president of marketing and community for Revolution Analytics, a software company and one of the guiding forces behind the open source R programming language. Forbes recently selected Smith as one of the top 20 influencers in the big data space.

In a phone interview with InformationWeek, Smith noted it's not uncommon for organizations to feel a twinge (or jolt) of disillusionment after jumping aboard the big data bandwagon, particularly if they're not exactly sure what they're doing.

"If you want to make use of all that data, you need some pretty serious expertise, especially with Hadoop systems, where a lot of the data is unstructured," said Smith. "It does take a lot of skill to tease out information from that unstructured data."

[ What's your perception of big data? Read Big Data Perceptions: Good, Bad And Ugly. ]

Not surprisingly, Smith touted the benefits of the R language, which is used by data scientists to develop predictive models. "It's taken off in academia over the last 10 years or so," he said. "And now we've made it scale to big data and run very quickly."

For programmers working with big data, R has two primary advantages, according to Smith. "It's designed to work with data and build models with data," he said. "[Programmers] can go from a concept to a working model in a fraction of the time it takes with legacy systems."

The second advantage is R's open source design. "You've got an entire community of statisticians and data scientists [who] are really pushing the envelope on data access, data platforms like Hadoop, data analysis techniques and also data visualization, which is an increasingly important part of the story," said Smith.

A potential problem with big data is that organizations run the risk of becoming big data hoarders, storing far more bits than they know what to do with.

However, storing petabytes of unstructured data isn't a bad thing as long as you have the tools and strategy to extract insights. The arrival of Hadoop a few years ago greatly helped organizations with this process, Smith noted. "Suddenly we had an inexpensive system, built on commodity hardware and open source technologies, which scales to the volumes of data these organizations were seeing, and they realized there's immense value in that data."

Chat logs, for instance, are an example of unstructured data that big data platforms have made relatively easy to store, process and analyze. A company can now quickly explore historical data to gauge whether its customers are pleased with its products or services. "[That's done] not by reading each chat log manually, but by using automated systems that can scan them and assign sentiment scores to them," Smith explained. "And suddenly you've got real, concrete, quantifiable information that was previously just locked up in messy text data."

Of course, data is a valuable resource -- if managed properly. "You can think about it in the same way that oil companies think about crude oil," Smith noted. "You dig this stuff out of the ground. It's dirty; it's got contaminants in it. It's not yet ready for putting into a vehicle. But with a distillation process, you get something that's very valuable."

Smith believes real-time applications are becoming increasingly important in the big data space as well. Several of Revolution Analytics' customers in online advertising, for instance, are making great use of real-time apps. "They have models that run a thousand times a second to choose which ad appears on a website, as you're visiting that website," Smith said.

InformationWeek is conducting a survey on the state of database technology in the enterprise. Take our InformationWeek 2013 Database Technology Survey now. Survey ends Feb. 15.

Comment  | 
Print  | 
More Insights
6 Tools to Protect Big Data
6 Tools to Protect Big Data
Most IT teams have their conventional databases covered in terms of security and business continuity. But as we enter the era of big data, Hadoop, and NoSQL, protection schemes need to evolve. In fact, big data could drive the next big security strategy shift.
Register for InformationWeek Newsletters
White Papers
Current Issue
InformationWeek Tech Digest September 18, 2014
Enterprise social network success starts and ends with integration. Here's how to finally make collaboration click.
Flash Poll
Video
Slideshows
Twitter Feed
InformationWeek Radio
Archived InformationWeek Radio
The weekly wrap-up of the top stories from InformationWeek.com this week.
Sponsored Live Streaming Video
Everything You've Been Told About Mobility Is Wrong
Attend this video symposium with Sean Wisdom, Global Director of Mobility Solutions, and learn about how you can harness powerful new products to mobilize your business potential.