Big Data 101: New Vendor-Neutral Guide

Big Data's Surprising Uses: From Lady Gaga To CIA

(click image for larger view and for slideshow)

What exactly is big data? Explanations vary, of course, but Gartner's popular 3V model -- essentially that big data is high-volume, high-velocity, and high-variety information that requires new tools to manage -- is the most commonly quoted definition. Still, a haze of ambiguity surrounds big data, an issue that the Open Data Center Alliance (ODCA) hopes to resolve with its new "Big Data Consumer Guide," a handbook for enterprises that want to know what big data is, why it matters, and how to gain from it.

Created in 2010, the ODCA is an industry consortium of global IT organizations with a primary goal of developing open standards for cloud computing. In a phone interview with InformationWeek, ODCA executives said that the alliance's "Big Data Consumer Guide" is a logical extension of its cloud computing efforts. "A lot of the cloud paradigms out there today are something that the big data environment is going to want to take advantage of," said ODCA lead technical advisor John Pereira.

[ Big data has value that's often not reflected in the books. Read What's Your Big Data Worth? ]

The essential nature of big data -- particularly the fact that data volume can grow significantly in a short period of time with little notice -- fits nicely with the cloud environment, Pereira noted.

"Because of the nature of big data, you're probably going to want to think of something in a more distributed environment. A cloud paradigm helps you move in that direction," he added.

The consumer guide summarizes how big data platforms can help a variety of industries. Banks, for instance, can correlate data from multiple, unrelated sources to potentially spot credit card fraud. In addition, the guide provides common definitions and lingo that organizations can use when working with big data providers.

The guide also references an astonishing statistic from IDC: Unstructured data comprises more than 90% of the information in today's enterprises, much of it stored away in documents, email, notes and Web content.

Unstructured information that falls under the "big data" umbrella includes machine-generated data from sensors, machine logs and cellphone GPS signals, as well as data from social media sites and online transactions.

The Consumer Guide refers to Apache Hadoop as a "leading Big Data technology," but points out that a variety of other open-source big data projects are available, including Riak, MongoDB, CouchDB, Redis, Hypertable, Storm, Spark and High-Performance Computing Cluster (HPCC).

"We really try to bring a relatively vendor-agnostic approach to our recommendations and direction," said Pereira. "We try to avoid beating the drum for a specific vendor."

Enterprises need to carefully plan their big data strategies in advance to avoid poor practices that waste both resources and money, the ODCA says.

"You want to be able to write the data in a way that is most efficient -- not replicate the same data set over and over. How you record the information up front is very important," said ODCA executive director Marvin Wheeler. "It's all about how the data's being written so that it doesn't sprawl as much as (with) traditional methods." Dealing with data sprawl is already a critical issue for enterprises. According to the McKinsey Global Institute, 15 of 17 U.S. business sectors have more data stored per company than the U.S. Library of Congress. And some researchers estimate that 90% of all data has been generated in the last two years alone.

The growing use of video analytics is one example.

"If you go back five years ago, who would have thought that the idea of keeping video -- and being able to do analytics on it to make potentially better business and shopping decisions -- was a topic that would become foremost in everybody's mind," said Pereira. "It does get to the heart of what big data is about, and it's one of the things that makes big data very interesting as a new technology and paradigm."

Emerging database technology promises to automate more analysis. Here's where it could replace relational systems. Also in the new, all-digital The Rise Of Semantic Databases special issue of InformationWeek: There's a big demand for big data and analytics experts. (Free registration required.)