Is Your Data Big Enough For Big Data?

Big Data's Surprising Uses: From Lady Gaga To CIA

(click image for larger view and for slideshow)

If you're familiar with Gartner's 3V definition of big data -- high volume, high velocity and high variety -- you're probably aware that managing massive data sets doesn't necessarily require a Hadoop-style solution.

If not, you've got some studying to do.

According to Jim Gallo, national director of business analytics for ICC, an IT services provider based in Columbus, Ohio, volume isn't everything. For organizations to accurately determine whether a big data platform is right for them, they must also study the variety and velocity of their data.

"If I have large volumes of data that are transactional or structured in nature, that's a use case for traditional data warehousing," Gallo said in a phone interview with InformationWeek.

[ Want more intel on whether big data is right for you? See Big Data: Hype Vs. Hope At Interop. ]

However, varied information that's non-structured by nature, such as text files or social media streams from Twitter, poses a new challenge. Namely, it's information that traditional databases typically can't manage.

Velocity is another issue. The ability to examine data in real time is something that traditional business intelligence (BI) solutions "can't do very well, if at all," said Gallo.

And that's unfortunate, because real-time data may hold a lot of valuable business insights.

"Analyzing data in motion, and making inferences on extremely large amounts of data to predict what may be happening at a point in time, or very soon afterward ... can add a lot of value to a business," Gallo added.

And then there's the third V in big data: Volume. Again, an organization must study its needs carefully to decide whether to try a big data solution. For instance, do you deal with unstructured data? Structured? Both?

"The question is, if it's purely structured data, am I better served by traditional BI platforms, or do I leverage a big data platform?" Gallo asked rhetorically.

Structured information, even very large data sets such as credit card transactions, is generally better served by a traditional BI system, he explained.

Of course, for processing unstructured information, big data is the way to go. Hadoop, the biggest player in this market, has the added cost benefit of being able to run on commodity hardware. But, Gallo warned, this benefit may be limited if an organization is working with extremely large data sets -- such as streaming analytics, smart meter data or Web clickstream data -- in the petabyte or even multi-petabyte range.

"Now there's a case where traditional hardware, at least today, can't scale that large," he said.

In addition to studying the 3 V's, you've got to know your big data goals, too. This will help you decide if the investment is worth the cost.

"It's return on investment, right? You have to describe some problems you're trying to solve or some opportunities you can create," Gallo said.

For example, real-time streaming data can help a company with fleet vehicles cut transportation costs by integrating weather and traffic information. "In real time you could notify your driver that there's a wreck or a snowstorm they're headed into, and redirect them," said Gallo.

The cost of big data systems will drop in the coming years, Gallo believes, as component integration improves, administrative tools improve and big data applications become increasingly common, bringing data science to the masses.

Big data apps could help automate many data science tasks and help with basic analytics, allowing average business users to make data-driven decisions on their own.

"We'll see tools that allow the average business person to look into big data, as opposed to a super-user community, which is where it is today," Gallo said.

Attend Interop Las Vegas May 6-10 and learn the emerging trends in information risk management and security. Use Priority Code MPIWK by March 22 to save an additional $200 off the early bird discount on All Access and Conference Passes. Join us in Las Vegas for access to 125+ workshops and conference classes, 300+ exhibiting companies, and the latest technology. Register today!