6 Lies About Big Data

To paraphrase an old saying, if you torture data long enough, it'll tell you what you want to hear. And putting big data through that torture only lets us tell bigger lies. Marketing can justify crazy ad campaigns: "Sentiment analytics shows our latest campaign is actually a huge hit with the under-25-urban-vegan demo!" The supply chain team can use it to get more funding: "Our geolocation analysis shows if we invest in robotic warehouse automation, we'll reduce costs by 15%." Sales can explain why it missed its numbers: "We don't have an iOS app, and smartphone data shows that's what 87.4% of customers use. It's not our fault."

Don't get us wrong. The ability to collect and analyze data is a core IT value proposition. Companies such as Wal-Mart, FedEx, and Southwest Airlines gained strategic advantage by digging into their core business data long before it was labeled "big." And there's no question that more data is available than ever before, especially information from the Web and smart mobile devices. Our beef, though, is that most businesses aren't good at using the data they have now. What are the odds they'll get better at analysis by adding volumes without changing their strategies?

Our InformationWeek 2013 Big Data Survey shows that some companies are making progress. For example, most have built the required infrastructure and support various roles, in terms of primary data users; about one-third say they encourage wide access to information for business users. However, when it comes to data acquisition and use models, the wheels start to fall off. There are major gaps in data analysis, even for the most common types of information: transaction data, system logs, email, CRM, Web analytics.

Research: Big Data, Smart Data

6 Big Data Lies

Get the 2013 Big Data Survey full report free with registration.

This report includes 32 pages of action-oriented analysis, packed with 23 charts. What you'll find:

• Outlook on use of cloud services for big data management
• Top 14 big data tools in use, from Excel to Exadata to EDW

Get This And All Our Reports

Worse, fewer than 10% of the respondents to our survey say that ideas for promising new data points are primarily driven by a collabo- rative or cross-functional team within their companies. The stats we gleaned from our survey suggest this percentage should be much higher: Nearly half of respondents have 500 terabytes of data or more under management; 13% have more than 10 petabytes.

Surely there are untapped riches.

IT organizations clearly know there's a problem, as only 9% of respondents rate their companies as extremely effective users of the data they have. However, just 4% admit they stink at putting their data to its best use. Fact is, many organizations are deluding themselves into thinking they're empowering their businesses. So before you buy more storage, upgrade your warehouse platform, or spin up a massive Hadoop instance, let's take a reality check. Here are six big data lies organizations tell themselves. How many have you heard lately?

Lie 1: We understand how much data we have today

We asked in our survey which of seven key data sources are actively managed, hoping to see respondents widen their view beyond servers, storage arrays, and archives. Unfortunately, only 30% of respondents factor in their organization's cloud data, and just 11% include supply chain information. All that information zipping around on mobile devices? Considered by just 35% of survey respondents.

If you don't include dynamic data sets, you're setting your analysis up for failure. How can you do vendor performance reviews without details on how well suppliers do getting the right goods to you at the right time for a competitive price? Likewise, if you're studying customer behavior, how can you get a true picture without Web or cloud-based CRM data?