To paraphrase an old saying, if you torture data long enough, it'll tell you what you want to hear. And putting big data through that torture only lets us tell bigger lies. Marketing can justify crazy ad campaigns: "Sentiment analytics shows our latest campaign is actually a huge hit with the under-25-urban-vegan demo!" The supply chain team can use it to get more funding: "Our geolocation analysis shows if we invest in robotic warehouse automation, we'll reduce costs by 15%." Sales can explain why it missed its numbers: "We don't have an iOS app, and smartphone data shows that's what 87.4% of customers use. It's not our fault."
Don't get us wrong. The ability to collect and analyze data is a core IT value proposition. Companies such as Wal-Mart, FedEx, and Southwest Airlines gained strategic advantage by digging into their core business data long before it was labeled "big." And there's no question that more data is available than ever before, especially information from the Web and smart mobile devices. Our beef, though, is that most businesses aren't good at using the data they have now. What are the odds they'll get better at analysis by adding volumes without changing their strategies?
Our InformationWeek 2013 Big Data Survey shows that some companies are making progress. For example, most have built the required infrastructure and support various roles, in terms of primary data users; about one-third say they encourage wide access to information for business users. However, when it comes to data acquisition and use models, the wheels start to fall off. There are major gaps in data analysis, even for the most common types of information: transaction data, system logs, email, CRM, Web analytics.
Worse, fewer than 10% of the respondents to our survey say that ideas for promising new data points are primarily driven by a collabo- rative or cross-functional team within their companies. The stats we gleaned from our survey suggest this percentage should be much higher: Nearly half of respondents have 500 terabytes of data or more under management; 13% have more than 10 petabytes.
Surely there are untapped riches.
IT organizations clearly know there's a problem, as only 9% of respondents rate their companies as extremely effective users of the data they have. However, just 4% admit they stink at putting their data to its best use. Fact is, many organizations are deluding themselves into thinking they're empowering their businesses. So before you buy more storage, upgrade your warehouse platform, or spin up a massive Hadoop instance, let's take a reality check. Here are six big data lies organizations tell themselves. How many have you heard lately?
Lie 1: We understand how much data we have today
We asked in our survey which of seven key data sources are actively managed, hoping to see respondents widen their view beyond servers, storage arrays, and archives. Unfortunately, only 30% of respondents factor in their organization's cloud data, and just 11% include supply chain information. All that information zipping around on mobile devices? Considered by just 35% of survey respondents.
If you don't include dynamic data sets, you're setting your analysis up for failure. How can you do vendor performance reviews without details on how well suppliers do getting the right goods to you at the right time for a competitive price? Likewise, if you're studying customer behavior, how can you get a true picture without Web or cloud-based CRM data?