Big Data Survey: Trouble Brewing For IT

Download the entire November issue of InformationWeek Tech Digest, distributed in an all-digital format (registration required).

The more you know, the more you know what you don't know. This year's InformationWeek Big Data and Analytics Survey stands as a prime example of that adage

We see companies moving in a positive direction on some fronts -- they're integrating more with external data sets, expanding their use of analysis tools, and focusing more on customer data. However, this progress is tempered by a disturbing shift. When we asked survey respondents (266 in all, at organizations with 50 or more employees) to rate how effectively their organizations identify and use data, more characterize their approaches as "limited and siloed" than "holistic and inclusive." Only 30% of respondents say they're very or extremely effective at identifying critical data and using it to make decisions -- that's down from 42% in 2013. A whopping 63% say they're only moderately to slightly effective, with the remaining 7% throwing in the towel and claiming defeat.

Add it up, data jockeys: 70% admit their companies are below par when it comes to data effectiveness, an increase of 12 points over 2013.

It's not as if businesses don't get what big data can do for them. In fact, the areas cited as most ripe for improvement are critical for any enterprise: competitive intelligence, business security, customer service, and product development. And we see good momentum in some key areas, especially pulling in external data sources such as Web analytics. And more than 55% of respondents plan to expand their analytics tool capabilities in 2015.

The right priorities, the right data, the right tools. Why are we faltering?

First, with new data sets and better tools come a (sometimes painful) realization of how far most of us have to go to become truly digital enterprises. Sure, we can develop a better Web presence, create an Internet of Things strategy, and expand our social footprint. But most survey respondents lack a commitment to using all the data they can now tap into. Very few orgs are pulling in all the sources needed to get a 360-degree view of customers, for example.

Lack of budget is the top barrier to successful use of big data, cited by 31% of respondents. But let's be real: Funding complaints always rate No. 1. In second place? Fourteen percent have the guts to say there are more important IT priorities, up three points from last year. We think that response gets at the root cause of big data dawdling: IT is often saying, if not in so many words: "Hey, CMO. You want to own big data and digital? Fine, have fun. Don't call us, we'll call you."

This "stepping away" of IT from a pivotal role in data analysis is confirmed by looking at who pushes new ideas for data analysis. Only 19% of the respondents to our survey say IT is the primary driver, down from 26% in 2013.

Enterprises are faltering in their ability to comprehensively analyze big data, and IT has opted to walk away.

Look, for years IT organizations have been told they don't own enterprise data, the business does. Lately we've heard about the rise of the CMO and how it takes that mindset to really know what data matters and how to mine it. So the message too many IT teams seem to be taking away: "This isn't an IT problem. We build the systems, keep the lights on, try to keep attackers out. We don't own big data. Our input isn't wanted."

Big data is tied to digital transformation, which is inextricably linked with the e-commerce, social, IoT, and mobile movements redefining how businesses operate. These are all becoming "non-IT" functions, so why hold on to big data responsibilities?

Simple -- because your organization doesn't have the skills outside of IT to handle this tsunami of data. Deloitte Consulting called the insurance industry "information rich, knowledge poor." But it's not just insurance. This moniker applies to almost every industry. You may have a crack team of business analysts, data scientists, and Excel jockeys, but they simply can't do the heavy lifting needed to bring in all the information needed to yield real knowledge.

Our survey shows that almost everyone is pulling in their key financial, sales, and product data. That's old-school stuff; no surprise there. Lots of companies are also tapping their server logs, email files, and CRM software. However, the wheels start to come off when it comes to unstructured data and data sources that aren't linked easily, such as phone logs, smartphone data, and partner sales data.

All of these sources probably are accessible and won't blow up your security model, so why haven't they been tapped? Because doing so would require IT -- not data analysis -- skills.

This pattern pops up again in response to our questions about external data. More than half of enterprises in our survey analyze Web and social data, and an increasing percentage analyze public records. Again, this is relatively clean data than any analyst can pull. Enterprises are struggling with data that requires developer time (translation: IT resources).

In fact, some of the richest, newest big data is sitting idle. We're talking sentiment analytics (aka analysis of brand chatter) -- only half of survey respondents do it. Geolocation, sensor, and RFID data? Also left out of the loop, as well as third-party intelligence services such as Dun & Bradstreet's. Even unstructured data stored in the cloud is getting the short end of the stick when it comes to analysis. IT organizations have helped fuel the explosive growth of Google Docs, Office 365, Box.com, yet only 42% of enterprises in our survey actually analyze that data.

This is what happens when IT bows out of big data. It's not a pretty picture.

To read the rest of this story,
download the November issue of InformationWeek Tech Digest.