Big Data Success Remains Elusive: Study

9 CIO Tech Priorities For 2015

Nearly eight out of ten organizations have big data projects underway, but only 27% describe their efforts as "successful," and a scant 8% as "very successful." But despite this dim view of their data-driven efforts thus far, 60% of executives surveyed recently by consulting firm Capgemini say big data will "disrupt their industry" within three years.

At first glance, the juxtaposition of failed implementations and high expectations may seem odd. Why the enthusiasm for a technology that doesn't appear to be paying off? According to Jeff Hunter, Capgemini's vice president of North American business information management, the high failure rate of big data initiatives isn't all that surprising, at least not initially.

"If we look at it in the analogy of other technologies that have come along the way -- a website, then digital presence, digital ecommerce, digital store, payments, and so forth -- we saw the same type of errors in the beginning of those technology trends," he said in a phone interview with InformationWeek.

The Capgemini survey of big data executives in November 2014 included 226 respondents in Europe, North America, and APAC (Asia-Pacific). It spanned multiple industries, including energy and utilities, financial services, manufacturing, pharmaceuticals, and retail.

A common situation with technology initiatives is that executives are often anxious to try something new and hyped -- in this case, big data -- partly in fear of falling behind their competitors. As a result, there's no clear big data mandate from the C-Suite, no well-defined strategy to improve, modify, or invent.

These ill-defined objectives are the primary cause of the majority of big data failures. "Generally, it's a disconnect between the output and a clearly defined business driver or goal," said Hunter. "And along the way, people get engulfed in the technology."

In some cases, organizations attempt to use their existing data management systems to process big data streams, often with poor results.

"Legacy systems that generally have been used to great efficiency for enterprise data management and content management, sometimes aren’t suited to these new data sources," Hunter said.

These sources may include social media streams, log data, and sensor data from the emerging Internet of Things to evaluate customers, transactions, and user sentiment. But this approach usually doesn’t go well, resulting in what he calls a "fumbling of the legacy systems."

Another problem is scattered silos of data. The Capgemini report states:

Seventy-nine percent of organizations have not fully integrated their data sources across the organization. This means decision-makers lack a unified view of data, which prevents them from taking accurate and timely decisions.

So how can organizations achieve greater success with their big data initiatives? The key is a well-defined organizational structure, a systematic implementation plan, and strong leadership, Hunter said.

[Does everyone need to be a data analyst? Read Data Analyst: Does Everybody Need To Be One?]

Big data initiatives are rarely "division-centric," but rather cut across multiple departments, the report states. Success rates for organizations with an analytics business unit are nearly 2.5 times higher than those with "ad-hoc, isolated teams," the survey found.

"The firms that we see succeed [have] centralized the concept of consumption of big data technology, leverage of data science, and application of analytics," said Hunter. "They’re the ones we see moving faster."

Attend Interop Las Vegas, the leading independent technology conference and expo series designed to inspire, inform, and connect the world's IT community. In 2015, look for all new programs, networking opportunities, and classes that will help you set your organization’s IT action plan. It happens April 27 to May 1. Register with Discount Code MPOIWK for $200 off Total Access & Conference Passes.