Big Data Success Remains Elusive: Study - InformationWeek

InformationWeek is part of the Informa Tech Division of Informa PLC

This site is operated by a business or businesses owned by Informa PLC and all copyright resides with them.Informa PLC's registered office is 5 Howick Place, London SW1P 1WG. Registered in England and Wales. Number 8860726.

Data Management // Big Data Analytics

Big Data Success Remains Elusive: Study

Just over a quarter of organizations say their big data initiatives are a success, according to a recent Capgemini study. So why are three of four unsuccessful?

9 CIO Tech Priorities For 2015
9 CIO Tech Priorities For 2015
(Click image for larger view and slideshow.)

Nearly eight out of ten organizations have big data projects underway, but only 27% describe their efforts as "successful," and a scant 8% as "very successful." But despite this dim view of their data-driven efforts thus far, 60% of executives surveyed recently by consulting firm Capgemini say big data will "disrupt their industry" within three years.

At first glance, the juxtaposition of failed implementations and high expectations may seem odd. Why the enthusiasm for a technology that doesn't appear to be paying off? According to Jeff Hunter, Capgemini's vice president of North American business information management, the high failure rate of big data initiatives isn't all that surprising, at least not initially.

"If we look at it in the analogy of other technologies that have come along the way -- a website, then digital presence, digital ecommerce, digital store, payments, and so forth -- we saw the same type of errors in the beginning of those technology trends," he said in a phone interview with InformationWeek.

The Capgemini survey of big data executives in November 2014 included 226 respondents in Europe, North America, and APAC (Asia-Pacific). It spanned multiple industries, including energy and utilities, financial services, manufacturing, pharmaceuticals, and retail.

A common situation with technology initiatives is that executives are often anxious to try something new and hyped -- in this case, big data -- partly in fear of falling behind their competitors. As a result, there's no clear big data mandate from the C-Suite, no well-defined strategy to improve, modify, or invent.

(Image: DARPA)
(Image: DARPA)

These ill-defined objectives are the primary cause of the majority of big data failures. "Generally, it's a disconnect between the output and a clearly defined business driver or goal," said Hunter. "And along the way, people get engulfed in the technology."

In some cases, organizations attempt to use their existing data management systems to process big data streams, often with poor results.

"Legacy systems that generally have been used to great efficiency for enterprise data management and content management, sometimes aren’t suited to these new data sources," Hunter said.

These sources may include social media streams, log data, and sensor data from the emerging Internet of Things to evaluate customers, transactions, and user sentiment. But this approach usually doesn’t go well, resulting in what he calls a "fumbling of the legacy systems."

Another problem is scattered silos of data. The Capgemini report states:

Seventy-nine percent of organizations have not fully integrated their data sources across the organization. This means decision-makers lack a unified view of data, which prevents them from taking accurate and timely decisions.

So how can organizations achieve greater success with their big data initiatives? The key is a well-defined organizational structure, a systematic implementation plan, and strong leadership, Hunter said.

[Does everyone need to be a data analyst? Read Data Analyst: Does Everybody Need To Be One?]

Big data initiatives are rarely "division-centric," but rather cut across multiple departments, the report states. Success rates for organizations with an analytics business unit are nearly 2.5 times higher than those with "ad-hoc, isolated teams," the survey found.

"The firms that we see succeed [have] centralized the concept of consumption of big data technology, leverage of data science, and application of analytics," said Hunter. "They’re the ones we see moving faster."

Attend Interop Las Vegas, the leading independent technology conference and expo series designed to inspire, inform, and connect the world's IT community. In 2015, look for all new programs, networking opportunities, and classes that will help you set your organization’s IT action plan. It happens April 27 to May 1. Register with Discount Code MPOIWK for $200 off Total Access & Conference Passes.

Jeff Bertolucci is a technology journalist in Los Angeles who writes mostly for Kiplinger's Personal Finance, The Saturday Evening Post, and InformationWeek. View Full Bio

We welcome your comments on this topic on our social media channels, or [contact us directly] with questions about the site.
Comment  | 
Print  | 
More Insights
Oldest First  |  Newest First  |  Threaded View
User Rank: Ninja
2/2/2015 | 9:33:31 PM
Data science
I think that the data science aspect of this is really important – companies that use an actual data scientist who knows what to do with data is really important I think. 

Big data is not just another facet of technology. You need the right people looking at the data in order to gain insight into what it all really means. 
Li Tan
Li Tan,
User Rank: Ninja
2/3/2015 | 2:30:22 AM
Re: Data science
I do agree with you. Big data is not something that can be handled well by pure technician. You need to understand the real business need behind to dig out valuable information from existing data mine.
User Rank: Apprentice
2/3/2015 | 10:51:00 AM
an accurate summation of the state of big data in the enterprise
Great stuff! The article and survey are consistent with what we see at Cazena. Enterprise business leaders see the promise of big data to transform the speed and quality of their decision-making, but struggle with the complexity of the new technologies and other practical matters that so far keep big data mostly on the fringes of true production-level success. As is pointed out, this is a natural phase of growth that we'll get through if we keep our eye on the prize!
User Rank: Ninja
2/4/2015 | 11:50:05 AM
Slowly maturing
There is a good reason why success is eluding organizations.  There needs to be a little more devotion to building the data maturity level.  According to a recent IDG survey, organizations do not rate themselves as highly effective in any of the critical big data tasks. For instance, only 11 percent say their organizations are extremely capable at knowing which questions to ask.  Likewise, only 5 percent see their organizations as effective at disseminating insights across the organization. When you couple this with businesses trying for the big wins immediately, you are bound to see disappointment. Bottomline, businesses need to start small, build confidence and ability with experience. 


Peter Fretty, IDG blogger working on behalf of SAS
2021 Outlook: Tackling Cloud Transformation Choices
Joao-Pierre S. Ruth, Senior Writer,  1/4/2021
Enterprise IT Leaders Face Two Paths to AI
Jessica Davis, Senior Editor, Enterprise Apps,  12/23/2020
10 IT Trends to Watch for in 2021
Cynthia Harvey, Freelance Journalist, InformationWeek,  12/22/2020
White Papers
Register for InformationWeek Newsletters
Current Issue
2021 Top Enterprise IT Trends
We've identified the key trends that are poised to impact the IT landscape in 2021. Find out why they're important and how they will affect you.
Flash Poll