Software // Information Management
01:00 PM
Connect Directly
Online Event: Top 4 DevOps Discoveries in 2015
Feb 25, 2016
As business technology becomes increasingly software-centric, optimizing the collaboration between ...Read More>>

6 Lies About Big Data

Our 2013 Big Data Survey shows we're not lacking facts, figures, or tools to wrangle them. So why do just 9% of respondents rate themselves as extremely effective users of data?

Lie 2: The data we have is good

We'll bet money that most respondents' data sets are inaccurate, incomplete, and/or misaligned with one another. Do you really have a single source of truth? Do different groups slice data in different ways? Are you making decisions based on inaccurate or incomplete data?

Case in point: 19% of companies in our survey use geolocation as part of their analysis strategy, pulling information from smart devices and Web visitors to understand behavior. However, Web location tracking is notoriously inaccurate when it comes to enterprise and institutional traffic. That's because most companies and government agencies work in private clouds with a limited number of egress points. If you're using Web location data to track the success of your sales and marketing programs by region, you're likely basing decisions on bad information. That big block of traffic from Boston may actually come from an enterprise with offices in the Midwest.

Who's checking data quality? Just one in four respondents identified a dedicated business analyst group as one of the top two users of data within their company. It's simply amazing how many reports and graphs we see without sampling or accuracy notes. For example, almost every company does customer surveys, yet very few indicate confidence levels or bias results. Got 25,000 customers? Your customer service survey should have 1,843 respondents if you want a 99% confidence level with a plus or minus 3% margin of error. Furthermore, results should be biased by revenue level. Reality is, we just don't see that done with any type of data.

Lie 3: Everything will be OK if we can just get more tools

A quarter of respondents plan to use more big data tools over the next 12 months. Now, we like Hadoop, NoSQL, Splunk, and the plethora of other big data options out there, but we recommend looking at what data sets are sitting idle before cutting a check. Given the low levels of use of the 20 internal and external data sets we asked about, it's clear the problem is related more to staffing than systems.

Unfortunately, fewer respondents plan to invest in big data staff versus spending money on technology. Only 33% plan to grow their training and development programs; 9% are cutting back. Net new hiring ranked at the bottom of our list, with 17% growing staffing levels compared with 14% cutting.

Nowhere is this "tools over people" focus more evident than in healthcare. The federal government's electronic records incentives have driven the industry to a new level of data collection and reporting. But now that healthcare providers have all this data, they're trying to figure out how to use it. "There is big money to be made in healthcare big data, so everyone and their brother was throwing up solutions," says Bill Gillis, CIO of Beth Israel Deaconess Physician Organization. But it's important to work with people who understand healthcare organizations and the complexity of their data, Gillis says. "The business need should drive the process," he says. "The tool alone will not change much. Finding a skilled hand that can effectively wield that tool will."

Lie 4: There's an expertise shortage

Speaking of staff, an oft-quoted McKinsey & Co. study estimates a shortfall of 140,000 to 190,000 people in "big data staffing" by 2018. Our own InformationWeek Staffing Survey shows that 18% of respondents focused on big data want to increase staff in this area by more than 30% in the next two years, but 53% say it will be difficult to find people with the right skills.

Roll the clock back a few years and substitute the words "virtualization engineer" or "Cobol programmer" or even "webmaster" for "big data specialist" and you'll similarly find people predicting doom. Don't get sucked in again. You already have much of this talent within your organization; you just need to set it free. Consider that 39% of respondent organizations have department-level analysts as the main users of their information. Break those people out of their department silos and move them toward a more holistic view of data.

For example, a U.S. retailer we worked with always had separate data teams aligned with various departments. The strongest team was within the catalog group, banging out circ plans, catalog yields, conversion rates, even profit per page. Great stuff, but that team was limited to using catalog and financial data. Siloed from the Web team, they were missing the transformation happening within the customer base. Separate departments, separate views of the truth.

Don't blame the analysts. The company built the structure, and IT wasn't involved enough to identify the information gaps between departments. Our point is, IT not only has to understand the data itself, but it must also become an integral part of identifying and growing the centralized talent pool. That means putting more emphasis on training and talent development. Competitors will steal some of your more talented big data pros if you don't give them a reason to stick around.

We scanned the online listings of the major hiring sites and found that the salary levels for data analysts still range from about $55,000 to low six figures. However, if you add "big data" to the standard titles, the average salary doubles. Expect everyone's LinkedIn titles to change in the next 12 months.

chart: What's your company's approach to data analysis?

2 of 3
Comment  | 
Print  | 
More Insights
Newest First  |  Oldest First  |  Threaded View
Engineer Veteran
Engineer Veteran,
User Rank: Apprentice
11/13/2012 | 2:06:29 PM
re: 6 Lies About Big Data
I couldn't agree more. There is so much hype about 'big data' when reality is that most organizations don't have the guts to actually do something that data clearly points out is needed.
Patrick Taylor
Patrick Taylor,
User Rank: Apprentice
11/10/2012 | 6:35:11 PM
re: 6 Lies About Big Data
Lie # 6 is the key - doing something with the analysis. Often the focus of analytic success stories is the really smart strategic decisions. But let's face it you don't make that many strategic decisions. Now think about tactical decisions - retaining a customer, what product to offer, what price to pay for supplies. There are several orders of magnitude more tactical decisions than strategic ones.

"Doing something with our analysis" at the front lines of business will drive ROI from Big Data.
The Agile Archive
The Agile Archive
When it comes to managing data, donít look at backup and archiving systems as burdens and cost centers. A well-designed archive can enhance data protection and restores, ease search and e-discovery efforts, and save money by intelligently moving data from expensive primary storage systems.
Register for InformationWeek Newsletters
White Papers
Current Issue
How to Knock Down Barriers to Effective Risk Management
Risk management today is a hodgepodge of systems, siloed approaches, and poor data collection practices. That isn't how it should be.
Twitter Feed
InformationWeek Radio
Sponsored Live Streaming Video
Everything You've Been Told About Mobility Is Wrong
Attend this video symposium with Sean Wisdom, Global Director of Mobility Solutions, and learn about how you can harness powerful new products to mobilize your business potential.