Data Analytics Can Backfire Without Experts - InformationWeek
IoT
IoT
Software // Information Management
Commentary
1/10/2012
02:13 PM
Commentary
Commentary
Commentary
50%
50%
RELATED EVENTS
[Cybersecurity] Costs vs. Benefits
Feb 28, 2017
This online event, hosted by the editors of Dark Reading, brings together IT security leaders, bus ...Read More>>

Data Analytics Can Backfire Without Experts

Software vendors will tell you their analytics tools provide complete "solutions," but skip skilled statisticians for the heavy lifting at your own peril.

Ever since the economic collapse of 2008, your senior management team has been clamoring for improved information and analyses to make quicker and better decisions. So this strong demand finally justified your buying one of those nouveau business analytics packages a year ago. Yay!

However, while your team has quickly developed goo-gobs of new analytical reports and graphs, they don't seem to offer anything more informative and decisive than was available before. In fact, a few of those fancy reports have added more confusion than clarity and may have even been flat out wrong!

What happened? Did you buy a lemon product?

Probably not. It's far more likely that you thought your organization was buying a "solution" when it was buying a "tool." If it's any consolation, you're not alone. Vendors are expert at selling their analytics products as magical software that can discover noteworthy trends, exceptions, and anomalies and automatically yield epiphanic reports.

Let's compare the problem to cabinet making. Advanced tools have made it quicker and easier to build high-quality cabinetry, but such is the case when used only by skilled craftspeople. I'm pretty good with a hammer, but the improved tools don't enable me to miraculously create beautiful cabinets.

You might be thinking that the rudimentary mathematical and statistical methods that underlie business analytics are just not that difficult. You shouldn't need an expert; this is stuff you learned in your Statistics 101 class, right?

Wrong. You've likely fallen prey to the Dunning-Kruger Effect, which stipulates that once novices have learned the general concepts of a particular subject, they assume they "know it all." Essentially, the less you know about a subject, the less you think there is to know. As the philosopher Bertrand Russell once said: "In the modern world, the stupid are cocksure while the intelligent are full of doubt."

If not our programmers and financial analysts, then who? It's a job for the skilled statistician and industrial engineer. This is the key ingredient missing in your recipe. In fact, it's the only key ingredient; every other one has alternatives and substitutes.

Not convinced? Let's consider a real world example (company name changed to protect the guilty).

Studies have consistently reported that one out of 20 retail merchandise employees commits company theft. Given this high incidence, the "MakeIt Co." initiated a "Fraud Identification" project. In the first phase, an established team decided to identify and report probable fraud based on the "case work" of the company's Security and Internal Audit department. Based on a review of numerous prior cases, the SIA determined that in 80% of the cases where it found coupon redemptions totaling more than 10% of sales for a given employee on a given day, fraud was involved. A programmer analyst designed a "High Risk Fraud" report.

The employee with the highest number of coupon redemptions was called into the manager's office. He was shown his top ranking on the "High Risk Fraud" report and asked to explain his questionable activity. Embarrassed and humiliated, the employee resigned. One week later, he filed a complaint against the company.

In its brief, the company cited that historical investigations proved there to be an "80% probability that employees with such coupon redemption activity were stealing." MakeIt defended its position that "80% probability is certainly in the realm of high risk." The plaintiff's attorneys agreed that an 80% probability would constitute a high risk, but they alleged that MakeIt's computation of that probability was incorrect. They argued that the actual probability was 17%, which constituted a low risk.

How in the world did the plaintiff's attorneys arrive at 17%?

Well, in addition to knowing that "in 80% of cases reviewed, high coupon redemptions were the result of employee theft," the company also knew that "1 out of 20 employees perform some type of theft." Indeed, the Fraud Identification project was initiated based on this initial fact.

What does that have to do with the matter? Everything!

In determining the probability that an employee is a thief, MakeIt must take into account both facts, calculating the probability using Baye's Probability Rule, a complicated formula well known to statisticians but foreign to most others. Indeed, the plaintiff's attorneys were correct! The company issued a formal apology to the former employee, who was promptly reinstated to his former position and compensated for the emotional distress. MakeIt discontinued its "Fraud Report" and canceled the Fraud Identification project.

There are many other examples of forecasting gone awry, misstated trends, and flawed market basket analyses. Here are the key takeaways:

-- The Dunning-Kruger effect leads us to underestimate the skills necessary to perform meaningful data analyses. As a result, incorrect mathematical/statistical methods are applied and erroneous (and sometimes misleading) results are presented.

-- The key ingredient to successful business analytics is the involvement of a skilled statistician (or industrial engineer, depending on the subject matter).

-- The particular tools chosen to perform the analytics are of little importance.

-- You can't buy a "solution" to your analytical needs. You must enlist well-educated and -trained specialists to perform the analyses and deliver meaningful results.

One final observation: The cost for your organization to acquire business analytics tools will likely be much higher than the cost of acquiring statistical talent. And the cost of incorrect analyses, and the resulting misguided decisions/directions, could be an even more significant cost to your company--and ultimately to your career.

Jeff Chasney is executive VP of strategic planning and CIO with CKE Restaurants Inc., which operates the Carl's Jr., Hardee's, and other restaurant brands.

InformationWeek is conducting our third annual State of Enterprise Storage survey on data management technologies and strategies. Upon completion, you will be eligible to enter a drawing to receive an Apple 32-GB iPod Touch. Take our Enterprise Storage Survey now. Survey ends Jan. 13.

Comment  | 
Print  | 
More Insights
Comments
Newest First  |  Oldest First  |  Threaded View
Kimberly K
50%
50%
Kimberly K,
User Rank: Apprentice
3/21/2013 | 6:08:31 PM
re: Data Analytics Can Backfire Without Experts
"I like this place and willingly could waste my time in it" (Shakespeare "As You Like "IT";p) Great article.
ACHAVDAME1
50%
50%
ACHAVDAME1,
User Rank: Apprentice
1/11/2012 | 6:00:57 PM
re: Data Analytics Can Backfire Without Experts
Good article. This helps push the case that businesses need to look at BI SaaS based solutions and need to ensure that they bring in 3rd party providers at the data extraction and migration stage rather than trying to do this themselves....
How Enterprises Are Attacking the IT Security Enterprise
How Enterprises Are Attacking the IT Security Enterprise
To learn more about what organizations are doing to tackle attacks and threats we surveyed a group of 300 IT and infosec professionals to find out what their biggest IT security challenges are and what they're doing to defend against today's threats. Download the report to see what they're saying.
Register for InformationWeek Newsletters
White Papers
Current Issue
2017 State of the Cloud Report
As the use of public cloud becomes a given, IT leaders must navigate the transition and advocate for management tools or architectures that allow them to realize the benefits they seek. Download this report to explore the issues and how to best leverage the cloud moving forward.
Video
Slideshows
Twitter Feed
InformationWeek Radio
Archived InformationWeek Radio
Join us for a roundup of the top stories on InformationWeek.com for the week of November 6, 2016. We'll be talking with the InformationWeek.com editors and correspondents who brought you the top stories of the week to get the "story behind the story."
Sponsored Live Streaming Video
Everything You've Been Told About Mobility Is Wrong
Attend this video symposium with Sean Wisdom, Global Director of Mobility Solutions, and learn about how you can harness powerful new products to mobilize your business potential.
Flash Poll