Data Analytics Can Backfire Without Experts

Ever since the economic collapse of 2008, your senior management team has been clamoring for improved information and analyses to make quicker and better decisions. So this strong demand finally justified your buying one of those nouveau business analytics packages a year ago. Yay!

However, while your team has quickly developed goo-gobs of new analytical reports and graphs, they don't seem to offer anything more informative and decisive than was available before. In fact, a few of those fancy reports have added more confusion than clarity and may have even been flat out wrong!

What happened? Did you buy a lemon product?

Probably not. It's far more likely that you thought your organization was buying a "solution" when it was buying a "tool." If it's any consolation, you're not alone. Vendors are expert at selling their analytics products as magical software that can discover noteworthy trends, exceptions, and anomalies and automatically yield epiphanic reports.

Let's compare the problem to cabinet making. Advanced tools have made it quicker and easier to build high-quality cabinetry, but such is the case when used only by skilled craftspeople. I'm pretty good with a hammer, but the improved tools don't enable me to miraculously create beautiful cabinets.

You might be thinking that the rudimentary mathematical and statistical methods that underlie business analytics are just not that difficult. You shouldn't need an expert; this is stuff you learned in your Statistics 101 class, right?

Wrong. You've likely fallen prey to the Dunning-Kruger Effect, which stipulates that once novices have learned the general concepts of a particular subject, they assume they "know it all." Essentially, the less you know about a subject, the less you think there is to know. As the philosopher Bertrand Russell once said: "In the modern world, the stupid are cocksure while the intelligent are full of doubt."

If not our programmers and financial analysts, then who? It's a job for the skilled statistician and industrial engineer. This is the key ingredient missing in your recipe. In fact, it's the only key ingredient; every other one has alternatives and substitutes.

Not convinced? Let's consider a real world example (company name changed to protect the guilty).

Studies have consistently reported that one out of 20 retail merchandise employees commits company theft. Given this high incidence, the "MakeIt Co." initiated a "Fraud Identification" project. In the first phase, an established team decided to identify and report probable fraud based on the "case work" of the company's Security and Internal Audit department. Based on a review of numerous prior cases, the SIA determined that in 80% of the cases where it found coupon redemptions totaling more than 10% of sales for a given employee on a given day, fraud was involved. A programmer analyst designed a "High Risk Fraud" report.

The employee with the highest number of coupon redemptions was called into the manager's office. He was shown his top ranking on the "High Risk Fraud" report and asked to explain his questionable activity. Embarrassed and humiliated, the employee resigned. One week later, he filed a complaint against the company.

In its brief, the company cited that historical investigations proved there to be an "80% probability that employees with such coupon redemption activity were stealing." MakeIt defended its position that "80% probability is certainly in the realm of high risk." The plaintiff's attorneys agreed that an 80% probability would constitute a high risk, but they alleged that MakeIt's computation of that probability was incorrect. They argued that the actual probability was 17%, which constituted a low risk.

How in the world did the plaintiff's attorneys arrive at 17%?

Well, in addition to knowing that "in 80% of cases reviewed, high coupon redemptions were the result of employee theft," the company also knew that "1 out of 20 employees perform some type of theft." Indeed, the Fraud Identification project was initiated based on this initial fact.

What does that have to do with the matter? Everything!

In determining the probability that an employee is a thief, MakeIt must take into account both facts, calculating the probability using Baye's Probability Rule, a complicated formula well known to statisticians but foreign to most others. Indeed, the plaintiff's attorneys were correct! The company issued a formal apology to the former employee, who was promptly reinstated to his former position and compensated for the emotional distress. MakeIt discontinued its "Fraud Report" and canceled the Fraud Identification project.

There are many other examples of forecasting gone awry, misstated trends, and flawed market basket analyses. Here are the key takeaways:

-- The Dunning-Kruger effect leads us to underestimate the skills necessary to perform meaningful data analyses. As a result, incorrect mathematical/statistical methods are applied and erroneous (and sometimes misleading) results are presented.

-- The key ingredient to successful business analytics is the involvement of a skilled statistician (or industrial engineer, depending on the subject matter).

-- The particular tools chosen to perform the analytics are of little importance.

-- You can't buy a "solution" to your analytical needs. You must enlist well-educated and -trained specialists to perform the analyses and deliver meaningful results.

One final observation: The cost for your organization to acquire business analytics tools will likely be much higher than the cost of acquiring statistical talent. And the cost of incorrect analyses, and the resulting misguided decisions/directions, could be an even more significant cost to your company--and ultimately to your career.

Jeff Chasney is executive VP of strategic planning and CIO with CKE Restaurants Inc., which operates the Carl's Jr., Hardee's, and other restaurant brands.

InformationWeek is conducting our third annual State of Enterprise Storage survey on data management technologies and strategies. Upon completion, you will be eligible to enter a drawing to receive an Apple 32-GB iPod Touch. Take our Enterprise Storage Survey now. Survey ends Jan. 13.