Big Data Debate: Do Analytics Trump Intuition?

16 Top Big Data Analytics Platforms

Will the next evolution in big data remove human intuition from key business decisions and rely exclusively on data-driven analytics?

Probably not, but organizations will increasingly rely on analytics to make real-time decisions based on a rising tide of big data, predicts Roei Ganzarski, CEO of BoldIQ, a Seattle-based predictive analytics company. This doesn't mean, however, that managers warmly embrace analytics over intuition. In fact, the opposite is often true. C-suite types often scoff at the notion that software can outperform insights acquired from years in the trenches.

"We run into that every day," says Ganzarski in a phone interview with InformationWeek. "Our response is, 'It's not that we think you're doing something wrong.'" Rather, he says, analytics provide "an additional tool that enables you to do things the human mind simply can't do. We're not here to replace you. We're here to enhance your ability to make decisions."

[Variety makes big data tough to secure. Learn why: Big Data Brings Big Security Problems.]

Obviously, the human brain still trumps computers at myriad tasks. But software is significantly better at split-second analysis, he says, providing a transportation industry example. "Let's talk about next-day and same-day deliveries. People talk about that as the Holy Grail of distribution and supply chain -- where you make an order and it shows up on your door that very day." This presents the sort of big-data challenge suited for analytics

"Once the order has been put in place, how does [the delivery company] make sure its vehicle-and-driver network is set up in such a way that, within milliseconds, it can tell you which vehicle and driver should do the pickup and delivery so that everything is done on time and in a profitable manner?"

The Human Algorithm
Barrett Thompson is general manager of pricing excellence solutions for Zilliant, an optimization company that helps businesses use in-house data to make better sales and pricing decisions. In a January interview with InformationWeek on the algorithm-versus-intuition topic, Thompson pointed out that the algorithmic approach to decision-making is based on the collective experiences of multiple individuals within a business.

"Let's say I'm looking at set of sales transactions completed over the past year, and I have five million of them sitting in the data warehouse," says Thompson. "What I have captured in that data, what I have to find a way to unlock, is the distilled wisdom and experience of five hundred salespeople who encountered tens of thousands, or hundreds of thousands, of unique selling circumstances."

A predictive algorithm, he says, doesn't create knowledge out of thin air. Rather it's an "accelerator" of what an organization already knows. It's driven by human experience, by "data points you've lived through."

"I can't see what the other 499 salespeople did, and I can't even remember what I did in March of last year. I make so many decisions that they become lost in memory," Thompson said. "If I had a software tool, an algorithm that could remind me of what I've learned, and reveal to me what everyone else has learned... I could look at the guidance coming out of the algorithm as a distilled and refined experience from myself and people just like me."

Of course, the big caveat here is that Thompson and Ganzarski are heavily vested in the analytics camp. Neither is an unbiased observer of the intuition-versus-algorithm debate.

In a December 2013 Harvard Business Review blog on intuition's role in a data-driven organization, analytics expert and author Tom Davenport argued that major big-data projects are often driven by intuition:

Fair enough, chalk one up for intuition. But in the enterprise, analytics will play an increasingly influential role in the intuitive process.

IBM, Microsoft, Oracle, and SAP are fighting to become your in-memory technology provider. Do you really need the speed? Get the digital In-Memory Databases issue of InformationWeek today.