Big Data // Big Data Analytics
News
5/27/2014
10:19 AM
Connect Directly
Google+
RSS
E-Mail
50%
50%

Big Data Debate: Do Analytics Trump Intuition?

C-suite types often scoff at the notion that software can outperform insights acquired from years in the trenches. Fear not: Analytics will complement, not replace us.

16 Top Big Data Analytics Platforms
16 Top Big Data Analytics Platforms
(Click image for larger view and slideshow.)

Will the next evolution in big data remove human intuition from key business decisions and rely exclusively on data-driven analytics?

Probably not, but organizations will increasingly rely on analytics to make real-time decisions based on a rising tide of big data, predicts Roei Ganzarski, CEO of BoldIQ, a Seattle-based predictive analytics company. This doesn't mean, however, that managers warmly embrace analytics over intuition. In fact, the opposite is often true. C-suite types often scoff at the notion that software can outperform insights acquired from years in the trenches.

"We run into that every day," says Ganzarski in a phone interview with InformationWeek. "Our response is, 'It's not that we think you're doing something wrong.'" Rather, he says, analytics provide "an additional tool that enables you to do things the human mind simply can't do. We're not here to replace you. We're here to enhance your ability to make decisions."

[Variety makes big data tough to secure. Learn why: Big Data Brings Big Security Problems.]

Obviously, the human brain still trumps computers at myriad tasks. But software is significantly better at split-second analysis, he says, providing a transportation industry example. "Let's talk about next-day and same-day deliveries. People talk about that as the Holy Grail of distribution and supply chain -- where you make an order and it shows up on your door that very day." This presents the sort of big-data challenge suited for analytics

"Once the order has been put in place, how does [the delivery company] make sure its vehicle-and-driver network is set up in such a way that, within milliseconds, it can tell you which vehicle and driver should do the pickup and delivery so that everything is done on time and in a profitable manner?"

(Image: Geralt on Pixabay)
(Image: Geralt on Pixabay)

The Human Algorithm
Barrett Thompson is general manager of pricing excellence solutions for Zilliant, an optimization company that helps businesses use in-house data to make better sales and pricing decisions. In a January interview with InformationWeek on the algorithm-versus-intuition topic, Thompson pointed out that the algorithmic approach to decision-making is based on the collective experiences of multiple individuals within a business.

"Let's say I'm looking at set of sales transactions completed over the past year, and I have five million of them sitting in the data warehouse," says Thompson. "What I have captured in that data, what I have to find a way to unlock, is the distilled wisdom and experience of five hundred salespeople who encountered tens of thousands, or hundreds of thousands, of unique selling circumstances."

A predictive algorithm, he says, doesn't create knowledge out of thin air. Rather it's an "accelerator" of what an organization already knows. It's driven by human experience, by "data points you've lived through."

"I can't see what the other 499 salespeople did, and I can't even remember what I did in March of last year. I make so many decisions that they become lost in memory," Thompson said. "If I had a software tool, an algorithm that could remind me of what I've learned, and reveal to me what everyone else has learned... I could look at the guidance coming out of the algorithm as a distilled and refined experience from myself and people just like me."

Of course, the big caveat here is that Thompson and Ganzarski are heavily vested in the analytics camp. Neither is an unbiased observer of the intuition-versus-algorithm debate.

In a December 2013 Harvard Business Review blog on intuition's role in a data-driven organization, analytics expert and author Tom Davenport argued that major big-data projects are often driven by intuition:

    Major big data projects to create new products and services are often driven by intuition as well. Google's self-driving car, for example, is described by its leaders as a big data project. Sebastian Thrun, a Google Fellow and Stanford professor, leads the project. He had an intuition that self-driving cars were possible well before all the necessary data, maps, and infrastructure were available. Motivated in part by the death of a friend in a traffic accident, he said in an interview that he formed a team to address the problem at Stanford without knowing what he was doing.

Fair enough, chalk one up for intuition. But in the enterprise, analytics will play an increasingly influential role in the intuitive process.

IBM, Microsoft, Oracle, and SAP are fighting to become your in-memory technology provider. Do you really need the speed? Get the digital In-Memory Databases issue of InformationWeek today.

Jeff Bertolucci is a technology journalist in Los Angeles who writes mostly for Kiplinger's Personal Finance, The Saturday Evening Post, and InformationWeek. View Full Bio

Comment  | 
Print  | 
More Insights
Comments
Oldest First  |  Newest First  |  Threaded View
Whoopty
100%
0%
Whoopty,
User Rank: Ninja
5/27/2014 | 12:11:09 PM
Turn your key sir
Of course human inventiveness is always going to trump analytics based intuition when it comes to looking to future trends and technology. Howver, when it comes to analysing a lot of things about customer behaviour, especially in the here and now rather than long term, I think data analytics probably has the edge.

Being a sci-fi fan though, I'm still keen to see a person involved in every decision. That's how we stop the machines taking over. 
GilAllouche
50%
50%
GilAllouche,
User Rank: Apprentice
5/27/2014 | 1:50:52 PM
It's about Synergy
There were several good points here. Big data analytics can't and shouldn't overtake human intuition. It's simply a tool that wasn't available before, and like any tool, there are leaders who use it well and those who don't. The key is for there to be a synergy between the intuition and the analytics.
Shane M. O'Neill
50%
50%
Shane M. O'Neill,
User Rank: Author
5/27/2014 | 5:23:48 PM
Help data help you
I hope human intuition always plays a major role in business decisions. Yet it's crazy to cast aside data analytics as secondary to school of hard knocks know-how. They work in tandem. Analytics tools indeed can "enable you to do things the human mind simply can't do." Use them to make better and more confident decisions. It's arrogant to think your intuition is always right.
Ariella
50%
50%
Ariella,
User Rank: Ninja
5/27/2014 | 6:43:21 PM
Re: Help data help you
@Shane good pint. It doesn't have to be either/or. We can use both. The trick is to know when to lean more toward one than the other. 
Alison_Diana
50%
50%
Alison_Diana,
User Rank: Author
5/28/2014 | 9:37:13 AM
Re: Help data help you
Humans have to know which questions to ask. Intuition (plus today's user-friendlier big data and analytics tools) empower people to ask unusual questions that provide insight in really fascinating, off-beat, or creative ways. That's what I find exciting about big data and one reason you're not going to see computers replace humans as those controlling big data.

It's easy enough to create a query about who has the most sales or whether rain affects retail in Seattle, but we've already seen some fascinating results when individuals wonder about indirect or direct correlations between the strangest things -- and find there is some cause and effect. Data is just that. It's how we use it that's important.
mokuri
50%
50%
mokuri,
User Rank: Strategist
5/28/2014 | 10:58:03 AM
Re: Help data help you
According to one company in this space, Razorsight, one of the biggest problems companies face in Big Data is their own guesswork on which questions to ask. Another challenge: not saving or formating data properly so they can get answers to their Qs down the road. To your point, though, analytics actually does play a role in helping companies know which questions to ask.
6 Tools to Protect Big Data
6 Tools to Protect Big Data
Most IT teams have their conventional databases covered in terms of security and business continuity. But as we enter the era of big data, Hadoop, and NoSQL, protection schemes need to evolve. In fact, big data could drive the next big security strategy shift.
Register for InformationWeek Newsletters
White Papers
Current Issue
InformationWeek Must Reads Oct. 21, 2014
InformationWeek's new Must Reads is a compendium of our best recent coverage of digital strategy. Learn why you should learn to embrace DevOps, how to avoid roadblocks for digital projects, what the five steps to API management are, and more.
Video
Slideshows
Twitter Feed
InformationWeek Radio
Archived InformationWeek Radio
A roundup of the top stories and trends on InformationWeek.com
Sponsored Live Streaming Video
Everything You've Been Told About Mobility Is Wrong
Attend this video symposium with Sean Wisdom, Global Director of Mobility Solutions, and learn about how you can harness powerful new products to mobilize your business potential.