Enterprise 2.0: Uncomfortable Truths About Big Data

"The world is one big data problem."

Andrew McAfee, principal research scientist at MIT and author of Race Against the Machine, said that's a sentiment he keeps hearing in recent meetings in Silicon Valley.

"There's a bit of arrogance in that, and a bit of truth as well," he told the Enterprise 2.0 Boston 2012 conference audience in a Wednesday morning keynote presentation.

Can you address any problem effectively once you have masses of data? Enterprise 2.0 and social technologies are feeding big data analysis, providing new and more personal data points. Embedded sensors in everything from athletic clothing to cars feed big data pools and research that was not possible before, as consultant and InformationWeek guest columnist Vinnie Mirchandani recently detailed. Big data analysis based on Hadoop platform tools is challenging traditional business intelligence wisdom, particularly in retail industries.

While enterprise use of big data analysis can solve tough business problems, some individuals may find big data analysis deeply unsettling to their own career prospects.

Here's the bad news, according to McAfee: Computers are getting smarter all the time. IBM's Watson computer proved it could beat the most talented humans at Jeopardy trivia questions, but that's really just the beginning, McAfee pointed out. He cited Narrative Science, a company that generates news prose from a computer algorithm, in effect replacing reporters, to write basic news stories. Computers have also shown in research studies that they can beat pathologists at reading slides to detect signs of cancer, McAfee said.

[ See our special report: Enterprise 2.0 Boston 2012.]

We were never all that good.

"We kind of come out on the losing end over and over again," McAfee said. Using big data techniques, algorithms can predict questions including how good this year's crop of Bordeaux wine will turn out and how the Supreme Court will decide pending cases, he noted. Big data analysis can outdo purchasing experts, who have spent years learning nuances of vendor strategies and contracts, he said.

In a group of 136 man-versus-machine studies that he examined, humans won in just eight cases. The kisser: This was before the era of big data, he said. The computers likely did not have enough data.

"I kind of see our robot overlords and computer overlords getting smarter and smarter," McAfee said. "Are we all thoroughly depressed about this?" he asked the audience, prompting wry laughs. However, he said, this does not mean we should fight this sort of change or smash machines in a Luddite-like response. Instead, he said, people need to race with the machines, even if they are playing new roles.

McAfee points to FoldIt, an online effort that uses crowdsourced puzzles to help crack questions around scientific research involving protein folding. The research will be targeted at medical problems including HIV, Alzheimer's, and cancer research that depend on new knowledge about human proteins. It turns out that people are still better at solving protein-folding puzzles, he said.

One piece of good news: Your chances of surviving a round of layoffs go up as your use of social goes up, an MIT colleague's research concluded, McAfee said.

But for global footwear brand Nike, social and consumer technologies have also driven home some uncomfortable truths about old IT models and the value of IT staffers, Nike's Art King told the Enterprise 2.0 audience. Nike's app deployment strategy to its internal customers is changing from push to pull, said King, who is global infrastructure architect lead and a futurist for Nike (with the colorful sneakers to prove it).

This move to pull acknowledges three realities, he said. First, users want to pick apps now, not have apps pushed from IT. This means IT recommends apps, instead of choosing them. And at Nike, users often look for an answer to a problem on the wiki, instead of calling IT support. Or, users look to people they know who are regarded as experts.

"We've observed a tribal effect, where friends take care of friends," King said. This means IT has less value, he observed.

Examples like this show how IT can become disconnected from business customers, King said. "We need more right brain thinking in our organization," he said.

New apps promise to inject social features across entire workflows, raising new problems for IT. In the new, all-digital Social Networking issue of InformationWeek, find out how companies are making social networking part of the way their employees work. Also in this issue: How to better manage your video data. (Free with registration.)