Good analysis and decision-making depend on the quality of data, not quantity, Kanter said.
Without tools or expertise to eliminate minutiae or select only relevant classes of information, the only alternative most companies would have is to hire forensic accountants or other data experts to spend hours or days filtering big data by hand--a process far more expensive, imprecise, and un-repeatable than using software designed to do the same thing in a fraction of the time, without the risk that perceptual biases will warp the analysis.
Fortunately, according to venture capitalist, author, and tech-entrepreneur Dave Feinleib, there is a rash of big data apps on the way to do just that. Some are designed to gather and analyze data generated by machines; others are designed to collect and contextualize data from Facebook and other social networks.
The fastest-growing categories, however, are those designed for specific vertical markets such as healthcare and finance, and those aimed at collecting and analyzing data for specific departments within the organization, such as marketing, IT, and customer service.
Though many are traditional on-site apps, most are appearing as online apps built on cloud platforms with the computing power to analyze huge data sets as well as the algorithms to find useful knowledge within masses of data, according to Raj De Datta, CEO of BloomReach.
"BDAs [big-data apps] don't just repackage your data in a cool interface or offer productivity improvements in data scalability, they harness the world's data to deliver you a better outcome--like more revenue," according to De Datta.
BDAs, publicly available at subscription prices rather than the full cost of licensing and implementing sophisticated analytics on-premise, could make big-data analysis available to vast numbers of mid-sized companies that lack the will or expertise to build their own tools.
In business it's almost always better to make informed decisions using hard data than to use experience and intuition, but most companies have only begun to shift toward data-driven decision-making, according to a study released last month by Capgemini.
Both enterprises and mid-sized companies have difficulty making that move, Capgemini found.
Of 607 business managers, 90% said they would have made at least one major decision differently during the past three years if they'd had access to better data.
Nevertheless, more than half of respondents said big data is not considered strategic at the highest levels of their companies, despite the quality of decisions that can be made based on it.
That doesn't mean top executives aren't looking for better data with which to make better decisions, according to Capgemini's South East Asia CEO Deepak Nangia.
It only means that the organization's data-handling practices are not prepped to give decision makers the data they need when they need it to make better decisions, which would lead them to see decisions based on big data--and therefore big data itself--as a strategic advantage worth pursuing, Nangia said.
The challenge for big-data advocates--as it is with many promising but early-stage technologies--is to demonstrate quickly the benefit of building complex data sets and analytics to help their companies make better decisions, and get top managers to agree to make the investment even without the heavily analyzed data that might help push them in the right direction, Castro said.
"There's still a lot of convincing to be done," he said.
