Whether they're prepared to deal with it or not, even mid-size companies have to face big data, not for the benefits it could offer, but because nearly all corporate data has become "big," according to reports from Ventana Research, Forrester, and other analysts.
Stereotypically big data sets--defined as any data set too large for conventional databases or analytical tools--are inflated with status reports, error messages, activity logs, and other data from the "Internet of Things," according to a seminal report from McKinsey & Co.
Big data is certainly a game changer, but it's at a stage of development in which hype overwhelms reality, according to Dan Vesset, the IDC analyst who was lead author on a March study that predicted the market for big data products and services would grow at 40% a year through 2015.
[ Do you see big data as a pain or an opportunity? Read Does IT Really Care About Big Data?. ]
The rapid growth, though it seems sudden, is actually the result of nearly 30 years' worth of development in analytic software, data warehouse techniques, and best-practice guidelines on how to manage a business based on data rather than intuition, Vesset said.
Helping the growth along is the rush among major vendors to adapt non-big-data databases and software to handle big data, as well as "more than half a billion dollars in venture capital" funding a host of startups, Vesset said.
That doesn't mean either users or vendors really understand the market.
Users are eager to build on reorganizations made to take advantage of ERP, CRM, and other products designed to put data, rather than guesswork, behind critical business decisions. Few top-level executives, however, understand the changes necessary in a company's data-gathering and decision-making processes well enough to make big-data migrations a real priority.
Still, users push their organizations toward big data, even at companies that find little value in data from the Internet of Things, according to Forrester analyst Mike Gualtieri.
Status reports from the inanimate aren't necessary for an app or a decision to qualify as being driven by big data, Gualtieri said. The only thing needed to give small data the impact of big is that the data be demonstrably causative. That is, the data being analyzed has to have a direct cause-and-effect connection with the decision being made. Big data needs to be causative; needs skilled data analysts to build and manage it; needs good-quality predictive-analytic software to boil data down to recommendations; and needs a program, platform, or set of processes that put changes in place based on decisions driven by data, Gualtieri said.
It's good for mid-size companies that all those requirements are available to them as well as to larger companies, but that availability also puts mid-size companies in the position of having to move faster on big data than they'd like, just to keep from being left behind, according to Daniel Castro, senior analyst with Information Technology and Innovation Foundation (ITIF), a Washington, D.C.-area think tank.
Among the problems mid-sized companies have adapting to big data is the kind of non-standard data it includes, primarily unstructured text from end users. That includes commentary or feedback from social networks, location data from mobile devices, forum postings, and other user-generated sources of data.
That data, almost completely excluded from BI or other assistive decision-making products until recently, has become the primary means of customer interaction for most companies, according to McKinsey analyst James Manyika.
The McKinsey study contrasts the traditional reliance on transactional information as a source of intelligence on customers with what it called "exhaust data" within big-data datasets. Exhaust data, like metadata, is content created as a by-product of other online activity. Unlike metadata, exhaust data exists on its own rather than simply as a way to provide context for records stored in traditional relational databases.
