Few top-level executives understand the changes necessary in data-gathering and decision-making processes well enough to make big-data migrations a real priority.
Big Data Talent War: 10 Analytics Job Trends
(click image for larger view and for slideshow)
Big data has become more than the hot buzzword in a technology sector still adapting to the fundamental changes created by the last wave of disruptive technologies--virtualization, cloud computing, mobile computing, and BYOD consumerization.
Whether they're prepared to deal with it or not, even mid-size companies have to face big data, not for the benefits it could offer, but because nearly all corporate data has become "big," according to reports from Ventana Research, Forrester, and other analysts.
Stereotypically big data sets--defined as any data set too large for conventional databases or analytical tools--are inflated with status reports, error messages, activity logs, and other data from the "Internet of Things," according to a seminal report from McKinsey & Co.
The rapid growth, though it seems sudden, is actually the result of nearly 30 years' worth of development in analytic software, data warehouse techniques, and best-practice guidelines on how to manage a business based on data rather than intuition, Vesset said.
Helping the growth along is the rush among major vendors to adapt non-big-data databases and software to handle big data, as well as "more than half a billion dollars in venture capital" funding a host of startups, Vesset said.
That doesn't mean either users or vendors really understand the market.
Users are eager to build on reorganizations made to take advantage of ERP, CRM, and other products designed to put data, rather than guesswork, behind critical business decisions. Few top-level executives, however, understand the changes necessary in a company's data-gathering and decision-making processes well enough to make big-data migrations a real priority.
Still, users push their organizations toward big data, even at companies that find little value in data from the Internet of Things, according to Forrester analyst Mike Gualtieri.
Status reports from the inanimate aren't necessary for an app or a decision to qualify as being driven by big data, Gualtieri said. The only thing needed to give small data the impact of big is that the data be demonstrably causative. That is, the data being analyzed has to have a direct cause-and-effect connection with the decision being made. Big data needs to be causative; needs skilled data analysts to build and manage it; needs good-quality predictive-analytic software to boil data down to recommendations; and needs a program, platform, or set of processes that put changes in place based on decisions driven by data, Gualtieri said.
It's good for mid-size companies that all those requirements are available to them as well as to larger companies, but that availability also puts mid-size companies in the position of having to move faster on big data than they'd like, just to keep from being left behind, according to Daniel Castro, senior analyst with Information Technology and Innovation Foundation (ITIF), a Washington, D.C.-area think tank.
Among the problems mid-sized companies have adapting to big data is the kind of non-standard data it includes, primarily unstructured text from end users. That includes commentary or feedback from social networks, location data from mobile devices, forum postings, and other user-generated sources of data.
That data, almost completely excluded from BI or other assistive decision-making products until recently, has become the primary means of customer interaction for most companies, according to McKinsey analyst James Manyika.
The McKinsey study contrasts the traditional reliance on transactional information as a source of intelligence on customers with what it called "exhaust data" within big-data datasets. Exhaust data, like metadata, is content created as a by-product of other online activity. Unlike metadata, exhaust data exists on its own rather than simply as a way to provide context for records stored in traditional relational databases.
The Agile ArchiveWhen it comes to managing data, donít look at backup and archiving systems as burdens and cost centers. A well-designed archive can enhance data protection and restores, ease search and e-discovery efforts, and save money by intelligently moving data from expensive primary storage systems.
2014 Analytics, BI, and Information Management SurveyITís tried for years to simplify data analytics and business intelligence efforts. Have visual analysis tools and Hadoop and NoSQL databases helped? Respondents to our 2014 InformationWeek Analytics, Business Intelligence, and Information Management Survey have a mixed outlook.