Big Data Squeezes Legacy IT Spending: IDC

Big data really isn't new. Data has been getting bigger for years, and companies have been finding ways to glean insights from their growing stockpiles of data for years. But it seems that calling it by a new buzzword is what it took to move it front and center for high level executives. It worked for cloud computing, didn't it?

Big data analyses helped Xerox slash employee turnover in its call centers by revealing a set of criteria predicting how long a new employee would stay. The criteria were simple enough that the main part of Xerox call-center candidate interviews is now done by software, rather than humans, according to the Wall Street Journal.

Predictive analytics have also helped companies discover other incentives that could boost retention. The finding that pay rate isn't a key factor can save companies' money by reducing the average size of incentive raises, according to a study by HR consultancy Mercer, a unit of Marsh & McLennan. Most companies--95%--look so closely at pay rates that they have little attention to spare for other factors.

Predictive modeling--a classic big data function--is far more accurate at identifying the reasons employees leave, but only 43% of companies polled by Mercer use any predictive modeling at all. Methods Mercer describes as less sophisticated analyses are far more popular. For example, 95% of companies use benchmarks from analysts, professional associations, or other external sources, while 90% use internal benchmarks. Only 64% use mathematical models that simulate specific personal situations to predict individual decisions--as researchers in sociology and psychology often do.

[ Big data is all the rage. Executives Push Big Data Projects, Not Sure Why. ]

The ability to collect and analyze data is not new, but it is exploding due to increases in computing hardware, the capacity of high-volume database-management systems such as Hadoop, and analytics that came first from the open source community.

"Analytics is top of mind for corporations" these days, according to IDC BI analyst Dan Vessett.

The overall market for business analytics--of which big data is one segment--will grow to more than $57 billion by 2016. "Driven by the attention-grabbing headlines for big data over three decades of evolutionary and revolutionary developments in technology, the business analytics software market has crossed the chasm into the mainstream mass market," Vessett said.

In 2011, IDC's 2011 Digital Universe Study predicted sales of big data analytics would grow at nearly 40% per year through 2015. This is seven times the average speed of other products in the same broad category, according to a March report by Vessett and Benjamin Woo, VP of storage systems at IDC.

By comparison, the 6.8% per year overall that global IT spending will grow--according to IDC predictions--is thin soup.

Making it thinner is the popularity and sales growth of technologies IDC predicts will become the four pillars supporting the next major era of corporate computing: mobile devices, cloud computing, big data, and social networking.

Together the four, heavily consumerized technologies will account for as much as 20% of total IT spending, reshaping the IT universe by adding new devices while starving legacy systems, routing every dollar toward new technology rather than maintaining older systems, according to IDC's predictions.

As a concept, big data woke end users to the potential for marketers to nano-target ever-narrower slices of the consumer market, and define corporate strategies according to accurate measures of a business environment, rather than broad approximations, according to Woo.

Three quarters of companies with more than 500 employees plan to invest in business analytics during the next 12 months, according to IDC, but very few seem to understand why. Only 38% said unequivocally that previous BI investments had paid off, while 15% said it hadn't, and 11% didn't know.

Thirty-six percent said they don't know how to measure the benefits of BI, while 48% said they couldn't even guess how long it would take a BI project to pay back the cost of using it.

Some of the reason is the fault of traditional methods and products, which were too inflexible, too limited in the data they could consider, and too lacking in both detailed analysis and predictive abilities to make their benefits clear.