Who Is Paying For Big Data Projects?

There is an interesting contradiction in data from a new survey of open source big data developers, data specialists, and IT managers: Big data projects designed to give business managers better information with which to make critical decisions are moving forward in many companies, but neither the business units nor IT seem aware of the progress--or be willing to pay for it.

The survey, from open source Hadoop, data integration, and big data analytics firm Jaspersoft found that a surprisingly high 62% of in-house corporate developers are either planning or actively deploying systems to deliver big data analytic functions, whether the projects are labeled "big data" or not. Only a third have a formal budget or any other funding to help them do it, however.

"Given the maturity of the big data market in general, we were expecting it would all be skunk works--projects in a kind of bootstrap prototyping/research phase," said Mike Boyarski, director of product marketing for the company.

JasperSoft's results may have been skewed because many of those it surveyed are highly technical, open source developers accustomed to doing much of the fundamental development work themselves.

Many, for example, are getting ready to deliver big data analytics without purpose-built big data software. Instead they're adapting existing analytics or database management systems to "function at an extreme level," on very large, very diverse, very changeable data sets, Boyarski said.

They're also getting support from forward-thinking business-unit managers interested in the kind of insight deep analysis of customer behavior can provide, but who aren't prepared with the business cases, return-on-investment analyses, or user requirements that might get a big data project budget approved, Boyarski said.

"There is a general awareness about the technology among business managers, but that has not been completely translated into budgeted, planned, clearly identifiable plans to move forward with it," Boyarski said. "They do get that analyzing new data of different varieties and different velocities is important, but not what to do about that. So on the IT side, technologists are doing what it takes to move forward even without extra funding."

[ Don't get sidetracked from the start. See Big Data Projects: 6 Ways To Start Smart. ]

Big data is not yet considered a strategic asset or potentially game-changing advantage, largely because business managers haven't had as much time to learn about or see the advantages in big data as they did with cloud computing, BYOD, and other technologies that were driven more by business units than IT, according to Frank Gillett, VP and principal analyst at analyst company Forrester.

The odd thing is, the dichotomy between wanting the advantages of big data and the will to pay for it may continue for the next several years, despite projections from Forrester and other analyst companies showing sales of big data products will grow as much as 40% per year until 2015.

Business managers aren't opposed to big data, they just don't know enough about the difference between it and traditional analytics to do any long-term planning, according to yet another survey, this time from market researchers at TheInfoPro.

According to InfoPro results, 56% of midsize and larger companies have no plans for any projects involving big data after 2013, even though almost none said they would willingly do without its advantages.

Of 607 business-unit managers surveyed by Capgemini in June, 90% said they would have made at least one decision differently during the past year if they'd had better information about the question--the kind of data they might have gotten from big data analysis.

Even so, more than half of Capgemini's respondents said big data is not on their list of technologies likely to deliver strategic advantages to the corporation.

"We compared our results to some of the other surveys and got very similar results," Boyarski said of JasperSoft's search for answers about big data. "This still seems very much like a bottoms-up initiative," he said. He said grassroots support more often goes to technologies more likely to end up in the hands of end users, not large-scale analytics likely to be available or useful to only a specialized subset of any organization that uses them.