Big Data Perceptions: Good, Bad And Ugly

Big data will cure the common cold! Eliminate head lice! End fear and loathing in lab mice!

We've all read the headlines about how big data -- however you choose to define it -- will make the world a better place. Indeed, gleaning insights from growing stockpiles of digital information is a worthy goal, and big data will certainly benefit humanity in many ways -- or at the very least generate gainful employment for armies of data scientists.

Then again, it's also clear that the Big Data Hype-0-Meter has moved into the red on more than a few occasions lately. And perhaps that explains why the latest projections for big data are all over the map -- good, bad, and befuddled.

Let's start with the good. Last week, U.K.-based research firm Ovum issued a news release stating that sentiment pertaining to big data vendors stayed upbeat last year, despite a choir of critics decrying the data-driven phenomenon as overhyped.

[ Company budgets reflect investments in social data analytics in 2013. Read more at Big Data Budgets: Marketers Plan Spending Spree. ]

Ovum analyzed tweets compiled by social data startup DataSift, which ranked mentions and sentiment of big data-related vendors and open source groups in 2012. Positive mentions of big data outnumbered negative comments by 3:1, Ovum said. For whatever reason, negative sentiment spiked in November when news broke of Hewlett-Packard's problem-plagued acquisition of software company Autonomy.

"The data provided by DataSift provides a good example of how social media mining provides a useful snapshot of popular thinking that supplements -- or replaces -- the traditional role of marketing focus groups," wrote Ovum principal analyst Tony Baer in a January 21 blog post.

Then there are the critics. Big data has taken its lumps in recent months, with some folks opining that proponents of the technology have gone too far and promised too much.

According to the Gartner Hype Cycle, where the research firm presents a graphical view of specific technologies' maturity and adoption, big data has reached its "peak of inflated expectations."

Next up: the fall.

"Last several weeks show that big data is falling into the trough of disillusionment," wrote Gartner research director Svetlana Sicular in a January 22 blog post.

Part of the problem, it seems, is that organizations are quickly learning that today's big data tools -- Hadoop in particular -- are very difficult to use. "Several days ago, a financial industry client told me that framing a right question to express a game-changing idea is extremely challenging: first, selecting a question from multiple candidates; second, breaking it down to many sub-questions; and third, answering even one of them reliably. It is hard," wrote Sicular.

So after optimism and pessimism, what's next? Confusion. Or so says a new survey from the Direct Marketing Association (DMA) and Neolane, a marketing technology provider. The survey of more than 250 marketers in a variety of industries shows that 60% of respondents don’t have -- or are unaware if their company has -- a big data strategy in place. In addition, more than 80% of respondents say they're "somewhat or not very prepared" to deal with new governance and compliance rules pertaining to big data.

One solution, albeit an expensive one, is to hire a team of big data gurus to manage an operation. But it's not for everyone. "While some organizations are planning to hire data scientists to manage customer data, not all have that luxury," said Neolane president Stephan Dietrich in a statement.

The bottom line is that big data is in its infancy, and organizations are still grappling with what the term means. Promises seldom match reality, of course. But once the hype dies down, organizations will still have to learn the best ways to manage the rapidly expanding volume, velocity and variety of information.

InformationWeek is conducting a survey on the state of database technology in the enterprise. Take our InformationWeek 2013 Database Technology Survey now. Survey ends Feb. 1.