Big Data Means Big Storage Choices

Big data can improve the operational efficiency of companies using it by as much as 26%, according to a report released this month from Capgemini North America. That's a huge leap that will grow even larger--to 41%--within three years, if opinions of the 600 C-level executives and senior IT people Capgemini surveyed for the report are to be believed.

Two thirds of respondents said big data will be an important factor in business decisions, and will accelerate decision-making processes that have been slowed by excessive, inefficiently managed data. Eighty-four percent of respondents said the goal is to analyze big data in real time and act on it immediately to keep on top of changes in the market.

So why hasn't big data taken over the market for customer-behavior analysis and marketing? It has, at least to the extent most companies can manage it, according to analysts. Big data, like cloud computing, is a technology reference only by fiat; there is no "big data" SKU an IT department can order to get into big data management. There's not even a common definition. Any data a CIO can manage that directly affects top-line revenue is data that is valuable, no matter what its size, said Forrester analyst Vanessa Alvarez.

[ Read Feds Face 'Big Data' Storage Challenge. ]

"Big data means big value," she said at the May Interop show in Las Vegas. The problem with big data isn't defining what it is; the problem is in keeping up with what you'd like to store, especially when most of the data that becomes "big" is unstructured text from outside the company--from blogs, wikis, surveys, social networks, and other sites as well as operational data coming in from intelligent monitors built into manufacturing and transportation systems, said Alvarez.

However valuable the insight from Big Data, every project comes with a major downside: the cost of "big storage".

Traditional databases don't write or process data fast enough to handle giant pools of data, which is why the open-source database Hadoop has become so popular, according to John Bantleman, CEO of big data database developer RainStor, in an article for Forbes. An average Hadoop cluster requires between 125 and 250 nodes and costs about a million dollars, Bantleman wrote. Data warehouses cost in the tens or hundreds of millions, so Hadoop delivers the goods at a huge discount. When you're talking about data sets such as the 200 petabytes Yahoo spreads across 50,000 network nodes, you get into real money.

In March, IDC released the first projection of the worldwide market for big data. It predicted the market would grow 40% per year--about seven times as fast as the rest of the IT industry. Most of that cost, or at least the biggest part, will come from infrastructure-investment-caliber storage projects that will drive spending in the storage market to growth rates above 61% through 2015, according to IDC analyst Benjamin Woo.

The data sets themselves are growing as well. Though most big data sets are not overly large yet, they are growing in size by an average of 60% per year or more, according to IDC. The result, according to a February Aberdeen Group report, is that many companies will have to double the volume of their data storage every 2.5 years just to keep up.

Data compression and deduplication can reduce the amount of storage required by almost a third and data tiering can reduce per-unit costs by putting data in low demand on low-cost media such as DVDs or tape. The most effective way large companies deal with out-of-control data growth, however, is with scale-out NAS deployments whose costs rise much more slowly than those of more sophisticated storage area networks, whose costs rise linearly with the volume of data stored, Aberdeen concluded.