Big Data Projects: 6 Ways To Start Smart

There are three solid rules describing how to successfully introduce big data into an organization. Unfortunately, to paraphrase W. Somerset Maugham, no one knows what they are.

In literary fiction, that's not a terrible drawback. At least it wasn't for Maugham, who wrote classics such as Of Human Bondage and The Razor's Edge, apparently without knowing the performance, design, or user-experience requirements for either project.

Big data projects require more precision if they're to succeed in delivering analytic tools or frameworks with enough power to handle the volume, variety, and timeliness required to qualify as "big" data.

One problem facing organizations is the lack of consistent, useful best practice guides that define issues common to most organizations, according to Mike Boyarski, director of product marketing for business intelligence/big data software vendor Jaspersoft.

As a category of business data analysis, big data is so new, so ill-defined, and the ecosystem of tools so immature that the long list of best practice documents published by big data vendors offers few consistent recommendations.

"There seems to be a lot of uncertainty about the value proposition, the ROI of big data, and about the tools that would let people take advantage of it," Boyarski said, citing a survey of prospective big data managers and developers that Jaspersoft will release Tuesday.

[ Get more more advice on avoiding bit data pitfalls and risks. See 10 Big Data Migration Mistakes. ]

End user organizations are a lot farther along in the development of their own big data implementations than Boyarski expected, even as respondents complained they lack sufficient guidance to be comfortable with their own project implementation plans.

As was the case with cloud computing, business unit managers seem to be pushing big data through the project pipeline as they anticipate the potential benefit of more complete, more insightful analysis of customer behavior than they've had until now, according to a survey from market researcher CSO Insights, which specializes in analyzing the effectiveness of corporate sales efforts.

Only 16% of organizations responding to the survey have any big data capabilities, but 71% of managers expect that adding one would have a significant positive impact on sales, the survey showed.

Despite significant, sustained demand for its reputed capabilities, the market for big data analytics is so fragmented and filled with small players that vendors are still trying to work out their own strategies and positioning according to IDC analyst Dan Vesset.

Companies that produce or collect non-traditional data--social networks or Web behavioral data collection and analysis companies, for example--step on the toes of traditional BI and database companies, who are still considering whether to get into the data collection business, according to Vesset.

There are some consistent elements that have to be taken into account or changed to support a broader analytic mission, however, according to Ash Ashutosh, CEO of data management vendor Actifio.

At their most basic, the challenge of moving into big data begins with the process of storing, processing, and managing the new data. Cloud computing platforms, storage area networks, and other scale-out systems can deal with big data storage demands; servers installed as purpose-built data processors can help avoid bottlenecks, according to Ashutosh.

Consider these six steps before you start your next big data project.

1. Identify missing pieces, whether they be tools or data.
The big gap is in tools designed to collect, deduplicate, tag, and process new types of metadata, and that give big data the context and meaning that make it valuable, according an IDC report on big data migrations published in June 2011.

Databases of text messages, asset-management information, and other content generated by users with smartphones is made vastly more useful with the addition of location data, but few analysis or data management tools are equipped to collect data from smartphone GPS chips or combine it with existing data or databases so it can be analyzed coherently, the report said.