Despite its high potential for delivering short-term value by identifying customers' behavior or attitudes, however, big data's long-term value rests in the ability and willingness of both vendors and users to make heavily processed, manipulated big data stores open to tools other than those designed specifically for high-intensity analytics, according to David Newman, research VP at Gartner.
Big data is an imprecise term describing the amalgamation of many types of data from many sources--often into databases that push the upper boundaries of the hardware and software assigned to manage them.
The leading file-management/database software used to handle big data--Hadoop--is already open source, as are many of the tools purpose-built to add new functions.
The data sets themselves are a different story, however.
Building a useful base of big data is an extremely complex process that requires careful selection of data sources, parsing the available sources to pick only the data that is appropriate in date, content, and context, and an even more rigorous series of efforts to remove duplicate, corrupt, or inappropriate data, convert the remainder to a common format, and store the lot with a database manager able to handle the volume, variety, and occasional conflicts among insufficiently processed bits.
"There are obvious integration issues when you're taking data from server logs and social networks and other non-standard sources, especially human-generated content," according to Mike Boyarski, director of product marketing for big data tools vendor Jaspersoft. "You have to be able to cull through the data and not create changes based on your cull, and you've got to prove the data are still correct and relevant. You need more than just the ability to collect data cheaply."
Once all that work is done and the big data set has answered the questions at hand, however, both the culled data and the work that went into it are going to waste if the data can only be used for that one purpose.
The cost of big data makes the most sense when its architects are able to use publicly available APIs, data conversion utilities, or common data and query formats to pull in additional data, transfer culled and cleaned data to a data broker that can pay for the privilege, or give employees access to the data through existing analytics or business intelligence applications.
"There is a positive relationship between the openness of information goods (for example, code, data, content, and standards) and information services (for example, services that offer information goods, such as the Internet, Wikipedia, OpenStreetMap and GPS) and the size and diversity of the community sharing them," according to the Gartner report. "From the viewpoint of enterprise information architects, this is known as the information-sharing network effect: the business value of a data asset increases the more widely and easily it is shared."
The primary method Gartner analysts recommend for companies wanting to share big data datasets or answers is the open API--a set of programming interfaces based on either an API set made available to customers of an enterprise application vendor, or a set of interfaces developed specifically to open source corporate data projects.
"The challenge for organizations is to determine how best to use APIs and how an open data strategy should align with business priorities," Newman said.
One additional tip about making money from big data as well as simply "getting smart," as Gartner's report puts it: Big data projects are difficult and expensive, so it makes sense to choose tools based on cost as well as functionality.
"Teams should use low-cost, open-source tools in early pilots to demonstrate the feasibility of big data projects," according to an April big data report, which predicted the best-practice habits established by enterprise architects could be the best investments for any company leaping into big data.
Open source tools tend to be less expensive upfront, have quicker and more ambitious roadmaps, and reflect (much more closely than other indicators) the real needs of the developers that use them, Boyarski said.
They also tend to preserve the value of both data and applications in the long run specifically because they don't trap either one in enterprise applications that rely on proprietary or hard-to-use APIs and data formats to keep users loyal to a single product set, according to Boyarski.
The community of open source users is much larger than Jaspersoft anticipated--more than a quarter million potential customers have downloaded various big data connectors from Jaspersoft's site.
Open source is also more secure and, in the long run, more useful because the people using the tools are often the ones building new features, reports, search algorithms, or other additions that enhance the value of the original software and make it possible to move apps, data, and reports out of kludged-together big data analytics frameworks and into brand new, high-function analytics, he said.
"After putting in all that work, the last thing you want is to have something trapped where you can't move it," Boyarski said.
Google in the Enterprise SurveyThere's no doubt Google has made headway into businesses: Just 28 percent discourage or ban use of its productivity products, and 69 percent cite Google Apps' good or excellent mobility. But progress could still stall: 59 percent of nonusers distrust the security of Google's cloud. Its data privacy is an open question, and 37 percent worry about integration.
CIOs Get Smart About BIIT’s tried for years to simplify business intelligence efforts. Have visual analysis tools and Hadoop and NoSQL databases helped? Respondents to our 2014 InformationWeek Analytics, Business Intelligence, and Information Management Survey have a mixed outlook.
InformationWeek Tech Digest, Nov. 10, 2014Just 30% of respondents to our new survey say their companies are very or extremely effective at identifying critical data and analyzing it to make decisions, down from 42% in 2013. What gives?