Startups Offer Tools To Simplify Big Data

Hot Big Data Tools, Startups Some software tools are clearly important, however. For example, software that can lock down some fields in a record while opening others to queries is a clear advance over current security, which generally allows a record to be either open or shut with no caveats or extra protections for those that are opened, Das said.

Among the hottest functions still to come is the ability to let non-specialists surf big data, run data visualizations that might show new trends, and get answers to their most pressing questions without requiring the help of a data scientist, according to Ted Cuzzillo, a decision support and business intelligence analyst who blogs at DataDoodle.

But perhaps the most urgent need, according to Das, is for tools that can manage, structure, and analyze unstructured data by applying metadata, cleaning up the language, and squeezing the content into fields that can be processed by analytic software.

Simply cleaning and structuring data isn't enough, however. A CIO Executive Board survey posted in March found that only half of all employees who work with data or analytics get any training in these areas at all. Of those who do, only half report that the training does any good. That means three quarters of the potential big data users in corporate America have had no useful training in what may turn into the most important part of their jobs.

With few users trained in complex tools, simplicity becomes far more important. This is true even for heavily cleaned and processed big data sets, because they still have to be sorted and sifted to find the subset of data that could contain the answers users are looking for, according to CIO Executive Board analyst Andrew Horne.

Even estimating how complex a tool--let alone which specific tool--they need to analyze a particular set of data is not an easy task for most knowledge workers, according to researchers at the University of Pennsylvania's Wharton School. These researchers wrote a paper and built a set of models designed to help non-specialists pick the right tool for the right problem, but to date the paper has not been published and the software is not funded.

Better Tools For Non-Specialists DataSift, which landed $7.2 million in venture funding in May, started out under the name Tweetmeme, searching for interesting conversational trends. The company then changed direction, devoting its complex data-filtering software to sorting social network and other unstructured data according to gender, location, or even opinion.

A tool from Karmasphere is designed to let users create a graphic representation of a big data set, run ad-hoc queries against it to find trends and patterns they think are significant, then post the results for colleagues.

A third startup, Domo, comes with $63 million in funding and a CEO who founded Omniture, one of the more successful online data analytics companies. Domo provides analytics as a cloud service that gives executives a data dashboard on which they can poke and prod data using almost any device. It's not designed specifically as a big data tool, however. Instead, its goal is to give business managers direct access to the data and business intelligence analytics they previously could get only through the IT department, which often meant the data had gotten stale by the time it was crunched.

WibiData, founded by another celebrity entrepreneur, offers a big data platform that combines the abilities of Hadoop, HBase, and Avro to collect, manage, manipulate, and analyze big data sets without having to go from platform to platform. It's not quite self-service big data building and management, but it comes as close as you'll find anywhere in the business today. (WibiData was founded in 2011 as ObiData, and recently changed its name.)

Startup MetaMarkets doesn't bother with structuring or filtering data. It is designed to look into thick flows of transactional data to detect subtle changes in traffic or trends, and to predict how those trends will change over time. Its strength, in fact, is in its predictive--not its current-- analytics.

Being able to tell the future is a valuable skill, of course. Metamarkets has gone through two rounds of funding so far, landing a total of $8.5 million to help with its own transactions. See the future of business technology at Interop New York Oct. 1-5. It's the best place to learn about next-generation technologies including cloud computing, BYOD, big data, and virtualization. End of Summer Discounts end Sept. 5. Save up to $800 on Interop New York Conference Passes with code WEYLBQNY07.