How Topsy Tames Twitter's Big Data Fire Hose

One of greatest potentials of big data is the ability to let users search and analyze streaming data in real time. A television network, for instance, could get immediate viewer feedback on a new show airing for the first time. And journalists would have an easier way to search social media sites for breaking events, identify credible news sources, and analyze historical content.

Real-time analysis of large data sets is already here, of course, but it's just the start of what's on the horizon. As social networks mature, the ability to quickly sift through past posts--to travel back in time and mine information from a particular point in history--will become invaluable to businesses, scholars, historians, and other users.

Topsy Labs, a 5-year-old data analytics firm based in San Francisco, is working with Twitter on a big data project that brings real-time analysis to the social network's endless stream of tweets. Two polling firms, The Mellman Group and North Star Opinion Research, are participating as well.

[ Is your enterprise ready to take on big data? Read more at Businesses 'Freak Out' Over Big Data. ]

Built to capture the "nuances of public opinion," the Twitter Political Index uses Topsy's search and analytics tools to sift through 400 million daily tweets to find those mentioning the two U.S. Presidential candidates. It's designed to capture public sentiment quickly--and perhaps even more accurately than traditional, Gallup-style opinion polls, according to the Twitter Blog.

How does Topsy do it?

"We've created very specific databases, systems, and processes from scratch to enable us to ingest huge amounts of authored content, meaning content that human beings produce," Topsy chief revenue officer Eddie Smith told InformationWeek in a phone interview.

The company used C and C++ to develop its own databases and related software. "We're not using third-party software; we're not pulling off huge Hadoop jobs to do what we do," Smith said. Topsy does augment its systems with open source NoSQL technology, however.

The company runs its own data center. "It's very large-scale, very unique in terms of the problem set that we're solving for, social data in particular," said Smith.

Topsy's customers use APIs to access the results of Topsy's Twitter data analyses, including tweets from a specific date and/or geographical region. Its competitors include Gnip, which also provides access to social media content via APIs.

"Customers plug in their APIs for any number of different metrics," Smith said. "We also have algorithms that go through and remove all noise, and bubble-up the most important content."

Topsy's technology could work with Facebook too, but the analytics firm isn't pursuing a partnership with the world's largest social network. "It's not something we're interested in," said Smith. "Our technology is designed to manage huge amounts of data, and Facebook doesn't have a comparable Twitter fire hose where you can just plug in and get all the data."

And there are strategic issues as well. "Quite frankly, there's a huge investment we've made to process all the Twitter data as we have," Smith said. "If we're going to go down that path with Facebook, we'd want to make sure that, from a business perspective, we've got their full cooperation to do so."

Topsy sees a growing market for its brand of real-time and historical analytics, as more businesses turn to social networks to learn what people are saying about their products, brand, and services. In fact, companies are starting to use social media measurements to adjust their content very quickly.

"It could be editorial content, or link placements on pages," Smith said. "We're starting to see, from a marketing and publishing perspective, organizations executing real-time content decisioning based on what people are saying on social networks."

See the future of business technology at Interop New York, Oct. 1-5. It's the best place to learn about next-generation technologies, including cloud computing, BYOD, big data, and virtualization. Register today with priority code YLBQNY02 and save up to $300 on passes with early-bird pricing.