Healthcare // Analytics
News
8/6/2012
11:44 AM
Connect Directly
Google+
RSS
E-Mail
50%
50%

How Topsy Tames Twitter's Big Data Fire Hose

Twitter's Political Index is just a taste of things to come as sophisticated data analysis becomes standard business practice.

One of greatest potentials of big data is the ability to let users search and analyze streaming data in real time. A television network, for instance, could get immediate viewer feedback on a new show airing for the first time. And journalists would have an easier way to search social media sites for breaking events, identify credible news sources, and analyze historical content.

Real-time analysis of large data sets is already here, of course, but it's just the start of what's on the horizon. As social networks mature, the ability to quickly sift through past posts--to travel back in time and mine information from a particular point in history--will become invaluable to businesses, scholars, historians, and other users.

Topsy Labs, a 5-year-old data analytics firm based in San Francisco, is working with Twitter on a big data project that brings real-time analysis to the social network's endless stream of tweets. Two polling firms, The Mellman Group and North Star Opinion Research, are participating as well.

[ Is your enterprise ready to take on big data? Read more at Businesses 'Freak Out' Over Big Data. ]

Built to capture the "nuances of public opinion," the Twitter Political Index uses Topsy's search and analytics tools to sift through 400 million daily tweets to find those mentioning the two U.S. Presidential candidates. It's designed to capture public sentiment quickly--and perhaps even more accurately than traditional, Gallup-style opinion polls, according to the Twitter Blog.

How does Topsy do it?

"We've created very specific databases, systems, and processes from scratch to enable us to ingest huge amounts of authored content, meaning content that human beings produce," Topsy chief revenue officer Eddie Smith told InformationWeek in a phone interview.

The company used C and C++ to develop its own databases and related software. "We're not using third-party software; we're not pulling off huge Hadoop jobs to do what we do," Smith said. Topsy does augment its systems with open source NoSQL technology, however.

The company runs its own data center. "It's very large-scale, very unique in terms of the problem set that we're solving for, social data in particular," said Smith.

Topsy's customers use APIs to access the results of Topsy's Twitter data analyses, including tweets from a specific date and/or geographical region. Its competitors include Gnip, which also provides access to social media content via APIs.

"Customers plug in their APIs for any number of different metrics," Smith said. "We also have algorithms that go through and remove all noise, and bubble-up the most important content."

Topsy's technology could work with Facebook too, but the analytics firm isn't pursuing a partnership with the world's largest social network. "It's not something we're interested in," said Smith. "Our technology is designed to manage huge amounts of data, and Facebook doesn't have a comparable Twitter fire hose where you can just plug in and get all the data."

And there are strategic issues as well. "Quite frankly, there's a huge investment we've made to process all the Twitter data as we have," Smith said. "If we're going to go down that path with Facebook, we'd want to make sure that, from a business perspective, we've got their full cooperation to do so."

Topsy sees a growing market for its brand of real-time and historical analytics, as more businesses turn to social networks to learn what people are saying about their products, brand, and services. In fact, companies are starting to use social media measurements to adjust their content very quickly.

"It could be editorial content, or link placements on pages," Smith said. "We're starting to see, from a marketing and publishing perspective, organizations executing real-time content decisioning based on what people are saying on social networks."

See the future of business technology at Interop New York, Oct. 1-5. It's the best place to learn about next-generation technologies, including cloud computing, BYOD, big data, and virtualization. Register today with priority code YLBQNY02 and save up to $300 on passes with early-bird pricing.

Comment  | 
Print  | 
More Insights
Comments
Newest First  |  Oldest First  |  Threaded View
Ellis Booker
50%
50%
Ellis Booker,
User Rank: Strategist
8/7/2012 | 2:01:59 PM
re: How Topsy Tames Twitter's Big Data Fire Hose
An interesting aside in this story is this bit: "Topsy's technology could work with Facebook too, but the analytics firm isn't pursuing a partnership with the world's largest social network. "It's not something we're interested in," said Smith."

Smith goes on to say that Facebook doesn't offer the volume of data that Twitter does. And while Smith doesn't come out and say it, I get the sense that Facebook's API may be an issue, too.

For the political junkies out there: How important is Facebook to understanding voter sentiment?

Ellis Booker,
InformationWeek Community Editor
Big Love for Big Data? The Remedy for Healthcare Quality Improvements
Big Love for Big Data? The Remedy for Healthcare Quality Improvements
Healthcare data is nothing new, but yet, why do healthcare improvements from quantifiable data seem almost rare today? Healthcare administrators have a wealth of data accessible to them but aren't sure how much of that data is usable or even correct.
Register for InformationWeek Newsletters
White Papers
Current Issue
InformationWeek Tech Digest - September 17, 2014
It doesn't matter whether your e-commerce D-Day is Black Friday, tax day, or some random Thursday when a post goes viral. Your websites need to be ready.
Flash Poll
Video
Slideshows
Twitter Feed
InformationWeek Radio
Sponsored Live Streaming Video
Everything You've Been Told About Mobility Is Wrong
Attend this video symposium with Sean Wisdom, Global Director of Mobility Solutions, and learn about how you can harness powerful new products to mobilize your business potential.