How Topsy Tames Twitter's Big Data Fire Hose - InformationWeek

InformationWeek is part of the Informa Tech Division of Informa PLC

This site is operated by a business or businesses owned by Informa PLC and all copyright resides with them.Informa PLC's registered office is 5 Howick Place, London SW1P 1WG. Registered in England and Wales. Number 8860726.

IoT
IoT
Healthcare // Analytics

How Topsy Tames Twitter's Big Data Fire Hose

Twitter's Political Index is just a taste of things to come as sophisticated data analysis becomes standard business practice.

One of greatest potentials of big data is the ability to let users search and analyze streaming data in real time. A television network, for instance, could get immediate viewer feedback on a new show airing for the first time. And journalists would have an easier way to search social media sites for breaking events, identify credible news sources, and analyze historical content.

Real-time analysis of large data sets is already here, of course, but it's just the start of what's on the horizon. As social networks mature, the ability to quickly sift through past posts--to travel back in time and mine information from a particular point in history--will become invaluable to businesses, scholars, historians, and other users.

Topsy Labs, a 5-year-old data analytics firm based in San Francisco, is working with Twitter on a big data project that brings real-time analysis to the social network's endless stream of tweets. Two polling firms, The Mellman Group and North Star Opinion Research, are participating as well.

[ Is your enterprise ready to take on big data? Read more at Businesses 'Freak Out' Over Big Data. ]

Built to capture the "nuances of public opinion," the Twitter Political Index uses Topsy's search and analytics tools to sift through 400 million daily tweets to find those mentioning the two U.S. Presidential candidates. It's designed to capture public sentiment quickly--and perhaps even more accurately than traditional, Gallup-style opinion polls, according to the Twitter Blog.

How does Topsy do it?

"We've created very specific databases, systems, and processes from scratch to enable us to ingest huge amounts of authored content, meaning content that human beings produce," Topsy chief revenue officer Eddie Smith told InformationWeek in a phone interview.

The company used C and C++ to develop its own databases and related software. "We're not using third-party software; we're not pulling off huge Hadoop jobs to do what we do," Smith said. Topsy does augment its systems with open source NoSQL technology, however.

The company runs its own data center. "It's very large-scale, very unique in terms of the problem set that we're solving for, social data in particular," said Smith.

Topsy's customers use APIs to access the results of Topsy's Twitter data analyses, including tweets from a specific date and/or geographical region. Its competitors include Gnip, which also provides access to social media content via APIs.

"Customers plug in their APIs for any number of different metrics," Smith said. "We also have algorithms that go through and remove all noise, and bubble-up the most important content."

Topsy's technology could work with Facebook too, but the analytics firm isn't pursuing a partnership with the world's largest social network. "It's not something we're interested in," said Smith. "Our technology is designed to manage huge amounts of data, and Facebook doesn't have a comparable Twitter fire hose where you can just plug in and get all the data."

And there are strategic issues as well. "Quite frankly, there's a huge investment we've made to process all the Twitter data as we have," Smith said. "If we're going to go down that path with Facebook, we'd want to make sure that, from a business perspective, we've got their full cooperation to do so."

Topsy sees a growing market for its brand of real-time and historical analytics, as more businesses turn to social networks to learn what people are saying about their products, brand, and services. In fact, companies are starting to use social media measurements to adjust their content very quickly.

"It could be editorial content, or link placements on pages," Smith said. "We're starting to see, from a marketing and publishing perspective, organizations executing real-time content decisioning based on what people are saying on social networks."

See the future of business technology at Interop New York, Oct. 1-5. It's the best place to learn about next-generation technologies, including cloud computing, BYOD, big data, and virtualization. Register today with priority code YLBQNY02 and save up to $300 on passes with early-bird pricing.

We welcome your comments on this topic on our social media channels, or [contact us directly] with questions about the site.
Comment  | 
Print  | 
More Insights
Comments
Newest First  |  Oldest First  |  Threaded View
Ellis Booker
50%
50%
Ellis Booker,
User Rank: Moderator
8/7/2012 | 2:01:59 PM
re: How Topsy Tames Twitter's Big Data Fire Hose
An interesting aside in this story is this bit: "Topsy's technology could work with Facebook too, but the analytics firm isn't pursuing a partnership with the world's largest social network. "It's not something we're interested in," said Smith."

Smith goes on to say that Facebook doesn't offer the volume of data that Twitter does. And while Smith doesn't come out and say it, I get the sense that Facebook's API may be an issue, too.

For the political junkies out there: How important is Facebook to understanding voter sentiment?

Ellis Booker,
InformationWeek Community Editor
Slideshows
10 Ways to Transition Traditional IT Talent to Cloud Talent
Lisa Morgan, Freelance Writer,  11/23/2020
News
Top 10 Data and Analytics Trends for 2021
Jessica Davis, Senior Editor, Enterprise Apps,  11/13/2020
Commentary
Can Low Code Measure Up to Tomorrow's Programming Demands?
Joao-Pierre S. Ruth, Senior Writer,  11/16/2020
White Papers
Register for InformationWeek Newsletters
Video
Current Issue
Why Chatbots Are So Popular Right Now
In this IT Trend Report, you will learn more about why chatbots are gaining traction within businesses, particularly while a pandemic is impacting the world.
Slideshows
Flash Poll