Software // Information Management
Commentary
12/16/2010
01:54 PM
Seth Grimes
Seth Grimes
Commentary
Connect Directly
Twitter
RSS
E-Mail
50%
50%

Five Ways To Fool A Twitter Sentiment Tool

Do sentiment-analysis tools pass the accuracy test? Here are five tests along with results using freely available products.

Test two is a polysemous word (a word with multiple meanings) where one meaning is a sentiment indicator. Try "kind" with Tweetfeel and you'll see the issue: It uniformly keys on "kind," a sign of positive feelings, where "kind" very often is used in the sense of "type" or "variety."

How do you disambiguate usage to fix this confusion? One basic step is to look at surrounding words. "A kind," "the kind," and "what kind" point to likely "type" or "variety" use.

Test three is sentiment analysis of messages with multiple opinion holders. Try a search on words such as "said" or, on Twitter, "RT" ("retweet"). Here's an example of a tweet that was, indeed, misclassified:

"RT @ShayIzKilla: Im hating RT @ChocolateWast3d: Oxtails stew on deck.< #Oxtail wait deh where my plate at" Tough stuff, and I don't mean oxtail, which is tender if you cook it long enough. Here, the original poster implicitly likes oxtail stew given that he or she is about to eat some: "Oxtails stew on deck." The response "Im hating" is negative but not so strongly, the equivalent of "yuck." That response elicits another, a positive one, "#Oxtail wait deh where my plate at."

Folks, this is how people communicate on social networks; it's "natural language." If you claim to do sentiment analysis, you have to handle it. Send the tweet + RT-response + RT-response to a Twitter sentiment engine. The freebie toy tools may get tripped up by the language, and regardless, they likely won't distinguish the three opinion holders and their three opinions. They'll give an overall sentiment rating which, whether correct or incorrect, is wrong given that what seems a single message is really three.

Test four involves inability to correctly resolve sentiment object. Here's a tweet rated negative by a socialmention search on "oxtail":

"Damn. I could eat jerk & oxtail a few times per wk RT @IMSTAIN: Had Jerk Chicken Once...#Lowkey Threw Up RT (cont) http://tl.gd/2jra75" I searched on "oxtail," so I expect the sentiment rating to be for "oxtail," reflecting the text "I could eat jerk & oxtail a few times per wk" with the intensifier "Damn." (I added the poster's name to the search, in order to bring up the particular message I was looking for, only after observing the incorrect classification.) Instead, socialmention either incorrectly keyed on "Damn" as a sign of negative sentiment -- although any tool that analyzes slang-filled social media should know better -- or is fooled by the "Threw Up" associated with jerk chicken.

The key to passing tests three and four is the ability to break messages into appropriate chunks, which may be phrases, quoted strings, or retweets within a longer message. For example, the folks from Conversition say that "Tweetfeel is meant to measure a subset of data around nouns," which I take to mean that it does focus sentiment analyses on searched-on terms.

Previous
2 of 3
Next
Comment  | 
Print  | 
More Insights
The Agile Archive
The Agile Archive
When it comes to managing data, don’t look at backup and archiving systems as burdens and cost centers. A well-designed archive can enhance data protection and restores, ease search and e-discovery efforts, and save money by intelligently moving data from expensive primary storage systems.
Register for InformationWeek Newsletters
White Papers
Current Issue
InformationWeek Tech Digest - July10, 2014
When selecting servers to support analytics, consider data center capacity, storage, and computational intensity.
Flash Poll
Video
Slideshows
Twitter Feed
InformationWeek Radio
Archived InformationWeek Radio
Join InformationWeek’s Lorna Garey and Mike Healey, president of Yeoman Technology Group, an engineering and research firm focused on maximizing technology investments, to discuss the right way to go digital.
Live Streaming Video
Everything You've Been Told About Mobility Is Wrong
Attend this video symposium with Sean Wisdom, Global Director of Mobility Solutions, and learn about how you can harness powerful new products to mobilize your business potential.