Software // Information Management
Commentary
12/16/2010
01:54 PM
Seth Grimes
Seth Grimes
Commentary
Connect Directly
Twitter
RSS
E-Mail
50%
50%

Five Ways To Fool A Twitter Sentiment Tool

Do sentiment-analysis tools pass the accuracy test? Here are five tests along with results using freely available products.

Test two is a polysemous word (a word with multiple meanings) where one meaning is a sentiment indicator. Try "kind" with Tweetfeel and you'll see the issue: It uniformly keys on "kind," a sign of positive feelings, where "kind" very often is used in the sense of "type" or "variety."

How do you disambiguate usage to fix this confusion? One basic step is to look at surrounding words. "A kind," "the kind," and "what kind" point to likely "type" or "variety" use.

Test three is sentiment analysis of messages with multiple opinion holders. Try a search on words such as "said" or, on Twitter, "RT" ("retweet"). Here's an example of a tweet that was, indeed, misclassified:

"RT @ShayIzKilla: Im hating RT @ChocolateWast3d: Oxtails stew on deck.< #Oxtail wait deh where my plate at" Tough stuff, and I don't mean oxtail, which is tender if you cook it long enough. Here, the original poster implicitly likes oxtail stew given that he or she is about to eat some: "Oxtails stew on deck." The response "Im hating" is negative but not so strongly, the equivalent of "yuck." That response elicits another, a positive one, "#Oxtail wait deh where my plate at."

Folks, this is how people communicate on social networks; it's "natural language." If you claim to do sentiment analysis, you have to handle it. Send the tweet + RT-response + RT-response to a Twitter sentiment engine. The freebie toy tools may get tripped up by the language, and regardless, they likely won't distinguish the three opinion holders and their three opinions. They'll give an overall sentiment rating which, whether correct or incorrect, is wrong given that what seems a single message is really three.

Test four involves inability to correctly resolve sentiment object. Here's a tweet rated negative by a socialmention search on "oxtail":

"Damn. I could eat jerk & oxtail a few times per wk RT @IMSTAIN: Had Jerk Chicken Once...#Lowkey Threw Up RT (cont) http://tl.gd/2jra75" I searched on "oxtail," so I expect the sentiment rating to be for "oxtail," reflecting the text "I could eat jerk & oxtail a few times per wk" with the intensifier "Damn." (I added the poster's name to the search, in order to bring up the particular message I was looking for, only after observing the incorrect classification.) Instead, socialmention either incorrectly keyed on "Damn" as a sign of negative sentiment -- although any tool that analyzes slang-filled social media should know better -- or is fooled by the "Threw Up" associated with jerk chicken.

The key to passing tests three and four is the ability to break messages into appropriate chunks, which may be phrases, quoted strings, or retweets within a longer message. For example, the folks from Conversition say that "Tweetfeel is meant to measure a subset of data around nouns," which I take to mean that it does focus sentiment analyses on searched-on terms.

Previous
2 of 3
Next
Comment  | 
Print  | 
More Insights
The Agile Archive
The Agile Archive
When it comes to managing data, donít look at backup and archiving systems as burdens and cost centers. A well-designed archive can enhance data protection and restores, ease search and e-discovery efforts, and save money by intelligently moving data from expensive primary storage systems.
Register for InformationWeek Newsletters
White Papers
Current Issue
InformationWeek Tech Digest, Dec. 9, 2014
Apps will make or break the tablet as a work device, but don't shortchange critical factors related to hardware, security, peripherals, and integration.
Video
Slideshows
Twitter Feed
InformationWeek Radio
Archived InformationWeek Radio
Join us for a roundup of the top stories on InformationWeek.com for the week of December 14, 2014. Be here for the show and for the incredible Friday Afternoon Conversation that runs beside the program.
Sponsored Live Streaming Video
Everything You've Been Told About Mobility Is Wrong
Attend this video symposium with Sean Wisdom, Global Director of Mobility Solutions, and learn about how you can harness powerful new products to mobilize your business potential.