Five Ways To Fool A Twitter Sentiment Tool - InformationWeek

InformationWeek is part of the Informa Tech Division of Informa PLC

This site is operated by a business or businesses owned by Informa PLC and all copyright resides with them.Informa PLC's registered office is 5 Howick Place, London SW1P 1WG. Registered in England and Wales. Number 8860726.

IoT
IoT
Software // Information Management
Commentary
12/16/2010
01:54 PM
Seth Grimes
Seth Grimes
Commentary
Connect Directly
Twitter
RSS
E-Mail
50%
50%

Five Ways To Fool A Twitter Sentiment Tool

Do sentiment-analysis tools pass the accuracy test? Here are five tests along with results using freely available products.

Test two is a polysemous word (a word with multiple meanings) where one meaning is a sentiment indicator. Try "kind" with Tweetfeel and you'll see the issue: It uniformly keys on "kind," a sign of positive feelings, where "kind" very often is used in the sense of "type" or "variety."

How do you disambiguate usage to fix this confusion? One basic step is to look at surrounding words. "A kind," "the kind," and "what kind" point to likely "type" or "variety" use.

Test three is sentiment analysis of messages with multiple opinion holders. Try a search on words such as "said" or, on Twitter, "RT" ("retweet"). Here's an example of a tweet that was, indeed, misclassified:

"RT @ShayIzKilla: Im hating RT @ChocolateWast3d: Oxtails stew on deck.< #Oxtail wait deh where my plate at" Tough stuff, and I don't mean oxtail, which is tender if you cook it long enough. Here, the original poster implicitly likes oxtail stew given that he or she is about to eat some: "Oxtails stew on deck." The response "Im hating" is negative but not so strongly, the equivalent of "yuck." That response elicits another, a positive one, "#Oxtail wait deh where my plate at."

Folks, this is how people communicate on social networks; it's "natural language." If you claim to do sentiment analysis, you have to handle it. Send the tweet + RT-response + RT-response to a Twitter sentiment engine. The freebie toy tools may get tripped up by the language, and regardless, they likely won't distinguish the three opinion holders and their three opinions. They'll give an overall sentiment rating which, whether correct or incorrect, is wrong given that what seems a single message is really three.

Test four involves inability to correctly resolve sentiment object. Here's a tweet rated negative by a socialmention search on "oxtail":

"Damn. I could eat jerk & oxtail a few times per wk RT @IMSTAIN: Had Jerk Chicken Once...#Lowkey Threw Up RT (cont) http://tl.gd/2jra75" I searched on "oxtail," so I expect the sentiment rating to be for "oxtail," reflecting the text "I could eat jerk & oxtail a few times per wk" with the intensifier "Damn." (I added the poster's name to the search, in order to bring up the particular message I was looking for, only after observing the incorrect classification.) Instead, socialmention either incorrectly keyed on "Damn" as a sign of negative sentiment -- although any tool that analyzes slang-filled social media should know better -- or is fooled by the "Threw Up" associated with jerk chicken.

The key to passing tests three and four is the ability to break messages into appropriate chunks, which may be phrases, quoted strings, or retweets within a longer message. For example, the folks from Conversition say that "Tweetfeel is meant to measure a subset of data around nouns," which I take to mean that it does focus sentiment analyses on searched-on terms.

We welcome your comments on this topic on our social media channels, or [contact us directly] with questions about the site.
Previous
2 of 3
Next
Comment  | 
Print  | 
More Insights
InformationWeek Is Getting an Upgrade!

Find out more about our plans to improve the look, functionality, and performance of the InformationWeek site in the coming months.

News
Becoming a Self-Taught Cybersecurity Pro
Jessica Davis, Senior Editor, Enterprise Apps,  6/9/2021
News
Ancestry's DevOps Strategy to Control Its CI/CD Pipeline
Joao-Pierre S. Ruth, Senior Writer,  6/4/2021
Slideshows
IT Leadership: 10 Ways to Unleash Enterprise Innovation
Lisa Morgan, Freelance Writer,  6/8/2021
White Papers
Register for InformationWeek Newsletters
Video
Current Issue
Planning Your Digital Transformation Roadmap
Download this report to learn about the latest technologies and best practices or ensuring a successful transition from outdated business transformation tactics.
Slideshows
Flash Poll