Expert Analysis: Is Sentiment Analysis an 80% Solution?

Sentiment-analysis technologies aren't perfect. But what critics are missing is the value of automation, the inaccuracy of human assessment, and the many applications that require only "good-enough" accuracy.
Applications Beyond Monitoring

"Sentiment is nice to know, but up until today I haven't heard a single commercial application for it," writes Mikko Kotila. He's like Miranda in Shakespeare's Tempest, who grows up on an island with only her father, Prospero, for human company. Miranda sees survivors of a shipwreck and exclaims, "O brave new world / That has such people in't!" Prospero's response: "'Tis new to thee."

There are many sentiment-analytics applications with proven ROI; others will prove themselves soon if they haven't already. You just need to look beyond social/media monitoring and measurement to business analytics and operations, to functions that such as customer experience management. I'll cite just a few examples, provided by vendors. These examples are focused, but they're representative of experience shared by many users.

First, ROI isn't always measured in dollar terms. Clarabridge reports that for customer Gaylord Hotels, "Automated analysis of survey comments showed that customer experience was measurably enhanced when bell services staff accompanied lost guests to their destinations within a resort, as opposed to merely pointing them in the direction they needed to go. This insight led management to incorporate the process improvement into what Gaylord calls the 'Service Basics.'"

Other commercial applications range from innovations such as an "emotional tone checker" from Lymbix, which targets corporate communications functions, and a similar application from Adaptive Semantics that automates comment moderation for user-generated, online content; to use by consumer-goods companies such as Unilever for marketing-campaign analysis, as I describe in a 2008 article, Sentiment Analysis: A Focus on Applications.

As I say, there are many applications of sentiment analysis that are already delivering business value. For those many, disparate applications, how accurate is accurate enough? Requirements depend on business needs, and "good enough" is a very valid concept. If automated sentiment analysis can help spot one dissatisfied customer who would otherwise have been missed, that's a start. There's always room for improvement, however, and higher accuracy is certainly on the to-do list, for researchers and vendors as hinted at above, and also for practitioners who apply the technology.

Improving Accuracy

My 2008 article describes how consultancy Anderson Analytics applied text technologies as one of three parts of a "triangulation" process seeking to understand consumer sentiment surrounding the Unilever's Dove-brand pro.age marketing campaign. It's a truism, common wisdom, that you can usually boost analytical accuracy by applying multiple methods. You can link text-extracted information to transactional and operational records, for instance, to determine what service center staff our French Toshiba critic interacted with, or for a survey, you can collect sentiment both using numerical rating scales and free text. And of course, you can also explore steps to improve accuracy with particular techniques.

"Coming to a Theater Near You! Sentiment Classification Techniques Using SAS Text Miner," a paper by Jake Bartlett and Russ Albright of SAS Institute, describes the authors' attempt to improve accuracy in sentiment analysis of a collection of movie reviews. They looked at four variations on out-of-the-box use of SAS Text Miner and found they could reduce misclassifications by up to one-third, to 14.4%. Now of course, you don't just throw techniques at problems and hope they will help. Bartlett and Albright cite weightings, synonyms, and part-of-speech tagging and more complex extraction and feature manipulation techniques, but they suggest that part-of-speech tagging may not be effective if you don't have well-formed, grammatical text.

Learning More

I've presented what I hope is a systematic, balanced appraisal of the state of the accuracy and usefulness of automated sentiment analysis. If you'd like to learn more, there are a number of places you can continue. Check out:

Finally, a plug: The Sentiment Analysis Symposium, April 13, in New York, will look at a broad set of sentiment-analysis approaches, business applications, solutions, and benefits. Please consider joining us for the conference.
Editor's Choice
Sara Peters, Editor-in-Chief, InformationWeek / Network Computing
John Edwards, Technology Journalist & Author
John Edwards, Technology Journalist & Author
James M. Connolly, Contributing Editor and Writer