Please Note: This project is not currently active. The content on this site is provided for reference and is not actively maintained.

Sentiment Analysis Milestone: More Than One Million Human Judgments

by June 27, 2011

judgment-shot We have developed a process, dubbed Pulse, to extract nuanced sentiment from social media, like Twitter. We recognized early on that tools weren’t available to adequately answer specific questions, such as: “What’s the mood about today’s weather?” or “What portion of Twitter authors who discuss global warming believe that it is occurring?” or “Did Apple or Google have a more favorable buzz during this year’s South-by-Southwest Interactive?” Specifically, we concluded that it was necessary to get humans involved in the process—especially for Twitter posts, or tweets, which are often cryptic and have meaning that might be missed by a computer algorithm.

So, we turned to crowdsourcing.

However, successfully leveraging the power of the crowd for our sentiment analyses required cultivating the crowd, which we have achieved by working with partner CrowdFlower. In short, CrowdFlower offers an approach where we can access various work channels (we have relied mostly on Amazon’s Mechanical Turk), yet do so by layering on a quality control filter. Specifically, we intersperse within jobs what CrowdFlower terms “gold” units—in our case, tweets for which we already know the sentiment.  Workers build trustworthiness scores by getting the gold units correct. If they miss a gold unit, they get some feedback from us that has been tailored to that unit, such as “This person is happy that their garden is getting rain, so this should be marked as a positive emotion about the weather.”

We have been running a lot of jobs through CrowdFlower, but only recently did I step back and add up the tweets processed. For more than 200,000 individual tweets, we have received more than 1,000,000 trusted, human judgments from the CrowdFlower workforce! I know our research team, who had to do a bunch of judgments early on as we worked out a viable strategy, are grateful that we could get help from the crowd.

cf-jobs

Leave a Reply