Please Note: This project is not currently active. The content on this site is provided for reference and is not actively maintained.

Posts Tagged ‘Twitter’

Training the Cloud with the Crowd: Training A Google Prediction API Model Using CrowdFlower’s Workforce

by February 29, 2012

 

kcocco_twitter_data_google_prediction_api

Can a machine be taught to determine the sentiment of a Twitter message about weather?  With the data from over 1 million crowd sourced human judgements the goal was to use this data to train a predictive model and use this machine learning system to make judgements.  Below are the highlights from the research and development of a machine learning model in the cloud that predicts the sentiment of text regarding the weather.  The following are the major technologies used in this research:  Google Prediction APICrowdFlowerTwitter,  Google Maps.

The only person that can really determine the true sentiment of a tweet is the person who wrote it.  When the human crowd worker makes tweet sentiment judgements only 44% of the time do all 5 humans make the same judgement.  CrowdFlower’s crowd sourcing processes are great for managing the art and science of sentiment analysis.  You can scale up CrowdFlower’s number of crowd workers per record to increase accuracy, of course at a scaled up cost.

The results of this study show that when all 5 crowd workers agree on the sentiment of tweet the predictive model makes the same judgement 90% of the time.  When you take all tweets the CrowdFlower and Predictive model return the same judgement 71% of the time.  Both CrowdFlower and Google Predictions supplement rather than substitute each other.  As shown in this study, CrowdFlower can successfully be used to build a domain/niche specific data set to train a Google Prediciton model.  I see the power of integrating machine learning into  crowd sourcing systems like CrowdFlower.  CrowdFlower users could have the option of automatically training a predictive model as the crowd workers make their judgements.  CrowdFlower could continually monitor the models trending accuracy and then progressively include machine workers into the worker pool.  Once the model hit X accuracy you could have a majority of data stream routed to predictive judgments while continuing to feed a small percentage of data the crowd to refresh current topics and continually validate accuracy.  MTurk hits may only be pennies but Google Prediction ‘hits’ cost even less.

(more…)

More »

Capturing Mood About Daily Weather From Twitter Posts

by September 29, 2011

After considerable preparation, we’ve just launched a version of our interactive tool, Pulse. Using Pulse, users can explore feelings about the weather as expressed on Twitter.

We began the process by choosing a topic that would yield a substantial volume of discussion on Twitter as well as be of general interest. Once we settled on weather, we wrote a survey designed to gauge Twitter users’ sentiments about the topic. With the help of workers from the “crowd” accessed through CrowdFlower, we had tens of thousands of relevant tweets coded as to the expressed emotion about the weather. These results were then used to create an “instance” of the Pulse tool, which manifests as a map of the United States that at a glance reveals Twitter users’ sentiments about the weather in their region on a given day. (You can read more about the coding process here and our choice of weather as a topic here.)

For our launch of Pulse for weather, we chose to feature tweets published over a month beginning in late April, 2011, a period in which many extreme weather events occurred—the devastating tornado in Joplin, MO; widespread drought throughout the South; and flooding of the Mississippi River, among others. The image below is from May 25, three days following the Joplin tornado (jump to the interactive map here).

may-25-pulse

We gathered tweets from all 50 states as well as for about 50 metro areas. Here you can see a zoom up on several states centered on Missouri.

zoom-may-25-pulse

The interactive map tells part of the story, namely a state’s or city’s overall sentiment about the weather, while the content under the “Analysis” and “Events” tabs reveal some of the “why” behind this sentiment: what were some of the most notable weather events occurring on a given day? [Note: our "events" feature has a bug in it and is currently turned off. In the future, icons will show up on the map to highlight out-of-the-ordinary weather events, like outbreaks of tornadoes, persistent flooding or drought, etc.] To what extent did the weather deviate from normal conditions? Why were tweets from, say, the South, uniformly negative during a certain time? What was happening when we saw a single positive state amidst a region that was otherwise negative?

We hope that weather is just the beginning. We envision using the Pulse tool to visualize nationwide sentiments about more complex, nuanced topics in the future—a sample of emotions about gas prices is just around the corner, and see our preliminary work on opinions about global warming. For now, you can explore the Pulse tool here, and let us know what you think!

More »

Sentiment Analysis Milestone: More Than One Million Human Judgments

by June 27, 2011

judgment-shot We have developed a process, dubbed Pulse, to extract nuanced sentiment from social media, like Twitter. We recognized early on that tools weren’t available to adequately answer specific questions, such as: “What’s the mood about today’s weather?” or “What portion of Twitter authors who discuss global warming believe that it is occurring?” or “Did Apple or Google have a more favorable buzz during this year’s South-by-Southwest Interactive?” Specifically, we concluded that it was necessary to get humans involved in the process—especially for Twitter posts, or tweets, which are often cryptic and have meaning that might be missed by a computer algorithm.

So, we turned to crowdsourcing.

However, successfully leveraging the power of the crowd for our sentiment analyses required cultivating the crowd, which we have achieved by working with partner CrowdFlower. In short, CrowdFlower offers an approach where we can access various work channels (we have relied mostly on Amazon’s Mechanical Turk), yet do so by layering on a quality control filter. Specifically, we intersperse within jobs what CrowdFlower terms “gold” units—in our case, tweets for which we already know the sentiment.  Workers build trustworthiness scores by getting the gold units correct. If they miss a gold unit, they get some feedback from us that has been tailored to that unit, such as “This person is happy that their garden is getting rain, so this should be marked as a positive emotion about the weather.”

We have been running a lot of jobs through CrowdFlower, but only recently did I step back and add up the tweets processed. For more than 200,000 individual tweets, we have received more than 1,000,000 trusted, human judgments from the CrowdFlower workforce! I know our research team, who had to do a bunch of judgments early on as we worked out a viable strategy, are grateful that we could get help from the crowd.

cf-jobs More »

Sleuthing Out Questions about Hybrid Cars from Twitter

by May 20, 2011

It is hardly news that we are all paying a lot at to fill up our vehicles. As we prepare to launch a multi-week analysis of mood about gas prices (here’s the background on how we extract sentiment from tweets), I’m curious what questions people have that may have been sparked by high gas prices.

Questions around the topic of hybrid cars/vehicles seem like a good starting point, given that one of the key benefits of hybrids is the potential to cut down on fuel expenses. One goal could be to create something like this Wired flow chart that is designed to help people choose a social search site. Not sure yet what the starting question would be to draw in as many people as possible on the topic of hybrids, but I think it would need to be responsive to feeling pain at the pump. One can imagine an interactive flow chart that offered up explanatory videos at various decision points.

st_flowchart_social_f

So, what are people asking about hybrids on Twitter? Below is a sampling of what I observed from a quick search on the string “hybrids ?” (by the way, I’m impressed with Storify’s handy interface for creating this kind of graphic). (more…)

More »

Oil Companies’ Profits to Increase Greatly This Year; People’s Energy-Related Questions to Follow Suit.

by May 5, 2011

The rapid increase in oil prices should equate to the oil industry having its best year since 2008, as reported by Chris Kahn for AP (via ABC). Exxon Mobil Corp., Chevron Corp. and ConocoPhilips are expected to report a combined $18.2 billion in first quarter earnings — a 40% increase from last year and just shy of the $20.2 billion that they earned in the first three months of 2008.

An increase in consumption, the constriction of supply (e.g., Libya’s reserve access is currently limited), and also a weaker US dollar are all speculated to contribute to an increase in oil prices.

While some stand to benefit from the rise in oil prices (shareholders), businesses and consumers will feel the hurt as gasoline prices inflate. Increases in gas prices tend to have ripple effects, increasing the prices of transportation and any good or service that is reliant on transportation — bread, toiletries, DVD players, air plane tickets, etc.

The broad societal effect of an increase in oil prices is precisely what makes this issue of interest to Dialogue Earth.  This will undoubtedly augment expressed sentiment related to energy across social media platforms, such as Twitter. (more…)

More »

Just Around the Corner: A Longer-Running Pilot On Weather Emotions

by April 27, 2011

This week the weather in the U.S. has been pretty unusual. We set a record for rainfall here in the Twin Cities, which is really a footnote to the week compared to the violent extreme weather in the Southeast and beyond. While understanding how people are feeling about the weather day-to-day won’t change the weather, we see it as a great starting point for developing our Pulse system for tracking public opinion on issues discussed in the social media.

As a follow-on to our first weather pilot, we are gearing up to monitor mood about the daily weather across the U.S. for weeks at a time. In fact, we are just completing a run of about 8000 Twitter tweets through our “crowd-based sentiment engine” using the CrowdFlower platform. Once we have double-checked the results, we are set up now to collect tweets continuously, automatically send them over to CrowdFlower for sentiment judgments, have the judgments returned to our database automatically, and then publish the data on our interactive Pulse display. We expect to be analyzing several thousand tweets through CrowdFlower on a daily basis in order to create a detailed map of weather mood for the U.S. (see more here about our data sampling strategy). Look for more on this in the coming days. The image below is a sneak peek at our interactive platform, which our team has overhauled in recent weeks. It should prove to be a much-improved user experience!

Pulse social media analytics tool More »

Global Warming Chatter: A Hot Topic on Twitter?

by March 28, 2011

Some months ago, our research team developed a strategy for inferring opinions about global warming from Twitter for our Pulse platform. We were lucky to be asked last week if we could present such data for the next issue of Momentum, the award-winning publication of the University of Minnesota’s Institute on the Environment. Of course, like all of us on a deadline, they needed it “yesterday.”

Not to be deterred, we rapidly spun up our collection system to grab those Twitter tweets that included the keywords global warming, climate change, and #climate. For a six day period ending on 23 March, we collected about 7600 tweets that had some geo-location information associated with them. Based on our recent experience focused on weather mood (described in this post), and because we had already generated a good number of quality control units (as described here), we posted a major job on the CrowdFlower platform within a day of the request from the Momentum team. Here’s a snapshot of the results:

momentum_dropshadow_300dpi (more…)

More »

Teasing Out Weather Mood From Twitter Posts: A Pulse Pilot

by March 8, 2011

In choosing a topic to use as a test case for our Pulse social media analytics tool, we wanted to pick something that is broadly discussed. What better topic to start with than people’s mood about the weather? It is hard to escape having a few thoughts about the weather on a regular basis. Snow storms, sunny warm days, and heatwaves, to mention a few, cross party lines and ideological divides. Plus, people love to discuss the weather, so we figured there would be lots of chatter in the social media—and we haven’t been disappointed. Read more on our weather strategy here.

In this post, I describe our first demonstration of the Pulse platform to describe weather mood across the U.S. using 12,500 tweets collected on February 4th. While our process is a work in progress, there are several key steps: identifying and collecting useful social media posts, getting reliable judgments about the sentiment in these posts made by crowd-sourced workers, publishing the data on our Pulse platform, and finally, combining our sentiment data with external data sources to tease out a story about the drivers of the observed sentiment.

pulse2

(more…)

More »

In Search of Quality Control with Crowd-Based Sentiment Judgments

by March 4, 2011

In a previous post, I described our evolving approach for developing a question that can be addressed on our Pulse platform. We’ve also described previously why we think crowdsourcing is a smart way to get lots of judgments made about sentiment expressed in the social media. But, what about quality control? How can we maintain an acceptable level of quality control while relying on the crowd to make thousands and thousands of judgments?

Quality through known answers and feedback to workers. We were drawn to CrowdFlower because of their approach for ensuring quality control using what they call “gold”. In a typical “assignment” set up on the CrowdFlower platform, a worker needs to make judgments for a group, or assignment, of “units” (a unit in our case would be an individual Twitter tweet). Within every assignment, CrowdFlower includes a gold unit for which we have indicated the correct answer. By setting an assignment to include 15 tweets, it means that a worker will be presented with a gold unit within each new batch of 15 tweets. (more…)

More »

Preparing to Extract Weather Mood from Tweets

by March 3, 2011

weather-tweet1

Yep, it was cold this morning in the Twin Cities. I didn’t need Twitter to tell that. Yet, we can’t always assume that, just because it is cold, people are upset, or that because it is warm, people are happy about the weather. But, we believe tweets will reveal something quite interesting: how people’s emotions are indirectly affected by the weather. For example, are people happy to be inside watching a movie even though it is “super chilly” outside? Or happy that the it is raining because it will help the garden, even though they may not be eager to be out in the rain themselves?

weather-tweet2

Having set the stage for tackling the issue of weather mood on our Pulse platform, here I describe our process for developing weather as a Pulse topic. (more…)

More »

Why Weather Mood for Pulse?

by February 14, 2011

As we developed our interactive platform for the analysis of dialogue in the social media, we needed to identify a topic to start with. Specifically, we needed to identify a topic that would have a high volume of chatter on Twitter, be of general interest, and present a decent challenge for our research team. Plus, we wanted a topic that was likely to vary geographically, because Pulse is fundamentally a platform for examining trends in dialogue across geographies. (more…)

More »

A Journey to Understand Social Media Sentiment

by February 14, 2011

Brand Bowl 2011

Chrysler stood atop the final standing for Brand Bowl 2011.

On Super Bowl Sunday, 106.5 million viewers were watching the big game—the largest TV audience ever, according to Nielsen. Many tuned in to witness the Packers battle the Steelers; even more, I imagine, were watching to see emerging brand Groupon face off against fan-favorite Go Daddy and advertising stalwarts Pepsi, Doritos and Volkswagen.

Millions were simultaneously browsing the Web, monitoring game stats and their Super Bowl pools, and checking out the brands advertised on the TV spots. A much smaller group of advertising and social media junkies were simultaneously glued to “Brand Bowl 2011,” a venture between ad agency Mullen and social media monitor Radian6 to monitor and rank the sentiment of Twitter references of Super Bowl advertisers. (more…)

More »