Please Note: This project is not currently active. The content on this site is provided for reference and is not actively maintained.

Posts Tagged ‘crowdsourcing’

Soliciting Questions from the Crowd, Looking for the Crowd to Rank Questions

by April 2, 2012

process-sketch-v01 Over the past few weeks, we’ve been preparing to launch the EarthQ project. Modeled after Spot.us, which was recently acquired by American Public Media, we’re soliciting questions for which people would like to see high-quality, evergreen content developed (see some example answers here). In order for a question to move up the queue, or have its EarthQ rank increased, it needs to be shared a lot through Twitter, Facebook, and other social media. We’re doing this because it will be a substantial research effort to develop the high-quality answers, and we want to ensure that the question is of interest broadly.

Then, just like Spot.us raises funds to support the reporting on a particular topic, we’ll be looking for donations to support our research team. We’ll be aiming to raise micro-donations in 99 cent increments.

Head over to the Questions page to share (rank) questions and feel free to click the Amazon donate button to throw your 99 cents behind a question.

Over the past few days, these questions have been submitted via the website:

We welcome your feedback and ideas on this new project.

More »

Training the Cloud with the Crowd: Training A Google Prediction API Model Using CrowdFlower’s Workforce

by February 29, 2012

 

kcocco_twitter_data_google_prediction_api

Can a machine be taught to determine the sentiment of a Twitter message about weather?  With the data from over 1 million crowd sourced human judgements the goal was to use this data to train a predictive model and use this machine learning system to make judgements.  Below are the highlights from the research and development of a machine learning model in the cloud that predicts the sentiment of text regarding the weather.  The following are the major technologies used in this research:  Google Prediction APICrowdFlowerTwitter,  Google Maps.

The only person that can really determine the true sentiment of a tweet is the person who wrote it.  When the human crowd worker makes tweet sentiment judgements only 44% of the time do all 5 humans make the same judgement.  CrowdFlower’s crowd sourcing processes are great for managing the art and science of sentiment analysis.  You can scale up CrowdFlower’s number of crowd workers per record to increase accuracy, of course at a scaled up cost.

The results of this study show that when all 5 crowd workers agree on the sentiment of tweet the predictive model makes the same judgement 90% of the time.  When you take all tweets the CrowdFlower and Predictive model return the same judgement 71% of the time.  Both CrowdFlower and Google Predictions supplement rather than substitute each other.  As shown in this study, CrowdFlower can successfully be used to build a domain/niche specific data set to train a Google Prediciton model.  I see the power of integrating machine learning into  crowd sourcing systems like CrowdFlower.  CrowdFlower users could have the option of automatically training a predictive model as the crowd workers make their judgements.  CrowdFlower could continually monitor the models trending accuracy and then progressively include machine workers into the worker pool.  Once the model hit X accuracy you could have a majority of data stream routed to predictive judgments while continuing to feed a small percentage of data the crowd to refresh current topics and continually validate accuracy.  MTurk hits may only be pennies but Google Prediction ‘hits’ cost even less.

(more…)

More »

Tracking the Mood About Gas Prices on Twitter: A Case Study

by January 25, 2012

As another test of our strategy for teasing out public opinion from social media, we explored measuring mood about gas prices on Twitter. This post summarizes the findings from this case study. Incidentally, we are set up to measure mood from Twitter on an ongoing basis, although we would need to find a partner to help defray the ongoing costs of crowdsourcing the sentiment judgments. (See this post to read more about our decision to examine the discussion about gas prices on Twitter.)

The sentiment we mapped was culled from tweets gathered from four weeks’ worth of data starting on May 22nd, 2011. This time period was chosen to coincide with Memorial Day, a holiday during which many Americans travel by car. Our team was curious to see whether there would be an uptick in either the volume of tweets about gas prices during this period or a noticeable change in sentiment about these prices. (more…)

More »

Bringing the Global Carbon Cycle to Quora

by December 3, 2011

In a recent post about building a knowledge base for general consumption on the topic of ocean acidification, I suggested that it would be wise to step back and address questions-and-answers about the carbon cycle. This is a cross-posting of a post I just put up on Quora: The Carbon Cycle, Starting in.

And, the first question is also up on Quora: What are the compartments in which carbon is stored on Earth? Do you have the credentials to answer this, or do you know someone who does? Please help out—I believe it will be a rewarding way to bridge the gap between the science community and the rest of society. Also, keep in mind that “upvoting,” adding comments for the author, or suggesting edits,  as soon as there is one or more answers will be a really great way to participate.

For anyone new to Quora, I’d encourage you to take a look at a new infographic and video that helps explain how to have a winning strategy within the Quora community.

Any and all feedback is welcome!

carbon-cycle-01_lower_res More »

How Do I Go About Answering A Question on Quora?

by December 3, 2011

As a follow-up to a first infographic designed to help newcomers make sense of Quora, a relatively new question-and-answer site on the Web, this new infographic targets those who are new and want to dive in and begin answering questions. It is posted as an answer to How do you answer a question on Quora? There already are some great resources on Quora for those who are getting started, such as Lucretia Pruitt’s post “Welcome to Quora. Do Yourself a Favor & Slow Down” (1663 upvotes, and counting) and What are the basic rules around using Quora?

As an experiment, I created a short (3 min) video that walks you through the infographic. Note that there are a bunch of links to Quora questions throughout the video—ensure annotations are turned on to see them. Feedback welcome!

And, the infographic…

new-answerer-infographic-sketch-v02

 

More »

So You’re On Quora, Now What?

by October 16, 2011

As we gear up to tackle some basic questions and answers on Quora about the carbon cycle as part of our EarthQ project, with a goal of soon being able to tackle issues like ocean acidification and global warming, it seemed that a roadmap to Quora would be key for collaborators who might never have hear of it. Hence, the idea of a Quora Infographic emerged late this past week. Here’s version 0.1 that I’m throwing out for feedback. What do you think? Helpful? Silly? Gaping holes?

infographic-sketch-v02_small

UPDATE: Now showing version 0.2, thanks to a comment from friend Mike Troiano to highlight the start arrow. Keep the comments coming!

Version 0.1
infographic-sketch-v01_small

See this also on my post on Quora. I’d appreciate you up-voting it there if you like the idea.

Creative Commons License
So you’re on Quora, now what? by Dialogue Earth is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 3.0 Unported License. Permissions beyond the scope of this license should be submitted here.

More »

Building A Knowledge Base About Ocean Acidification on Quora

by October 13, 2011

Some months ago, we did a proof-of-concept video contest on the subject of ocean acidification. Here’s the winning video, about a snail that doesn’t have a shell, goes to learn why, and then forms a band to spread the word about this phenomenon.

Our overall vision is that we’d only initiate video, or other multimedia, projects once we’d nailed down the “science behind the story.” Furthermore, to us, nailing down the story means that we’ve ferreted out those key points for which we can get widespread agreement from experts drawn from across society–those from industry, NGOs, government, as well as academics.

To be blunt, we skipped this (huge) step for ocean acidification for two reasons. We were trying to proof out a crowd-based process for creating videos that conveyed science points fairly. Also, I had a good bit of background on the issue from my graduate work in oceanography.

102__0x0_quora-picture Now that we’re aiming to cultivate Q&As on Quora on topics related to the environment, it seems only logical to revisit the topic of ocean acidification. We’re dubbing this Quora-based project EarthQ (see also this post about putting candidates on the spot about global warming). For reference, see this post I did on the science behind the story of ocean acidification.

The goal of EarthQ will be to identify the top questions about various issues people are likely to ask about ocean acidification–assuming we can shepherd solid answers that are endorsed by a wide spectrum of experts. Taking this approach ties into Anticipating Questions People Will Ask About a Topic. Of course, if we’re successful, we will have created a great network of experts ready to jump to assistance as new questions emerge organically on Quora.

I’m a big advocate of working with building blocks on an issue. Having given this a good deal of thought recently, it is clear to me that step one will be to develop Q&A that fully explain the carbon cycle. Then, we can move on to the topic of ocean acidification. Then we should have worked out enough of the kinks to take on the topic of global warming. Onward!

More »

Putting Candidates and Others on the Spot About Global Warming

by September 29, 2011

This opinion piece, written in response to a recent Star Tribune op-ed, was published on Yale’s Forum for Climate Change and the Media on September 29, 2011.

The presence, or perhaps more so the absence, of serious climate change dialogue in the run-up to the presidential elections could make the issue a volatile unknown: Few politicians in the national spotlight want to be caught holding strong opinions in favor of aggressive policies to slow, curb, or reverse anticipated climate change.

James Lenfestey, a former Star Tribune editorial writer on climate and education, recently argued in Minneapolis Star Tribune (op-ed 9/17) that journalists should drill presidential candidates on six questions related to climate change, presumably to highlight those who are silent, or openly hostile, toward taking action in the face of this global threat.

Commentary

The media clearly can do a better job in raising serious issues like global warming with those leading, or aspiring to lead, our country. BUT putting candidates on the spot is unlikely to yield what our country desperately needs: a serious, ongoing dialogue about this global issue, which has connections to nearly every aspect of our society. Such a dialogue MUST be built on a foundation of trust and respect for political and ideological differences, as well as different scientific backgrounds.

Lenfestey’s six questions can be boiled down to these two: Do you believe global warming is occurring? And, do you believe we should have aggressive policies to counter this threat?

Lumping together opinions about how persuaded one is by the underlying science with questions about the appropriateness of a policy response is a recipe for an unproductive dialogue. Having spent a number of years working with experts drawn from across society to describe the condition and use of our ecosystems, I learned that it is essential to separate these aspects of the conversation—especially for the more contentious issues. In other words, we collectively need to find some common ground before arguing about appropriate policy responses.

We need to build an initiative that is viewed as trustworthy to all of us, no matter our ideological and political positions, or our understanding of science. WE can do that. We can run a transparent process—open to full scrutiny and input from anyone—that develops a series of questions and broadly accepted answers relevant to the issue of climate change. Not 100 questions, but rather more like 10. Each answer would be endorsed by experts drawn from across society and viewed as trustworthy by citizens and politicians from across the spectrum, both politically and ideologically.

As an example, these answers would undoubtedly include:

  • Carbon dioxide, a colorless gas present in tiny concentrations in the atmosphere, can trap radiation from the Earth that would otherwise dissipate into space.
  • There is strong evidence of a marked increase in carbon dioxide directly linked with the growth of industrialized societies across the globe.
  • Our “fingerprints” are detectable in the altered chemical nature of atmospheric carbon dioxide.
  • Long-term temperature trends indicate significant increases in recent decades.

These would be the first building blocks to a serious national dialogue.

Let’s take on this challenge. Let’s develop a robust, multimedia Web presence that brings to life the science behind the questions-and-answers for general audiences. Ideally, this effort would have its foundation on a public Q&A site, like Quora (http://quora.com).

Circling back to Lenfestey’s questions, let’s ask each candidate if he or she would support such an open, transparent process to create robust common ground across society. Global warming, which has the potential to irreversibly alter our world, demands a serious dialogue, and now is the time for to initiate it. Time is short, and there is a lot to do. Let’s make it happen.

Kent Cavender-Bares is an environmental scientist and the founder of Dialogue Earth, a nonprofit media project that has received major seed funding from the University of Minnesota’s Institute on the Environment and the Foundation for Environmental Research. The views represented are his alone and do not represent those of the University of Minnesota.

More »

Capturing Mood About Daily Weather From Twitter Posts

by September 29, 2011

After considerable preparation, we’ve just launched a version of our interactive tool, Pulse. Using Pulse, users can explore feelings about the weather as expressed on Twitter.

We began the process by choosing a topic that would yield a substantial volume of discussion on Twitter as well as be of general interest. Once we settled on weather, we wrote a survey designed to gauge Twitter users’ sentiments about the topic. With the help of workers from the “crowd” accessed through CrowdFlower, we had tens of thousands of relevant tweets coded as to the expressed emotion about the weather. These results were then used to create an “instance” of the Pulse tool, which manifests as a map of the United States that at a glance reveals Twitter users’ sentiments about the weather in their region on a given day. (You can read more about the coding process here and our choice of weather as a topic here.)

For our launch of Pulse for weather, we chose to feature tweets published over a month beginning in late April, 2011, a period in which many extreme weather events occurred—the devastating tornado in Joplin, MO; widespread drought throughout the South; and flooding of the Mississippi River, among others. The image below is from May 25, three days following the Joplin tornado (jump to the interactive map here).

may-25-pulse

We gathered tweets from all 50 states as well as for about 50 metro areas. Here you can see a zoom up on several states centered on Missouri.

zoom-may-25-pulse

The interactive map tells part of the story, namely a state’s or city’s overall sentiment about the weather, while the content under the “Analysis” and “Events” tabs reveal some of the “why” behind this sentiment: what were some of the most notable weather events occurring on a given day? [Note: our "events" feature has a bug in it and is currently turned off. In the future, icons will show up on the map to highlight out-of-the-ordinary weather events, like outbreaks of tornadoes, persistent flooding or drought, etc.] To what extent did the weather deviate from normal conditions? Why were tweets from, say, the South, uniformly negative during a certain time? What was happening when we saw a single positive state amidst a region that was otherwise negative?

We hope that weather is just the beginning. We envision using the Pulse tool to visualize nationwide sentiments about more complex, nuanced topics in the future—a sample of emotions about gas prices is just around the corner, and see our preliminary work on opinions about global warming. For now, you can explore the Pulse tool here, and let us know what you think!

More »

Could A Social Q&A Site Like Quora Serve As Home On the Web for Information About Global Warming?

by September 21, 2011

quora-picture Over the past few weeks, I have been giving a good bit of thought to this question. Quora, began as a vibrant Q&A site for the start-up community in Silicon Valley. Its creators, who were key technologists with Facebook, know how to build a good interface. Their goal is not modest:

“Quora is a continually improving collection of questions and answers created, edited, and organized by everyone who uses it. The most important thing is to have each question page become the best possible resource for someone who wants to know about the question.”

Some months back, I posed a still-unanswered question: Should Quora be used to gather and distill expert answers on polarizing topics? It is not at all clear if the answer should be “yes” or “no.” Perhaps, as a Quora admin put it in a response to another question of mine, the right strategy is to follow Wikipedia’s “be bold” policy, which is summed up in three words “Just do it.” In other words, it may be necessary just to start doing it and figure out a way to make it work. (more…)

More »

Sentiment Analysis Milestone: More Than One Million Human Judgments

by June 27, 2011

judgment-shot We have developed a process, dubbed Pulse, to extract nuanced sentiment from social media, like Twitter. We recognized early on that tools weren’t available to adequately answer specific questions, such as: “What’s the mood about today’s weather?” or “What portion of Twitter authors who discuss global warming believe that it is occurring?” or “Did Apple or Google have a more favorable buzz during this year’s South-by-Southwest Interactive?” Specifically, we concluded that it was necessary to get humans involved in the process—especially for Twitter posts, or tweets, which are often cryptic and have meaning that might be missed by a computer algorithm.

So, we turned to crowdsourcing.

However, successfully leveraging the power of the crowd for our sentiment analyses required cultivating the crowd, which we have achieved by working with partner CrowdFlower. In short, CrowdFlower offers an approach where we can access various work channels (we have relied mostly on Amazon’s Mechanical Turk), yet do so by layering on a quality control filter. Specifically, we intersperse within jobs what CrowdFlower terms “gold” units—in our case, tweets for which we already know the sentiment.  Workers build trustworthiness scores by getting the gold units correct. If they miss a gold unit, they get some feedback from us that has been tailored to that unit, such as “This person is happy that their garden is getting rain, so this should be marked as a positive emotion about the weather.”

We have been running a lot of jobs through CrowdFlower, but only recently did I step back and add up the tweets processed. For more than 200,000 individual tweets, we have received more than 1,000,000 trusted, human judgments from the CrowdFlower workforce! I know our research team, who had to do a bunch of judgments early on as we worked out a viable strategy, are grateful that we could get help from the crowd.

cf-jobs More »

Teasing Out Opinions About Global Warming From Twitter

by June 24, 2011

snapshot-ca A couple of months ago, we posted results from a quick sampling of mood about global warming in the Twittersphere that was featured in Momentum, the publication of the University of Minnesota’s Institute on the Environment. Along with our work on weather mood and mood about gas prices, we are on the verge of releasing a more in-depth analysis of sentiment about global warming. Here, we explain the method behind our sentiment analysis related to global warming, building off an earlier post that presented some of the details of our methodology on studying global warming chatter. (more…)

More »

Don Shelby: Climate Experts to Ratchet Up Language

by June 2, 2011

In his most recent article for MinnPost, Don Shelby describes a meeting he had with three climate science thought leaders about communicating their scientific findings to the public more effectively.  He describes a current message shift that many of them will undertake when fielding questions from reporters about global warming.

The example Shelby gives is when a reporter asks a climate scientist if current weather phenomena are due to global warming.  Rather than the typical, “no single event can be contributed to global warming,” the response will shift to, “no single event can be attributed to global warming, but we told you this was going to happen.”  The former statement has the potential to reinforce complacency by those who are skeptical of global warming or its predicted impacts.  Those supporting this new strategy, as Shelby explains, are hoping that adding a twist will reduce or eliminate some of this fodder for complacency.

(more…)

More »

Relevancy and Context are “Critical” with Sentiment Analysis

by May 24, 2011

September 11 Whenever I come across a piece that highlights how tricky sentiment analysis truly is, I tend to be encouraged more often than dissuaded to keep trying to figure it out.

Sentiment analysis is tough—not as in strict, like a teacher is tough, or in resilient, like a marathoner is tough. More like hard, like an AP calculus test is tough.  Not hard, like a block of concrete is hard.  Hard, as in difficult.  Eh, nevermind.

A colleague of mine just sent me a piece from the Miller-McCune site discussing a flawed mood study about September 11 pager text messages.

Researchers from Johannes Gutenberg University in Germany had concluded that there was an escalating level of “anger” words communicated to pagers as time passed on September 11 (here’s the study).  I’ve included the original data graph in this post. (more…)

More »

Sleuthing Out Questions about Hybrid Cars from Twitter

by May 20, 2011

It is hardly news that we are all paying a lot at to fill up our vehicles. As we prepare to launch a multi-week analysis of mood about gas prices (here’s the background on how we extract sentiment from tweets), I’m curious what questions people have that may have been sparked by high gas prices.

Questions around the topic of hybrid cars/vehicles seem like a good starting point, given that one of the key benefits of hybrids is the potential to cut down on fuel expenses. One goal could be to create something like this Wired flow chart that is designed to help people choose a social search site. Not sure yet what the starting question would be to draw in as many people as possible on the topic of hybrids, but I think it would need to be responsive to feeling pain at the pump. One can imagine an interactive flow chart that offered up explanatory videos at various decision points.

st_flowchart_social_f

So, what are people asking about hybrids on Twitter? Below is a sampling of what I observed from a quick search on the string “hybrids ?” (by the way, I’m impressed with Storify’s handy interface for creating this kind of graphic). (more…)

More »

YouTube’s Hot Spot Analyses: Changing Behavior Raises Questions

by May 18, 2011

A few days ago, I wrote a post describing how three of our videos on the topic of ocean acidification measured up using the hot spots tool within YouTube’s Insights dashboard. Returning to this today, I planned to look at another batch of videos. When doing some comparisons, I realized that the hot spots graph had changed rather dramatically for one of the videos, “New Neighbor.” Here’s the before taken on 5.16.11:

new-neighbor

Today, I found a considerably different hot spots trace:

new-neighbor-hot-spots-5-17-11 (more…)

More »

Hope for Human Sentiment Analysis Coding

by May 13, 2011

I just read an interesting blog post on Social Times discussing the advantages of machine-based sentiment analysis. In the piece, author Dr. Taras Zagibalov challenges the critics of “automatic” sentiment analysis, who claim that humans can better determine than computers the sentiment of social media text. He asserts that, with the proper tuning of a system’s classifier—creating specific classifiers for each domain (subject matter) and keeping them current—a machine-based sentiment analysis system can outperform human accuracy.

The discussion of human vs. machine sentiment is core to our work at Dialogue Earth, where we are developing Pulse—a social media analytics tool to help tease out nuances in the social media dialogue about key societal topics. Pulse social media analytics tool (more…)

More »

“Momentum” for Dialogue Earth

by May 11, 2011

We are thrilled that Dialogue Earth is featured in the most recent issue of Momentum magazine, an award-winning publication from the University of Minnesota’s Institute on the Environment.

momentum_dropshadow_300dpi While we work to optimize key aspects of our business—from the incentives we provide crowdsourced video creators, to the quality of the underlying data for Pulse, our social media analytics tool—we’re also rapdily ramping up our efforts to engage and broaden our base of supporters and collaborators.

Indeed, this Momentum feature piece comes at a great time for us.  There’s a ton going on.

Our Pulse tool is just about ready for prime time.  In a matter of weeks, we’ll have an version of Pulse that will provide daily information on the Twittersphere’s mood about the weather.  On the heels of that, we’ll be looking at Twitter chatter related to gas prices.

(more…)

More »

Incentives for Video Producers in Crowdsourced Contests: Setting the Stage for a Survey of Creators

by May 10, 2011

4047757987_4342baa844_m

Photo by Steve Garfield

In doing research for a forthcoming survey of creators who participate in video contests, I ran across this excellent piece posted by Josh Tabb on the Case Foundation blog.

The take-home for me is that a variety of incentives are key, ideally with some of them being offered on a regular basis—such as a daily viewers’ choice award. Also key is promotion in order to get sufficient participation. Tabb summarized insight he heard from Ramya Raghavan, who was at the time YouTube’s Nonprofits and Activism Manager: “it was incentives and promotion which proved to be the most imperative elements for making a contest succeed – or fail miserably.”

As discussed in this earlier post on incentives in crowd-based video production, we are extremely interested in understanding how to optimize incentives for storytellers from the crowd. Our goal is to create a win-win situation for creators and Dialogue Earth so that we can produce videos on a large number of topics in a sustainable manner.

Building from our experiences, we feel that a smart next step will be to elicit feedback from those who have participated in video contests through a survey. We expect the survey results to be of use to all those in the crowdsourcing realm who sponsor and conduct video contests. We are definitely open to input upfront so that we tune the survey to meet the needs of as many groups out there as possible.

More »

Have We at Dialogue Earth Broken Free of Randy Olson’s “Nerd Loop”?

by May 9, 2011

nerd-loop-piecePrior to reading Andy Revkin’s post Climate, Communication and the ‘Nerd Loop’ just now, I was unaware of Randy Olson’s newly coined term the “Nerd Loop.” It is a term that he recently gave to in-the-box strategies for communicating science to general audiences (read about it on his blog, The Benshi).

Olson argues passionately that there needs to be more risk taking in the science communication realm. I equate this to needing more out-of-the-box approaches, some of which will fail and some of which will help members of the public to understand a bit more about important issues like global warming, energy, food, water, land use, and so on. There won’t be a single approach that will work in all cases. Nor do I expect that there will be massive uptake of new information. It’ll be a slow, gradual process.

For me, I think the key for out-of-the-box approaches to work is that there needs to be an underlying genuine quality. Is there an effort to change people’s minds, or just to inform? If the goal is ultimately to change people’s minds, I deeply believe that even the most out-of-the-box efforts to raise literacy on a number of key issues connected to the environment will face barriers.

That’s why I’m committed to a non-advocacy approach with Dialogue Earth. We’re advocates for good information being present in societal dialogue and decision making. Period.

I believe that our strategy based in understanding the public dialogue, building credibility by drawing in a wide spectrum of experts, and ultimately delivering highly-engaging, crowd-based multimedia products holds lots of promise.

Ultimately, we can convince ourselves that we’ve stepped outside of the box, but our opinion amounts to very little. What do you think?

More »