Paper: Steven Dow, Anand Kulkarni, Scott Klemmer, and Björn Hartmann. 2012. Shepherding the crowd yields better work. In Proceedings of the ACM 2012 conference on Computer Supported Cooperative Work (CSCW ’12). ACM, New York, NY, USA, 1013-1022. DOI=http://dx.doi.org/10.1145/2145204.2145355
Discussion Leader: Anamary Leal
Summary
The research goal of this work is: How can crowdsourcing support learning? How do we get better, multi-faceted, creative, complex work from unskilled crowds?
Their solution is shepherding – providing meaningful real-time feedback, with the worker iterating on their work with an assessor who will give them feedback.
For the task of writing a product review, a worker would write the review, and then the shepherd gets a notification of a review submission. Then, in real-time, the shepherd gave structured feedback based on a ranking of the quality, checklist of things to cover in the review, and an open-ended question. Then the worker has chance to improve the work.
The authors conducted a comparative study structured around product reviews with a control condition (no feedback), self-assessment, and this external assessment (shepherding). They measured task performance, learning, and perseverance (amount of revisions and string edit distance). The workers and the shepherd were all from the crowd, though the shepherd came from a single reviewer from ODesk. Self-assessment did slightly better than shepherding, and both feedback conditions did significantly better than the no feedback condition. Shepherding resulted in more worker edits, along with more and better revisions.
Reflection
This paper is a great stepping stone for more questions to explore. In addition to the attrition question and different learning strategies, I wonder in the longer term, how well do these crowds learn. Short-bursts of learning is one thing (like cramming) but I wonder if those same workers, through more feedback, get better and writing reviews than others. How well do these lessons stick? The role of feedback can help in bringing the dream of having crowdsourcing work we would want our kids to do.
Another stepping stone is to measure with respect to iterations, even if it’s in the short term. How many gains happen if the worker gets 2+ iterations with the assessor, or even in self-assessment?
Feedback, especially external feedback, helped motivate workers to do more and better work. I’m not well versed in educational research, but engagement in teaching the material and assessment are quite important.
The authors took care to mention the attrition rate, and what composed of that rate. I wonder what can be done about that population. Most of the attribution is dropped out too early, but a decent portion was due to plagiarism. I wonder what those participants saw in the task to discourage them to not do the task.
The external condition probably would have not been as successful if the feedback form was not appropriately structured, with a helpful checklist of items to cover. I can imagine that a ton of design work went into that form to guide shepherds to provide prompt constructive feedback that the worker can deliver upon.
In their studies, it looks like workers cannot substitute expert sheepherders and provide quality feedback. But I wonder if that too can be learned? It’s harder to teach something than to just be good at something.
Discussion
- How do we get this non-trivial audience who dropped out, to participate? Feedback encouraged more work, so in a general sense, would more feedback lure them in?
- If the task turned to assessing reviews compared to writing reviews, which one do you think would require more iterations to get better? Which one would be easier to learn, to write or critique others?)
- How much feedback do you think is needed for an unskilled worker to get better at these creative, multi-faceted, complex tasks? Are there some examples, like writing a story, which may need more cycles of writing and review to be better at it?
- How do you see (or not see) real-time external assessment fit into your projects, and what do you think the gains would be, after reading this paper?
