Alexander J. Quinn and Benjamin B. Bederson. 2011. Human computation: a survey and taxonomy of a growing field. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’11). ACM, New York, NY, USA, 1403-1412. DOI=10.1145/1978942.1979148 http://doi.acm.org/10.1145/1978942.1979148
Summary
This paper presents a classification of “human computation” and related concepts, including crowdsourcing, social computing, collective intelligence, and data mining. This work is motivated by a growing body of research that uses many of these terms interchangeably, resulting in potential confusion. By surveying a large body of literature, it proposes definitions for each of these areas and describes how they overlap and differ from one another. A Venn diagram helps illustrate these relationships. The authors suggest that human computation was popularized by Luis von Ahn’s 2005 dissertation and its defining trait is a computational process that uses human effort to solve problems that computer can’t. Crowdsourcing was coined by Jeff Howe and is defined as taking a job originally meant for one person and opening it up to a large group of people. The authors identify six key dimensions of human computation, citing examples for each. Three of these — motivation, human skill, and aggregation — are based on analysis of clusters of related projects with a defining attribute. The others — quality control, process order, and task-request cardinality — cut across project clusters. Finally, the authors suggest some opportunities for future work revealed by underexplored areas in their taxonomy. These include exploring new pairings of dimensions, new values for dimensions, and new kinds of work.
Reflection
This strikes me as a great overview of crowdsourcing and human computation research as it stood in 2011 when this paper was published. The definitions of crowdsourcing, human computation, social computing, et al. were not necessarily what I intuitively expected, but they seem reasonable and helpful. With these terms used and abused to mean so many different things, I appreciated a serious attempt to provide some clarity and common ground to researchers working in these areas. In my mind, social computing is the broadest category and encompasses most of these other concepts, but the authors’ more limited scope (humans interacting naturally, mediated through tech) was interesting. I thought the point that human computation need not be collaborative was interesting and hadn’t considered it before. I also note that the authors don’t cite Daren Brabham who is well known for his writings on crowdsourcing and what it means (and doesn’t mean). This would have been an interesting point of contrast but the authors may have skipped him purposefully or accidentally because he doesn’t often publish in CS/HCI venues. There are also a ton of new human computation and crowdsourcing projects in the 4 years since this was published; I wonder how well they fit into this taxonomy and/or require revisions. Certainly, the values for the “human skill” dimension have expanded to include more complex and creative tasks like visual design, writing, and even scientific research. I also note that “learning” or “feedback” are quality control mechanisms that aren’t listed but have become increasingly important.
Questions
- Do we agree or disagree with the definitions provided here? Have they become more focused or broader in recent years?
- What are some other crowdsourcing or human computation examples you can think of that aren’t listed here, and where would they fall in each of the dimensions? Can you think of some that don’t fit the existing values or dimensions?
- What other fields might be included in the Venn diagram and how are they related?
- Taxonomies like this are supposed to help researchers, in the authors’ own words, identify opportunities in unexplored or underexplored areas. What opportunities do you see here? Any the authors didn’t mention?
- Amy Bruckman writes about the usefulness of category theory in describing different kinds of online communities as being more or less like a set of “prototypes”. How might this argument extend to human computation?
- How does a paper like this help you (or not help you) understand the fields of crowdsourcing and human computation research?
