Summary:

The paper briefly speaks of the history of human computation. The first dissertation (2005), workshop (2009), and the different backgrounds of scholars in human computation. The authors agree with Von Ahn’s definition of the human computation as: “… a paradigm for utilizing human processing power to solve problems that computers cannot yet solve.” and mention multiple definitions from other papers and scholars. They believe that two conditions need to be satisfied to constitute human computation:

• The problems fit the general paradigm of computation, and so,  might someday be solvable by computers.
• The human participation is directed by a computational system or process.

They present a classification for human computation systems made of 6 main factors divided into two groups: 

• Motivation, human skill, aggregation.
• Quality control, process, task-request cardinality.

The authors also explain how to find new research problems based on the proposed classification system:

• Combining different dimensions to discover new applications.
• Creating new values ​​for a given dimension.

Reflection:

The interesting issue I found the authors discussing was that they believe that the Wikipedia model does not belong to human computation. Because current Wikipedia articles are created through a dynamic social process of discussion about the facts and presentation of each topic among a network of authors and editors. I never thought of Wikipedia as human computation although there are tasks in there that I believe could be classified as such. Especially when looking at non-English articles. As we all know, the NLP field has created great solutions for the English language, yet some languages, even widely spoken ones, are playing catch up. So, this brings me to disagree with the authors’ opinion about Wikipedia. I agree that some parts of Wikipedia are related to social computing like allowing collaborative writing, but they also have human computation aspects like Arabic articles linked data identification (for the info box). Even though using NLP techniques might work for English articles on Wikipedia, Arabic is still behind when it comes to such task and the machine is unable to complete it correctly. 

On another note, I like the way the authors broke up their classification and explained each section. It clarified their point of view and they provided an example for each part. I think that the distinctions were addressed in detail and they left enough room to consider the classification of future work. I believe that this was the reason that other scientists have adapted the classification. Seeing that the paper was cited more than 900 times, it makes me believe that there’s some agreement in the field. 

Discussion:

1. Give examples of human computation tasks.
2. Do you agree/disagree with the author’s opinion about Wikipedia’s articles being excluded from the human computation classification?
3. How is human computation different from crowdsourcing, social computing, data mining, and collective intelligence?
4. Can you think of a new human computation system that the authors didn’t discuss? Classify it according to the dimensions mentioned in the paper.
5. Do you agree with the authors’ classification system? Why/Why not?
6. What is something new that you learned from this paper?

Read More

Summary of the Reading

This paper is a survey of the research in the field of human computation. The paper aims to classify human computation systems so that the similarities between different projects and the holes in current research can be seen more clearly. The paper also explores related fields like crowdsourcing.

The paper starts by defining human computation as “a paradigm for utilizing human processing power to solve problems that computers cannot yet solve.” The paper then goes on to discuss the differences between human computation and crowdsourcing, social computing, data mining, and collective intelligence.

The paper then goes on to classify human computation systems based on 6 dimensions: motivation, quality control, aggregation, human skill, process orders, and task-request cardinality. Each of these dimensions has several discrete options. Putting all of these dimensions together allows for the classification of any arbitrary human computation system. The paper also provides examples of systems that have various values on each dimension.

Reflections and Connections

I think this paper provides a much needed tool for further human computation and crowdsourcing research. The first step to understanding something is being able to classify that thing. This tool will allow current and future researchers to classify human computation systems so that they can apply existing research to other, similar systems and also allow them to see where existing research falls short and where they need to focus future research.

This research also provides an interesting perspective on current human computation systems. It is interesting to see how current human computation systems compare to each other and what each system does differently and what they have in common. 

I also like the malleability of the classification system. They say in the future work section that this system is very easy to add to. Future researchers who continue the work on this project could easily add values to each of the dimensions to better classify the human computation systems. They could also add values to the dimensions if new human computation systems are invented and need to be classified using this system. There are a lot of good opportunities for growth from this project.

One thing that I thought this paper was missing is a direct comparison of different human computation systems on more than one dimension. The paper uses human computation systems as examples in the various values for each of the dimensions of the classification system, but it doesn’t put these dimensions together and compare the human computation systems on more than one dimension. I think this would have added a lot to the paper, but it would also make for a great piece of future work for this project. This idea is actually very similar to the other paper from this week’s bunch, titled “Beyond Mechanical Turk: An Analysis of Paid Crowd Work Platforms” and I think it would be helpful and build on both of these papers. 

Questions

1. Do you think the classification system presented in this paper has enough dimensions? Does it have too many?
2. What is one application you see for this classification system?
3. Do you think this classification system will help crowdsourcing platforms deal with some of their issues?

Read More

Summary

In the paper, Quinn and Bederson reflect on the current state of human computation research and define a framework for current and future research in the field. They make sure to impart to the reader that human computation is not crowdsourcing, nor collective intelligence – rather, it is a space where human effort is used where computer may be able to solve the problems in the future. They then define several dimensions on how to classify a human computation study; these are motivation (which can include pay or altruism among others), quality control (or how the study ensures reliable results), how the study aggregates the data, what human skill is used (visual perception etc.), process order (how the tasks are deployed) and task-request cardinality (how many tasks are deployed for how many requests). Using these dimension definitions, the authors define new research areas for growth, through pointing out uncombined dimensions or by creating new dimensions to explore.

Personal Reflection

I read this paper after reading the human computation/human computer collaboration affordances survey, and it was interesting to compare and contrast the two papers for how they approached very similar problems in different ways. This paper did a good job in defining dimensions rather than research areas. It was much easier to understand how one can change the dimensions of research as a sort of toggle on how to tackle the issues they purport to solve.

Also, the beginning seemed to be on a tangent about what human computation is really defined as, but I thought this section helped considerably narrow the scope of what they wanted to define. I had thought of human computation and crowdsourcing as synonyms, so getting them separated early on was a good way of setting the scene for the rest of the paper.

Also, this paper opened my eyes to see how wide the dimensions could be. For example, while I had known of a few methods for quality control, I hadn’t realized there were so many different options.

Lastly, I am very happy they addressed the social issues (in my opinion) plaguing this field of research in the conclusion. Treating these workers as faceless mercenaries is dehumanizing at best. I wish there was a little more interaction between the two parties than there is currently, but it is being at least thought of in these survey studies.

Questions

1. What dimension do you think has the most promising potential for new growth, and why?
2. Do you think you can start a new research project by just choosing a set of 6 choices (1 for each dimension) and then design a project?
3. If a project has the same collection of dimensions as another proven study, is there merit in researching it? Or should it just work?
4. Can you think of any study that might fit under two different discrete values of the same dimension? I.E., is there a many (studies) to one dimensional value relationship, or is it many to many?

Read More