Summary

In this paper, the authors aim to classify the Human Computation systems and compare/contrast the term with other terms like crowdsourcing, social computing, and data mining. The paper starts by presenting some definitions of the human computation where all refer to it as a utilizing the human power to solve problems that can’t be solved by the computers yet.

The paper present different computational systems that share some properties with the human computation and yet they are different. The authors highlight some computational systems like social computing, crowdsourcing, and data mining and show the similarities and distinctions with the human computation systems. All systems grouped together under the collective intelligence where humans’ intel solve a big problem.

The paper presents a classification system for human computation systems that is based on six dimensions. The dimensions include motivation, quality control, aggregation, human skills, process order, and task-request cardinality.

The authors presented different ways to that can motivate people to participate in the systems like pay, altruism, enjoyment and others. And presents the pros and cons for each approach.

The authors also presented different approach to improve the quality of the systems like output or input agreements, expert review, and multilevel reviews and ground truth seeding. All these approaches try to get better quality and ways to measure the performance of the system.

There are different aggregation approaches that collect the results of the tasks completed and to formulate the solution of the global problem.

Other dimensions like human skills, process order, and task-request cardinality discuss that skills required, the way order is processed and the pipeline the request can go through.

Reflection

I found one interesting definition of human computation interesting. It defines it as “systems of computers and large numbers of humans that work together in order to solve problems that can’t be solved by either computers or humans”. It is try that if humans can solve problems then there will be no need to use computers and also if systems can solve the problems and automate the solution then there will be no need for humans so both need to work together to solve bigger problems.

I also found the comparison between different systems including the human computation interesting. I personally was thinking that some systems like crowdsourcing is a human computation system, but it appears it is not.  

I agree with the dimensions that define or classify human computation systems as they are accurate measures that help researchers to build new system and to evaluate it.

To connect to other ideas, I found the work is like dynamic programming where we have to solve small problems to eventually solve the global problem. Small tasks are distributed to workers to solve a small problem and then aggregation methods will take these solutions to solve the global problem.

I also found the ground truth seeding quality control approach in similar to the training and testing data in any machine learning algorithm.

Questions

• What other dimensions can we define to classify a human computation system?
• There are different approaches that can measure the quality of a human computation systems. Which one is the best?
•  Can we combine to motivation methods together to get better results? Like combining both pay and the enjoyment to solve a global problem?

Read More

Within the recent emergence of Human Computation, there has been many advancements that have pushed further out of the into the industry. This growth has been so sporadic as many of the terminologies and nomenclature has not been well defined within the scientific area. Though all of these “ideas”are all considered within the umbrella term Human Computation, the common explanation for Human Computation is not strictly defined, as it has been used frequently loosely tied papers and ideas. This work states Human Computation as coupling both the problems that can eventually be migrated to computers and “human participation is directed by the computational system”. Furthermore, the study starts to define related terms that are equally as loosely defined. These terms, under the collective idea of Human Computing, included the common technological terms such as Crowdsourcing, Social Computing, Data Mining, and Collective Intelligence. Following among these more collectively defined definitions,various Crowd Sourcing platforms are compared in a more inclusive classification system. Within the system, various aspects of the Crowd Sourcing platforms are categorized various labels retrieved by common usage in industry and literature.Through those labels include the following terms that are used throughout (to some extent)of each of the crowd sourcing platforms; motivation human skill, aggregation, quality control, process order, and task-request cardinality. Underneath each of these top categories is more sub categories better defining each of the platforms, for example a label like Motivation (for use with each platform) has the following sub-labels underneath it, Pay, Altruism (peoples inherit will to do good), enjoyment, Reputation (to work with a big company) ,and implicit work (underlying work from the system). From helping to tie this vocabulary down to a clearer definition, it is the hope of the authors to better understand each platform and to better realize how to make sure each system is humanly good.

I disagree with how the labeling is created through this system. It is always a fundamental idea that with the classification there may seem to be more “gray area” within some of the platforms put under the label. In addition, this may also stifle some of new creative ideas since these could be the “broader” buckets people use to standardize their ideas. This could be related to ideas such as a standardized test that may miss the general learning while enforcing strictly learning a singular path.

While I do potentially agree with the upper level labeling system itself, I believe the secondary labeling should be left more open-ended.This would be again due to the limiting or even “under shadowing” the of a new discovery by entertaining the idea of making a more distinctive approach to attempt to relate the ideas to commonly collected ideas.

• I would like to see how many of the crowd sourcing examples are cross listed within any of the dimensions. It seems from their current system the examples listed may be easily (relatively) defined, the others unlisted may be able to fit into the categories that would be dropped from the table.
• Since this is a common classification system, I would like to see if there has been a user survey (amongst people actively using the technology) done to see if these labels accurately represent the research area.
• My final question pertaining to this system is how much this has been used actively in the industry. Potentially between advertisements or cores of the new platforms

Read More

What is human computation?

Human computation is a process that makes use of human’s power to solve the problem that could not currently be solved (well) by computers. As several related field emerging in recent years, understanding similarities and difference between them and human computation will contribute in the research and applications.

Collective intelligence consists of crowdsourcing, social computing, a large portion of human computation and a small portion of data mining. Crowdsourcing comes in when a designated job is outsourced to a (large) group of people; social computing refers to when communication in online communities is mediated by technology; data mining is the process of finding patterns among huge amount of data;  collective intelligence is a broad field where products of many people’s action are proved to be wiser.

Human computation can be classified according to 6 dimensions: motivation, quality control, aggregation, human skill, process order and task-request cardinality. Current research and projects on human computation share a combination of different values.

Reflection

This is a comprehensive paper and it covers a lot of dimension in human computation. I found myself stuck for a long time and then came up with many tiny passing thoughts. It’s hard to tell something about the whole similarities/difference or analyze if the number of dimensions should be six…

I believe a human computation system requires a lot of effort, just like the making progress on machine intelligence. Design, implementation, improvement, extra budget, etc. How do people determine the trade-off between the choice of the research towards a better model and the choice of a human computation assistance? I think for now the preference is to find help from human computation, but it’s changing gradually. As the paradox of automation evolves into a higher level, the tasks are also going through an improvement – they require higher-level human skills. While the smartness of machine approach 100% authentic, the role of human in human computation will approach “verifying the decision” rather than “filling the gap”.

If a guy gets all his paycheck from different tasks in human computation, his career will be totally different from someone sitting in the room focusing on certain responsibilities for a long time. The goals of the tasks vary so he/she has to switch really quickly between fields, actions and routines. Precision on first few objects would be lower and goes up gradually. All these require careful observation and adjustment compared to collecting the results from traditional employees.

All the implicit work is probably written in user privacies, but I don’t’ read them:P For example, situations where identity confirmation is needed always annoy me. As a CS student I should understand how important it is to secure my important information, however, “Select all squares with traffic lights” really bothers me. As I know more about machine learning and I realized that I’m helping to train Google’s AI, I became even more angry (I shouldn’t, I actually participated in several human computation with a motivation of altruism). I don’t know if it’s just me or it bothers other people, too.

Questions

How to find a good combination in human computation? An urgent task may pay (higher) to collect the results in need. How to prevent contributors from rushing for money? How to make sure different motivation reflect less on the outcome of the tasks? Will a paycheck have absolutely no influence on contributors? If not, how do we create a balanced inspiration to avoid the bias/influence?

What is the best practice to reveal what’s behind smart machine-move step by step? Contributors have been around for more than 10 years. If it’s not this course asking me to understand and utilize the tool of crowdsourcing, I may not realize that the underlying non-macine intelligence is all around. Why I haven’t heard much about this group of people and their job? Does it come naturally with the feature of their job or is there anything holding their opinion back from public?

Read More

Summary:

In this paper, the authors detail the emergence of Human Computation as a new area for research. They provide a solid definition of area’s constitution as:

• The problems fit the general paradigm of computation, and as such might someday be solvable by computers.  
• The human participation is directed by the computational system or process.

They further solidify the definition by comparing it with other similar fields and finding the niche differences between them. Additionally, the paper provides a classification system for human computation using six dimensions: 

1. Motivation: Speaks about different reward mechanisms for humans in the framework. 
2. Quality Control: Discusses various measures used to maintain quality of work and control humans that cheat the system’s reward.
3. Aggregation: Strategies for combining work done parallely by independent human labour into a block of usable data.
4. Human Skill: The extent of human’s ability to facilitate machines. Defines the places where general human computational abilities are better.
5. Process Order: Different frameworks of defining workflows with humans (requesters/workers) and computers.
6. Task-Request Cardinality: This defines the task assignment combinations such as if one task can be given to multiple humans or multiple tasks to the same human.

Reflection:

The authors define human computation by relying on previous definitions. But, the previous definitions show the dynamic nature of the field and hence, the definition should evolve over time. This, also, affects the derived system classification and its nomenclature. But, the paper is interesting in its effort to provide clear differences between human computation and relevant fields.

I was very interested in learning the different motivations (except monetary benefit) for human workforce. Similar motivations can be seen in early software development, where developers worked on open-source software without a monetary reward but just out of their nature for contributing to society. I believe given any field, monetary reward can never be the only motivation. Human beings are social animals with stronger instinct to contribute to society. Monetary benefit, on the other hand, is just perceptual. The classification nomenclature given by authors is very static to current worker platforms and I believe is subject to change overtime and hence, I believe it has limited advantage.

Questions:

1. Can this definition for human computation be static or is it subject to change overtime?
2. How can the task-request process, itself, change over-time?
3. To what extent should humans support machines before the challenges overcome the rewards?
4. What exactly is defined as a growing field? Does it mean the field will grow in research or will it spring out more sub-fields? 

Read More

Summary

The author of the paper started explaining about human computation, crowd sourcing, social computing, data mining, and collective intelligence and the differences between these terminologies. The author also explained the intersection of these areas and the areas where they are disjoint. After that the author explained about different dimensions of human computation work to control the work quality, in which each dimension has many values. And the author explained each value and its contribution to the system. And in the final area the author proposed possible future research areas that can be researched for new topics in this area.

Reflections

1. It was interesting to know how quality control is handled in human computations given money is involved people try to exploit the system.
2. Money is not the only factor that motivates people to work in this area there are many incentives that motivate people to complete this task.
3. There are different methods that exist to control the quality of work like review, voting, economic modeling, input-output agreement and automatic check etc.
4. The process order of the task also has a significant meaning for each task.

Questions

1. Is the present system for quality check enough to ensure quality? There might be tasks which might need expertise in which none of the workers exist to complete tasks.
2. Can collection of data can be considered as crowdsourcing, where we need images of different people for image recognition model?
3. Do you think 90% jobs that provide pay of less than 0.1$  die after some time due to the need for expertise?

Read More

In this article, Alexander J. Quinn, Benjamin B. Bederson. First introduced the background, then discussed the definition of human computer, distinguished and compared with related technologies, and then put forward the classification system for human computer system, and explained how to find new research points based on the proposed classification system. The article first proposed the definition of human computation. The author believes that human computation should meet:

1. The problems fit the general paradigm of computation, and as such might someday be solvable by computers.
2. The human participation is directed by the computational system or process. Then the author compares human computation with other concepts, which mainly include: crowd sourcing, social computing, data mining, and collective intelligence.

The main differences from these concepts are the presence of computers and the application direction. Then the author proposed a new classification dimension. According to the proposed dimension, problems can be considered the following aspects:

1. Combining different dimensions to discover new applications.
2. Create new values for a given dimension.
3. When encountering a new human computer system, classify it according to the current dimensions to discover new things

I think this article is similar to another article “An Affordance-Based Framework for Human Computation and Human-Computer Collaboration”. This article is all about the new direction of human computation. To help people find new methods through new classification systems, and find new applications based on the combination of different dimensions. The other article is about a new research method “Affordance”, which achieves better research results based on the relationship between humans and machines. And I think the arguments of the two articles coincide: The classification system mentioned in this article has six dimensions, motivation, quality control, aggregation, human skill, process order, task request cardinality. Among them, human skills can correspond to human advantages, that is the part of “affordance” that humans can take part in human computation. And motivation, quality control, aggregation as the description in another article, humans cannot be like computers, People cannot completely give up subjective thinking and realize unbiased analysis. The process order reflects different interaction methods and different interaction orders in human computation. Task request cardinality can correspond to other “affordance” methods. When the number of participants is large, there will be different methods. So I think in some ways the two articles are complementary. At the same time, in this article, the author also mentioned the difference between human computation and other concepts. I think this is very important in future research. In future research, there will be more and more interdisciplinary crossings, so it is important to distinguish these disciplines, determine the boundaries of the disciplines, and lay a solid foundation for different disciplines. The foundation, universal methods, and efficient solutions are not only good for the development of each discipline, but also have a very important impact in the interdisciplinary process.

What is the significance of distinguishing human computation from other definitions?

What are the characteristic of human computation corresponding with the six dimensions mentioned in article?

Is there a new dimension, and if it is combined with the dimensions mentioned in the article, what new applications will it have?

Read More

Summary:
“Human Computation: A Survey and Taxonomy of a Growing Field” by Quinn and Bederson classifies human computation systems different dimensions. They also point out the subtle differences between human computation, crowd sourcing, social computing, data mining and collective intelligence. Traditionally, human computation is defined as “a paradigm for utilizing human processing power to solve problems that computers cannot yet solve.” Although, there is some overlap between human computation and crowd-sourcing, the major idea behind crowds-sourcing is that it works by employing members of the public in place of traditional human workers. Social computing, on the other hand, differs from human computation in the manner such that it studies natural human behavior mediated by technology. The third term, data mining, focuses on using technology to analyse the data generated by humans. All these terms partly fall in the category of collective intelligence. Well-developed human computation systems are also examples of collective intelligence. For example, to create a gold standard dataset for machine translation, several humans have to provide translations for the same system. This is an example of the collective intelligence of the crowd as well as human expertise to solve computationally challenging tasks. The authors present a classification system based on six different dimensions: (i)Motivation, (ii)Quality Control, (iii)Aggregation, (iv)Human Skill, (v)Process Order and, (vi)Task Request Cardinality. The paper further talks about the various sub categories in these dimensions and how these sub categories influence the output. This work presents a useful framework that can be help researchers categorize their work into one of these categories.

Reflections:
This was an interesting read as it presented an overview of the field of human computation. The paper does a commendable job of highlighting the subtle differences in the different sub-fields of HCI. The authors classify human computation into one of the six dimensions depending on the sample values. However, I think there might be some overlap between these sample values and sub categories. For example, the amount of ‘pay’ a person receives definitely determines the motivation for the task but it may also determine the quality of work. If the worker feels that he is adequately compensated, that might prove an incentive to produce quality work. Similarly, ‘redundancy’ and ‘multi-level review’ are part of the ‘quality check’ dimension but they can also fall in the ‘task request cardinality’ dimension as multiple users are required to perform similar tasks. Another point to be considered here is that although, the authors differentiate between crowd-sourcing and human computation, several parallels can be drawn between them using the dimensions presented. For example, it will be interesting to observe whether the features employed by different crowd platforms can be categorized into the dimensions highlighted in this paper and whether they have the same sub class or do they vary depending on the kind of task at hand.

Questions:
1. Can this work extend to related fields like social computing and crowd-sourcing?
2. How can we categorize ethics and labor standards based on these dimensions?
3. Is it possible to add new dimensions to human computation?

Read More

Summary:

The key motivation of this paper is to distinguish the definition of human computation with other terms such as “crowdsourcing”. The author also explored the accurate definition of human computation and developed the classification system so that it provides directions for research on human computation. The author also analyzed human computation and demonstrated the key idea with a graph. The graph includes the motivation of people who work on this computation, how to control the quality and how to aggregate the work. Finally, what kind of skills required for human computation, and the regular process order for them. There are several overlaps between human computation and other terms. The definition of human computing is summarized by the author based on all kinds of definition source as two main points: First, the problem fit the general computing paradigm so could be solved one day. Second, human participants are guided by computing systems or processes. The author compared human computation with related ideas and present a classification system based on the six most significant distinguishing factors. This paper is mainly about the taxonomy of human computation, however, the author indicates the future usage of these classifications and the future direction of research such as the issues related to ethics and labor standards.

Reflection:

This article provides a clear definition of human computation, which I think is the most important step before trying to explore more information or expressing any opinion on this topic. I prefer the definition “…a technique to let humans solve tasks, which cannot be solved by computers.”, although we know these problems could be solved one day with the development of technology. Just look at the motivation indicated by the author, I would consider the human computation is an inevitable trend as the reveal of the deficiency of Artificial Intelligent and as the need of network users. There is an interesting contradiction that happens to me when I was read the paper. When I check the graph of the “classification system for human computation systems overview”, I found the author indicates one of the motivations is altruism. I was skeptical and I do not believe until I saw the example, “thousands of online volunteers combed through over 560,000 satellite images hoping to determine Gray’s location, who was missing during a sailing trip in early 2007”, I think it’s the best reason. The book “ghost work” could be one type of work that included in human computation definition in the paper. The motivation of their work is payment, and a different job has a different process order. The tag of the picture could be considered as “Worker->Requester->Computer(WRC)”, while the Uber driver’s case might be “Computer-> Woker->Requester”. This paper is a summary and classification of the present state of human computation, without any innovative ideas. However, the follow-up work that the author put forward at the end is worth discussing. Especially the topic of issues related to ethics and labor standards. We do not have any kind of regulation for this mode of work. Thus, how to protect the workers? How to prevent the product from intentional destruction. How human computation will develop in the future？

Questions:

• Do you think the field of human computation will exist for a long time? Or will soon be replaced by the highly developed AI？
• What aspect of human computation do you think will involve an ethical problem?
• Which description in this paper more in line with ghost work?
• Can we find any other examples of the motivation or quality control of human computation?

Read More

This paper attempts to define a region where human computation belongs, including its definition and similar ideas. According to the paper’s quote from Von Ahn’s dissertation on human computation, it is defined as a way of solving a computation problem that a machine cannot yet handle. The paper compares human computation with crowdsourcing, social computing, and data mining and explain how they are similar but different. The paper continues to study the dimensions related to human computation, starting with motivation. These include factors such as pay, altruism, and joy. The next dimension that the paper discuss is quality control, the method of ensuring an above-threshold accuracy of human computation results. These included multi-response agreement, expert review, and automatic check. Then, the paper introduces how the gathered computations by many humans can be aggregated together to solve the ultimate problem. These included collection, statistical processing, improvement, and search. Finally, the paper discusses a few more small dimensions such as process order and task-request cardinality.

I enjoyed the paper’s attempt to generate a taxonomy for human computation which can be easily ill-defined. I think the paper did a good job at it by starting with the definition and breaking it down into major components. In the paper’s discussion about aggregation, it was interesting to me that they included “none”, which means the individual human computations by themselves are the major problem that the requester wants solved, and there is no need for aggregation of all the results. Another thing I found fascinating about the project was their mentioning of motivation for the humans performing the computation. Even though it is natural that people will not perform the tasks for nothing, it did not occur to me that this would be a major factor to consider when utilizing human computation. Of the list of possible motivations, I found altruism to be a humorous and unexpected category.

I was also reminded of a project that used human computation, called “Place” held in a community called Reddit, where a user of the community could place a colored pixel on a shared canvas once in a few minutes. The aggregation of human computation of “Place” would probably be considered as iterative improvement.

These are the questions that I could come up with while reading the paper:

1. The aggregation category “none” is very interesting, but I cannot come up with an immediate example. What would be a good case of utilizing human computation that doesn’t require aggregation of the results?

2. In the Venn diagram figure of the paper showing relationships between human computation, crowdsourcing, and social computing, what kind of problems would go into the region where all three overlap? This would be a problem where many people on the Internet with no explicit relation to each other socially interact and cooperate to perform computation that machines cannot yet do. The collected results may be aggregated to solve a larger problem.

3. Data mining was not considered a human computation because it was about an algorithm trying to discover information from data collected from humans. If humans sat together trying to discover information from data generated by a computer, would this be considered human computation?

Read More

In this paper, the authors focused on the problem that due to the rapid growth of computing technology, current methods are not well supported by a single framework that can understand each new system in the context of old helpfully. Based on this research question, authors categorized multiple human computations systems aiming at identifying parallels between different systems, classifying systems into different dimensions, and disclosing defects which existed in current systems and work. Then, the authors compared human computing with other related ideas, terms, and areas. For example, deafferenting human computing with social computing, crowdsourcing. For the classification, the authors divided different systems into six dimensions: motivation, quality control, aggregation, human skill, process orders, and task request cardinality. For each dimension, the authors explained sample values and listed one example. Due to the development of human computation, new systems can be categorized into current dimensions, or new dimensions and sample values will be created in the future.

From this paper, I knew that human computing is a wild topic which is hard to be defined clearly. There are two main parts that consist of human computing: 1) problems fit the general paradigm of computation, 2) the human participation id directed by the computational systems or process. Human computation binds human activities and computers tightly. For the six dimensions, I am kind of confused that how authors categorized these systems into these six dimensions. I think that authors need to talk more about how and why. From this form, I can find that one system can be categorized into multiple dimensions due to its complex features, for example, Mechanical Turk. And I think this is one possible reason that systems are hard to be classified in human computing easily. Because one system may solve many human computing problems and implements multiple features increasing the difficulty of understanding its context. What’s more, I am quite interested in the “Process order” dimension. From this part, it helps me to understand how people interact with computers. For different process order, people can generate different questions that need them to solve. And it is impossible to come up with a solution as a panacea that works well in each processed order. We should consider questions like feedback, interactions, learning effects, curiosity and so on.

What’s more, I am interested in the idea that focusing on only one style of human computation may become a tendency that can potentially missing more suitable solutions to a problem. Thinking differently in multiple ways would help us quickly solve the research questions. We are not supposed to limit us on one narrow topic or one single area.

Question 1: how can we use this classification of human computation systems?

Question 2: how and why authors come up with these six dimensions? I think more explanations are needed.

Question 3:  If one system is classified into multiple dimensions and sample values, can I treat these values equally? Or there is one majority values and dimension?

Read More