1/29/2020 – Jooyoung Whang – Human Computation: A Survey and Taxonomy of a Growing Field

January 28, 2020January 28, 2020 Vikram Mohanty Leave a comment

This paper attempts to define a region where human computation belongs, including its definition and similar ideas. According to the paper’s quote from Von Ahn’s dissertation on human computation, it is defined as a way of solving a computation problem that a machine cannot yet handle. The paper compares human computation with crowdsourcing, social computing, and data mining and explain how they are similar but different. The paper continues to study the dimensions related to human computation, starting with motivation. These include factors such as pay, altruism, and joy. The next dimension that the paper discuss is quality control, the method of ensuring an above-threshold accuracy of human computation results. These included multi-response agreement, expert review, and automatic check. Then, the paper introduces how the gathered computations by many humans can be aggregated together to solve the ultimate problem. These included collection, statistical processing, improvement, and search. Finally, the paper discusses a few more small dimensions such as process order and task-request cardinality.

I enjoyed the paper’s attempt to generate a taxonomy for human computation which can be easily ill-defined. I think the paper did a good job at it by starting with the definition and breaking it down into major components. In the paper’s discussion about aggregation, it was interesting to me that they included “none”, which means the individual human computations by themselves are the major problem that the requester wants solved, and there is no need for aggregation of all the results. Another thing I found fascinating about the project was their mentioning of motivation for the humans performing the computation. Even though it is natural that people will not perform the tasks for nothing, it did not occur to me that this would be a major factor to consider when utilizing human computation. Of the list of possible motivations, I found altruism to be a humorous and unexpected category.

I was also reminded of a project that used human computation, called “Place” held in a community called Reddit, where a user of the community could place a colored pixel on a shared canvas once in a few minutes. The aggregation of human computation of “Place” would probably be considered as iterative improvement.

These are the questions that I could come up with while reading the paper:

1. The aggregation category “none” is very interesting, but I cannot come up with an immediate example. What would be a good case of utilizing human computation that doesn’t require aggregation of the results?

2. In the Venn diagram figure of the paper showing relationships between human computation, crowdsourcing, and social computing, what kind of problems would go into the region where all three overlap? This would be a problem where many people on the Internet with no explicit relation to each other socially interact and cooperate to perform computation that machines cannot yet do. The collected results may be aggregated to solve a larger problem.

3. Data mining was not considered a human computation because it was about an algorithm trying to discover information from data collected from humans. If humans sat together trying to discover information from data generated by a computer, would this be considered human computation?

01/29/2020-Donghan Hu-Human Computation: A Survey and taxonomy of a Growing Field

January 28, 2020January 28, 2020 Donghan Hu Leave a comment

In this paper, the authors focused on the problem that due to the rapid growth of computing technology, current methods are not well supported by a single framework that can understand each new system in the context of old helpfully. Based on this research question, authors categorized multiple human computations systems aiming at identifying parallels between different systems, classifying systems into different dimensions, and disclosing defects which existed in current systems and work. Then, the authors compared human computing with other related ideas, terms, and areas. For example, deafferenting human computing with social computing, crowdsourcing. For the classification, the authors divided different systems into six dimensions: motivation, quality control, aggregation, human skill, process orders, and task request cardinality. For each dimension, the authors explained sample values and listed one example. Due to the development of human computation, new systems can be categorized into current dimensions, or new dimensions and sample values will be created in the future.

From this paper, I knew that human computing is a wild topic which is hard to be defined clearly. There are two main parts that consist of human computing: 1) problems fit the general paradigm of computation, 2) the human participation id directed by the computational systems or process. Human computation binds human activities and computers tightly. For the six dimensions, I am kind of confused that how authors categorized these systems into these six dimensions. I think that authors need to talk more about how and why. From this form, I can find that one system can be categorized into multiple dimensions due to its complex features, for example, Mechanical Turk. And I think this is one possible reason that systems are hard to be classified in human computing easily. Because one system may solve many human computing problems and implements multiple features increasing the difficulty of understanding its context. What’s more, I am quite interested in the “Process order” dimension. From this part, it helps me to understand how people interact with computers. For different process order, people can generate different questions that need them to solve. And it is impossible to come up with a solution as a panacea that works well in each processed order. We should consider questions like feedback, interactions, learning effects, curiosity and so on.

What’s more, I am interested in the idea that focusing on only one style of human computation may become a tendency that can potentially missing more suitable solutions to a problem. Thinking differently in multiple ways would help us quickly solve the research questions. We are not supposed to limit us on one narrow topic or one single area.

Question 1: how can we use this classification of human computation systems?

Question 2: how and why authors come up with these six dimensions? I think more explanations are needed.

Question 3: If one system is classified into multiple dimensions and sample values, can I treat these values equally? Or there is one majority values and dimension?

01/29/2020-Donghan Hu-An Affordance-Based Framework for Human Computation and Human-Computer Collaboration

January 28, 2020January 28, 2020 Donghan Hu Leave a comment

Many researchers’ primary goals are to develop tools and methodologies that can facilitate human-machine collaborative problem solving and to understand and maximize the benefits of the partnership of size and complexity. The first problem is how do we tell if a problem would benefit from a collaborative technique? This paper mentioned that even though deploying various collaborative systems has led to many novel approaches to difficult problems, it has also led to the investment of significant time, expense and energy. However, these problems might be solved better by only depending on human or machine techniques. The second problem is how do we decide which tasks to delegate to which party and when? The authors stated that we are still lacking a language for describing the skills and capacity of the collaborating team. For the third question, how does one system compare to others trying to solve the same problem? Lacking of no common language or measures by which to describe new systems is one important reason. About the research contributions, authors picked out 49 publications from 1271 papers which represent the state of the art in the study of human-computer collaboration and human computation. Then authors identify grouping based on human- and machine-intelligence affordances which form the basis of a common framework for understanding and discussing collaboration works. Last, the authors talked about the unexplored areas for future work. Each of the current frameworks is specific to a subclass of collaborative systems which is hard to extend them to a broader class of human-computer collaborative systems.

Based on the definition of “affordance”, I know both humans and machines bring to the partnership opportunities for action, and each mush be able to perceive and access these opportunities in order for them to be effectively leveraged. It is not surprised for me that the bandwidth of information presentation is potentially higher in the visual perception than any of the other senses. I consider that visual perception as the most important information processing for humans in most cases, that’s why there are a plethora of research studies combined with human visual processing to solve various problems. I am quite interested in the concept of sociocultural awareness. Individuals understand their actions in relation to others and to the social, cultural and historical context in which they are carried out. I think this is a paramount view in the study of HCI. Different individuals in different environments with different cultural backgrounds would behave different interactions with the same computers. In the future, I consider that cultural background should become an important factor in the studies of HCI.

I found that various applications are categorized into multiple affordances. If so, how can the authors answer the third question? For example, if two systems are trying to solve the same problem, but each of them have different human or computer affordance, how can I say which is better? Does different affordance have different weight values? Or we should treat them equally?

Less tools are designed for creativity, social and bias-free affordance, what does this mean? Is it mean that these affordances are less important or researchers are still working on these areas?

01/29/20 – Lulwah AlKulaib- Human Computation: A Survey & Taxonomy of a Growing Field

January 28, 2020January 28, 2020 Lulwah AlKulaib 1 Comment

Summary:

The paper briefly speaks of the history of human computation. The first dissertation (2005), workshop (2009), and the different backgrounds of scholars in human computation. The authors agree with Von Ahn’s definition of the human computation as: “… a paradigm for utilizing human processing power to solve problems that computers cannot yet solve.” and mention multiple definitions from other papers and scholars. They believe that two conditions need to be satisfied to constitute human computation:

The problems fit the general paradigm of computation, and so, might someday be solvable by computers.
The human participation is directed by a computational system or process.

They present a classification for human computation systems made of 6 main factors divided into two groups:

Motivation, human skill, aggregation.
Quality control, process, task-request cardinality.

The authors also explain how to find new research problems based on the proposed classification system:

Combining different dimensions to discover new applications.
Creating new values for a given dimension.

Reflection:

The interesting issue I found the authors discussing was that they believe that the Wikipedia model does not belong to human computation. Because current Wikipedia articles are created through a dynamic social process of discussion about the facts and presentation of each topic among a network of authors and editors. I never thought of Wikipedia as human computation although there are tasks in there that I believe could be classified as such. Especially when looking at non-English articles. As we all know, the NLP field has created great solutions for the English language, yet some languages, even widely spoken ones, are playing catch up. So, this brings me to disagree with the authors’ opinion about Wikipedia. I agree that some parts of Wikipedia are related to social computing like allowing collaborative writing, but they also have human computation aspects like Arabic articles linked data identification (for the info box). Even though using NLP techniques might work for English articles on Wikipedia, Arabic is still behind when it comes to such task and the machine is unable to complete it correctly.

On another note, I like the way the authors broke up their classification and explained each section. It clarified their point of view and they provided an example for each part. I think that the distinctions were addressed in detail and they left enough room to consider the classification of future work. I believe that this was the reason that other scientists have adapted the classification. Seeing that the paper was cited more than 900 times, it makes me believe that there’s some agreement in the field.

Discussion:

Give examples of human computation tasks.
Do you agree/disagree with the author’s opinion about Wikipedia’s articles being excluded from the human computation classification?
How is human computation different from crowdsourcing, social computing, data mining, and collective intelligence?
Can you think of a new human computation system that the authors didn’t discuss? Classify it according to the dimensions mentioned in the paper.
Do you agree with the authors’ classification system? Why/Why not?
What is something new that you learned from this paper?

1/29/2020 – Jooyoung Whang – An Affordance-Based Framework for Human Computation and Human-Computer Collaboration

January 28, 2020January 28, 2020 Vikram Mohanty Leave a comment

In this paper, the author reviews more than 1200 papers to identify how to best utilize human-machine collaboration. Their field of study was visual analytics, but the paper was well-generalized to fit many other research areas. The paper discusses two foundational factors to consider when designing a human-machine collaborative system: Allocation and affordance. In the many papers that the authors reviewed, systematic methods of trying to appropriately allocate work for each human and computer in a collaborative setting was studied. A good rule was introduced by Fitts, but it was found outdated later due to the increasing computational power of machines. The paper decides that inspecting affordance rather than allocation is a better way to utilize human-machine collaborative systems. Affordance can be best understood as what something an agent is good at than others. For example, humans can provide excellent visual processing skills while computers accel at large-data processing. The paper also introduces some case studies where multiple affordances from each party was utilized.

I greatly enjoyed reading about each of the affordances that human and machine can each provide. The list of affordances that the paper provides will serve as a good resource to come back to when trying to design a human-machine collaborative system. One machine affordance that I do not agree with is bias-free analysis. In machine learning scenarios, a learning model is very often easily biased. Both humans and machines can be biased in analyzing something based on previous experience or data. Of course, it is the responsibility of the designer of the system to ensure unbiased models, but as the designer is a human, it is often impossible to avoid bias of some kind. The case study regarding the reCAPTCHA system was an interesting read. I always thought that CAPTCHAs were only used for security purposes, and not machine learning. After learning how it is actually used, I was impressed how efficient and effective the system is at both securing Internet access as well as digitalizing physical books.

The followings are the questions that I came up with while reading the paper:

1. The paper does a great job at summarizing what each a human and a machine is relatively good at. The designer, therefore, simply needs to select appropriate tasks from the system to assign to each human and machine. Is there a good way to identify what affordance the system’s task needs?

2. There’s another thing that humans are really good at compared to a machine: adapting. Machines, upon their initial programming, does not change their response to an event according to time and era while humans very much do. Is there a human-machine collaborative system that would have a task which would require the affordance “adaptation” from a human collaborator?

3. Many human-machine collaborative systems register the tasks that needs to be processed using an automated machine. For example, the reCAPTCHA system (the machine) samples a question and asks the human user to process it. What if it was the other way around where a human register a task and assigns the task to either a machine or a human collaborator? Would there be any benefits to doing that?

01/29/20 – The Future of Crowd Work – Subil Abraham

January 28, 2020January 28, 2020 Subil Abraham 1 Comment

Reading: Aniket Kittur, Jeffrey V. Nickerson, Michael Bernstein, Elizabeth Gerber, Aaron Shaw, John Zimmerman, Matt Lease, and John Horton. 2013. The Future of Crowd Work. In Proceedings of the 2013 Conference on Computer Supported Cooperative Work (CSCW ’13), 1301–1318. https://doi.org/10.1145/2441776.2441923

What can we do to make crowd work better than the current state of simple tasks, to allow more complexity and satisfaction for the workers? The paper tries to provide a framework to improve crowd work in that direction. It does this through framing it in terms of 12 research directions that need to be studied so that they can be improved upon. The research foci are envisioned to promote the betterment of the current, less than stellar, sometimes exploitative nature of crowd work and make it into something “we would want our children to participate” in.

I like their parallels to distributed computing because it really is like that, trying to coordinate a bunch of people to complete some larger task by combining the results of smaller tasks. I work on distributed things so I appreciate the parallel they make because it fits my mental framework. I also find it interesting that one of the ways of quality control is to observe the worker’s process rather than just evaluating the output but it makes sense that evaluating the process allows the requester to maybe give guidance on what the worker is doing wrong and help improve the processes, whereas with just looking at the output, you can’t know where things went wrong and can only guess. I also think that their suggestion that crowd workers can move up to be full employees as somewhat dangerous because it seems to incentivize the wrong things for companies. I’m imagining a scenario where a company is built entirely on utilizing high level crowd work where they’re advertising that you have opportunities to “move up”, “make your own hours”, “hustle will reach the top”, where the reward is job security. I realize I just described what tenure track may be like for an academic. But that kind of incentive structure seems exploitative and wrong to me. This kind of set up seems normal because it may have existed for a long time in academia and prospective professors accept it because they are single mindedly determined (and somewhat insane) that they are willing to see this through. But I would hate for something like that to become the norm everywhere else.

Did anyone feel like there was any avenue that wasn’t addressed? Or did the 12 research foci fully cover every aspect of potential crowd work research?
Do you think the idea of moving up to employee status on crowd work platforms as a reward for doing a lot of good work is a good idea?
What kind of off-beat innovations can we think of for new kinds of crowd platforms? Just as a random example – a platform for crowds to work with other crowds, like one crowd assigns tasks for another crowd and they go back and forth.

01/29/20 – Affordance-Based Framework for Human Computation and Human-Computer Collaboration – Subil Abraham

January 28, 2020January 28, 2020 Subil Abraham 1 Comment

Reading: R. Jordon Crouser and Remco Chang. 2012. An Affordance-Based Framework for Human Computation and Human-Computer Collaboration. IEEE Transactions on Visualization and Computer Graphics 18, 12: 2859–2868. https://doi.org/10.1109/TVCG.2012.195

This paper is creating a summary of data visualization innovations as well as more general human computer collaboration tools for interpreting and making conclusions for data. The goal of the paper is to create a common language by which to categorize these tools and thereby provide a way of comparing the tools and understanding exactly what is needed for a particular situation rather than relying on just researcher intuition. They set up a framework in terms of affordances, what a human or computer can find opportunity and are capable of doing to do given the environment. By framing things in terms of affordances, we are able to identify how a human and/or computer can contribute to the goal of a given task, as well as be able to frame a system in comparison to other systems in terms of their affordances.

The idea of categorizing human-computer collaborations in terms of affordances is certainly an interesting and intuitive idea. Framing the characteristics of the different tools and software we use in these terms is a useful way of looking at things. However, as useful as the framework is, having read a little bit about function allocation, I don’t see how hugely different affordances are from function allocation. They both seem to be saying the same thing, in my view. The list of affordances is a bit more comprehensive than the Fitts HABA-MABA list. However, they both seem to be conveying the same information. Perhaps I do not have the necessary width of knowledge to see the difference, but the paper doesn’t make any convincing argument that is easy for an outsider to this field to understand.

Questions for discussion:

How effective of a system is affordances? What use is it actually able to provide besides being one more set of standards? (relevant xkcd: https://m.xkcd.com/927/)
There is a seemingly clear separation between human and machine affordances. But human adaptability seems to be third kind of affordance, a hybrid affordance where a machine action is used to spark human ingenuity. Does that seem like a valid or would you say that adaptability falls clearly in one of the two existing categories?
Now that we have a language to talk about this stuff, can we now use this language, these different affordances, to combine together to create new applications? What would that look like? Or are we limited to just identifying an application by its affordances after its creation?

01/27/20 – Mohannad Al Ameedi – Ghost Work

January 28, 2020January 28, 2020 mohada4 Leave a comment

Summary of the Reading

The authors talk about a hidden or invisible power named as a Ghost Work that work side by side with software and AI systems to make sure that data is accurate when the AI system fails to do so. Users of systems like google search, YouTube, Facebook, and other applications don’t know that there are people working behind the sense to make sure that the internet is safe. There are thousands of workers that work on demand on a specific tasks in a form of projects that eventually either built a product or help to make sure that the software is working correctly. The author called these workers a shadow workforce and their work as a Ghost Work. The author suggesting that 38% of the current work will turn into ghost work in the future because of the advancement in technology and automation. Companies like Amazon are assigning people tasks to people to manually inspect some content like hate speech in Twitter or inappropriate pictures. Such tasks can’t be automatically inspected by AI systems. The human being in the loop when the AI can’t do the perfect job. The way workers can get tasks is competitive. The workers must claim their tasks using a request dashboard and it is first in first serve. The mix between human and computer computation is called crowdsourcing, microwork, crowdwork, and human computation. The ghost work doesn’t follow employment laws, or any government regulation and it is cross boarders and the countries how do the work more are US and India because of English language fluency. Work can be micro tasks like deterring if a post on Facebook is a hate speech or macro work like ImageNET project that labeled millions of pictures to be used in AI systems. Ghost work is also known as on-demand gigs economy and there are 2.5 million adults in the United States that participate in this kind of work. Amazon was one of the first companies that depends of ghost work and have built MTruck software handle its work load to give work to thousands of people to correct spelling or typos or any incorrect information. Ghost work is important to label data to make better prediction for algorithms like machine learning, machine translation, and pattern recognition. As automation advances people might start losing their jobs but the ghost work will be more requested and that can balance out the need for the workforce.

Reflections and Connections

What I found interesting about the reading was the way people around the world are working together on tasks that eventually produce a product or service and to make the internet safe. My understanding was companies uses AI systems to flag content and then employees or contractors but didn’t know that people across the globe are working together on task. I also found interesting that the interaction between human and the AI systems will continue which can decrease the concerns that the machine will take over people job in the future.

I agree with what was mentioned in the paper regarding the fusion of the code and human and the future of the workforce as they will turn more to invisible or ghost work.

As GitHub acquired by Microsoft, I think the software development will also follow the same patterns by assigning tasks to software developers across the globe to build software modules that will participate in building large scale applications.

Questions

What things we need to put in our consideration while doing research in the AI area that might affect human in the loop?
How do we build software systems that can automatically take the feedback of manual inspection or tagging and adjust its behavior for similar incidents?
What is our role as graduate students or researchers to shape the future of the ghost work?

01/29/20 – Human Computation: A Survey and Taxonomy of a Growing Field

January 28, 2020January 29, 2020 Dylan Finch 1 Comment

Summary of the Reading

This paper is a survey of the research in the field of human computation. The paper aims to classify human computation systems so that the similarities between different projects and the holes in current research can be seen more clearly. The paper also explores related fields like crowdsourcing.

The paper starts by defining human computation as “a paradigm for utilizing human processing power to solve problems that computers cannot yet solve.” The paper then goes on to discuss the differences between human computation and crowdsourcing, social computing, data mining, and collective intelligence.

The paper then goes on to classify human computation systems based on 6 dimensions: motivation, quality control, aggregation, human skill, process orders, and task-request cardinality. Each of these dimensions has several discrete options. Putting all of these dimensions together allows for the classification of any arbitrary human computation system. The paper also provides examples of systems that have various values on each dimension.

Reflections and Connections

I think this paper provides a much needed tool for further human computation and crowdsourcing research. The first step to understanding something is being able to classify that thing. This tool will allow current and future researchers to classify human computation systems so that they can apply existing research to other, similar systems and also allow them to see where existing research falls short and where they need to focus future research.

This research also provides an interesting perspective on current human computation systems. It is interesting to see how current human computation systems compare to each other and what each system does differently and what they have in common.

I also like the malleability of the classification system. They say in the future work section that this system is very easy to add to. Future researchers who continue the work on this project could easily add values to each of the dimensions to better classify the human computation systems. They could also add values to the dimensions if new human computation systems are invented and need to be classified using this system. There are a lot of good opportunities for growth from this project.

One thing that I thought this paper was missing is a direct comparison of different human computation systems on more than one dimension. The paper uses human computation systems as examples in the various values for each of the dimensions of the classification system, but it doesn’t put these dimensions together and compare the human computation systems on more than one dimension. I think this would have added a lot to the paper, but it would also make for a great piece of future work for this project. This idea is actually very similar to the other paper from this week’s bunch, titled “Beyond Mechanical Turk: An Analysis of Paid Crowd Work Platforms” and I think it would be helpful and build on both of these papers.

Questions

Do you think the classification system presented in this paper has enough dimensions? Does it have too many?
What is one application you see for this classification system?
Do you think this classification system will help crowdsourcing platforms deal with some of their issues?

01/22/20 – Sushmethaa Muhundan – Ghost Work

January 28, 2020January 28, 2020 Sushmethaa Muhundan Leave a comment

Automation is taking over the world and a vast majority of people are losing their jobs! This might not necessarily be true. As the boom of automation and AI increases, there is a huge, silent effort from humans behind making this possible. Termed ghost workers, they are the people who “help” machines become smarter and make decisions that a machine is not capable of making. The paradox of automation’s last mile refers to humans who train AI, which ultimately goes on to function entirely on its own thereby making the human redundant. The phase in which human input is required to assist AI to become smarter is where a hidden world of ghost workers work day and night for meager pay. This human labor is often intentionally hidden from the outside world.

I agree with the author in that the current smart AI systems which are required to respond within seconds often require inputs from humans to help solve issues that are too complicated for their AI “mind”. Human discretion is almost always required to gauge sentiments correctly, identify patterns or adapt to the latest slang. These are things that the computers would not be able to decipher on their own and would require human intervention to solve real-time problems.

The consumers, however, remain until today, unaware of the fact that a human is actually involved in the transaction, behind-the-scenes. Worse, the conditions of work and pay are not widely known as well. This came as a shock to me. The current conditions could potentially lead to the isolation of ghost workers. Treating ghost workers as nothing more than a means to get a job done strips the job of any protection whatsoever and also dehumanizes the workers in the requester’s eyes. Therefore, I resonate with the author’s thoughts about bringing this to light and feel that a call for transparency is the need of the hour.

While the author is talking about the negative impacts of this form of ghost working, he also highlights why the workers prefer to do this job and why they return to the platform to search for jobs every single day. It is because of the anonymity that this platform provides by removing from the individual any attributes that might otherwise hinder their ability to seek jobs outside. Remote access is another aspect of ghost working that attracts people to engage in this work. A surprising fact I learnt from the extract is that the quality of the ghost workers often surpasses that of full-time employees (who get paid a lot more and also get benefits). Fear of losing subsequent tasks via the platform motivates these workers to raise the bars and deliver extremely high-quality work.

The nature of jobs posted is seasonal and depends on the requirements of the requesters. As mentioned in the extract, a few days bloom with requests while the rest of the days pass without a single request. Is there anything that can be done to streamline work so that the ghost workers are guaranteed work? Is there a way to provide alternate employment to the ghost workers that guarantees regular employment? This would be immensely helpful since the vast majority of ghost workers (if not all) depend on income from these sources to meet their living expenses.

Can task starvation be avoided by innovations?

While this platform indeed provides anonymity to the workers, why are the ghost workers not paid a fair amount? What can be done to ensure that a fair amount is distributed to the workers?