Summary 

The paper talks about breaking larger tasks into smaller sub-tasks and then evaluating the performance of such systems. Here, the authors approach of dividing a large piece of work, mainly online work, into smaller chunks which would then use crowdworkers to perform the required tasks. The authors created a prototype system called “Knowledge Accelerator”. Its main goal is to use crowdworkers and help and find answers to open-ended, complex questions. However, the workers would only see part of the entire problem and work on a small amount of the task. It is mentioned that the maximum payment for any one task was $1. This gives an idea about how granular and simple tasks the authors wanted the crowdworkers to accomplish. The experiment was divided into two phases. In the first phase, the workers had to label some categories which were later used in the classification task. The second phase, on the other hand, required the workers to clean the output the classifier produced. This task involved the workers looking at the existing clusters and then tagging the new clips into an existing or a new cluster.

Reflection

I liked the way the authors approach the problem by dividing a huge problem into smaller manageable parts which in-turn becomes easy for workers to annotate. For our course project, we initially wanted the workers to read an entire chapter from an electronic thesis and dissertation and then label the department from which they think the document should belong to. We were not considering the fact that such a task is huge and would take a person around 15-30 minutes to complete. Dr. Luther pointed us in the right direction, where he asked us to break the chapter in parts and then present it to the workers. The paper also mentioned that too much context for workers could prove to be confusing. We can further decide better on how to divide the chapters so that we provide just the right amount of context.

I liked how the paper mentioned their ways of finding the sources, the filtering, and clustering techniques. It was interesting to see the challenges that they encountered while designing the task. This portion helps future researchers in the field to understand the mistakes and the decisions the authors took. I would view this paper as a guideline on how to best break a task into pieces so that it is easy as well as detailed enough for Amazon Mechanical Turkers. 

Finally, I would like to point out that it was mentioned in the paper that only workers from the US were only considered. The reason was also mentioned in the footnote, that because of currency conversion, the value of $ is relative. I thought this was a very thoughtful point to add and bring light to. This helps maintain the quality of the work involved. Although, I think a current currency converter (API) could have been incorporated to compensate accordingly. Since the paper deals with searching for relevant answers for complex questions, involving workers from other countries might help improve the final answer. 

Questions

1. How are you breaking a task into sub-tasks for the course project? (We had to modify our task design for our course project and divide a larger piece of text into smaller chunks)
2. Do you think that including workers from other countries would help improve the answers? (After considering the currency difference factor and compensating the same based on the current exchange rate.)
3. How can we improve the travel-related questions? Would utilizing workers who are “travel-enthusiasts or bloggers” improve the situation?

Note: This is an extra submission for this week’s reading.

Read More

Summary

Hahn’s paper “The Knowledge Accelerator: Big Picture Thinking in Small Pieces” utilizes a distributed information synthesis task as a probe to explore the opportunities and limitations of accomplishing big picture thinking by breaking it down into small pieces. Most traditional crowdsourcing work targets simple and independent tasks, but real-world tasks are usually complex and interdependent, which may require a big picture thinking. There are a few current crowdsourcing approaches that support the breaking-down of complex tasks by depending on a small group of people to manage the big picture view and control the ultimate objective. This paper proposes the idea that a computational system can automatically support big picture thinking all through the small pieces of work conducted by individuals. The researchers complete the distributed information synthesis in a prototype system for and evaluate the output of the system on different topics to validate the viability, strengths, and weaknesses of their proposed approach.

Reflection

I think this paper introduces an innovative approach for knowledge collection which can potentially replace a group of intermediate moderators/reviewers with an automated system. The example task explored in the paper is to answer a given question by collecting information in a parallel way. That relates with the question about how the proposed system enhances the quality of answer by a structured article compiled with the pieced information collected. To facilitate the similar question-answer task, we actually have a variety of online communities or platforms. Take Stack Overflow for example, it is a site for enthusiast programmers to learn and share their programming knowledge. A large number of professional programmers answer the questions on a voluntary basis, and usually a question would receive several answers detailing different approaches with the best solution on the top with a green check. You can check other answers as well in case you have tried one but that does not work for you. I think the variety of answers from different people sometimes enhance the possibility the problem can be solved. Somehow the proposed system reduces that kind of diversity in the answers. Also, one informative article is the final output of the system to a given question, then its quality would be important, but it seems hard to control the vote-then-edit pattern without any reviewers to ensure the quality of the final answer.

In addition, we need to be aware that much work in the real world can hardly be conducted via crowdsourcing because of the difficulty in decomposing tasks into small, independent units, and more importantly, the objective is beyond to accelerate the computational time or collect complete information. For creative work such writing a song, editing a film, designing a product, the goal is more like to encourage creativity and diversity. In those scenarios, even with a clear big picture in minds, it is very difficult to put together the small pieces of work by a group of recruited crowd workers to create a good piece of work. As a result, I think the proposed approach is limited to comparatively less creative tasks where each piece can be decomposed and processed in an independent way.

Discussion

I think the following questions are worthy of further discussion.

• Do you think the proposed system can completely replace the role of moderators/reviewers in that big picture? What are the advantages and disadvantages?
• This paper discusses the proposed system in the task of question-answer. What are the other possible applications the system could be helpful?
• Can you think about any possible aspect of improving the system to scale it up to other domains or even non-AI domains?
• Do you consider the breaking-down approach in your course project? If yes, how would you like to approach that?

Read More

Summary

Most of the crowdsourcing tasks in the real world are submitted to platforms as one big task due to the difficulty in decomposing tasks into small, independent units. The authors argue that by decomposing and distributing tasks we could utilize more of the resources provided by crowdsourcing platforms at a lower cost than the existing traditional method. They propose a computational system that can frame interdependent, small tasks to represent one big picture system. This proposal is difficult, so to investigate its viability, the authors prototype the system to test the distributed information combination after all tasks are done and to evaluate the output across multiple topics. The system compared well to existing top information sources on the web and it exceeded or approached quality ratings for highly curated reputable sources. The authors also suggested some design patterns that should help other researchers/systems when thinking of breaking big picture projects into smaller pieces. 

Reflection

This was an interesting paper. I haven’t thought about breaking down a project into smaller pieces to save on costs or that it would get better quality results by doing so. I agree that some of the existing tasks are too big, complex, and time consuming and maybe those need to be broken down to smaller tasks. I still can’t imagine how breaking tasks so small that they can’t cost more than $1 generalizes well to all existing projects that we have on Amazon MTurk.  

The authors mention that their system, even though it has a strong performance, was generated by non-expert workers that did not see the big picture, and that it should not be thought of as a replacement to expert creation and curation of content. I agree with that. No matter how good the crowd is, if they’re non-experts and they don’t have full access to the full picture, there would be some information missing which could lead to mistakes and imperfection. That shouldn’t be compared to a domain knowledge expert who would do a better job even if it costs more. Cost should not be a reason we favor the results of this system.

The design patterns suggested were a useful touch and the way they were explained help in understanding the proposed system as well. I think that we should adapt some of these design patterns as best as we could in our projects. Learning about this paper late enough in our experiment design would make it hard to implement breaking our tasks down to simpler tasks and test that theory on different topics. I would have loved to see how we each reported since we have an array of different experiments and simplifying some tasks could be impossible. 

Discussion

• What are your thoughts on breaking down tasks to such a small size?
• Do you think that this could be applicable to all fields and generate similarly good quality? If not, where do you think this might not perform well?
• Do you think that this system could replace domain experts? Why? Why not?
• What applications is this system best suited for?

Read More

Summary:
“The Knowledge Accelerator: Big Picture Thinking in Small Pieces” by Hahn et. al. talks about interdependent pieces of crowdsourcing. Most of the crowdsourcing tasks involve relying on a small number of humans to complete all the tasks required to complete a big picture. For example, most of the work in Wikipedia is done by a small number of highly invested editors. The authors bring up the idea of using a computational system such that each individual sees only a small part of the whole. This is a difficult task as much of the real-world tasks cannot be broken down into small, independent units and hence, the Amazon Mechanical Turk (AMT) cannot be used efficiently as it is used for prototyping. Also, it is a challenging problem to maintain the coherency of the overall system while breaking down the big task into smaller and mutually independent chunks for the crowd workers to work on. Moreover, the quality of the work being done is also dependent on the division of the tasks into coherent chunks. The authors present their idea which mitigates the need for a big picture view by a small number of workers, by ensuring small contributions by the individuals who see only a small chunk of the whole.

Reflections:
This is an interesting work as it talks about the advantages and the limitations of breaking down big tasks into small pieces. They built a prototype system called “Knowledge Accelerator” which was constrained such that no single task would amount for more than $1. Although the authors used this metric for tasks division, I’m not sure if this is a good enough metric to judge the independence and the quality of the small task. Also, the authors mention that the system should not be seen as a replacement for expert creation and curation of content. I disagree with this as I feel that with some modifications to the system, the system has the potential and might be able to completely replace humans for this task in the future. As is, the system has some gaping issues. The absence of a nuanced structure in the digests is problematic. It might also help to include iterations in the system after the workers have completed a part of their tasks and require more information. Finally, the authors would benefit by taking into account the cost of producing these answers on a large scale. The authors could use a computational model to dynamically decide how many workers and products to use at each stage such that the overall cost is minimized. The authors can also check if some of the answers could be reused across questions and across users. Incorporating contextual information can also help improve the system significantly.

Questions:
1. What are your thoughts on the paper?
2. How do you plan to use the concepts present in the paper in your project?
3. Are you dividing your tasks into small chunks such that crowd workers only see a part of the whole?

Read More

Summary

In this paper, the authors aim to provide a framework to deconstruct complex systems into smaller tasks that can be easily managed and done by crowd-workers without need of supervision. Currently, crowdsourcing is predominantly used for small tasks within a larger system dependent on expert reviewers/content managers. Knowledge Accelerator provides a framework to build complex systems solely based on small crowdsourcing tasks.

The authors argue that major websites like Wikipedia depend on minor contributors but require an expensive network of dedicated moderators and reviewers to maintain the system. They eliminate these points by a two phase approach: inducing structure and information cohesion. Inducing structure is done through collecting relevant web pages, extracting relevant text and creating a topic structure to encode the clips to the categories. The information cohesion is achieved by crowd-workers gathering information and improving sections of the overall article without global knowledge and adding relevant multimedia images. 

Reflection

The paper introduces a strategy for knowledge collection that completely removes the necessity for any intermediate moderator/reviewer. KA shows the potential of unstructured discussion forums as sources of information. Interestingly, this is exactly the end goal of my team’s course project. The idea of small-scale structure collection from multiple crowd-workers without any of them having context to the global article is generalizable to several areas such as annotating segments of large geographical images, annotating segments of movies/speech and fake-news detection through construction of an event timeline.

The paper introduces itself as a break-down strategy for all complex systems into simpler tasks that can be crowdsourced. However, it settles into the problem of structure and collection. E.g. Information structures and collection are not enough for jobs that involve original creation such as softwares, network architectures, etc. 

The system heavily relies on crowd-sourcing tasks. Some modules have effective AI counterparts. E.g. Inducing topical structure, searching relevant sources of information and multimedia components. I think a comparative study would help me understand the reasons for the decision. 

The fact that Knowledge Accelerator works better than search sites opens up new venues of exploration that collect data by inducing structure in various domains. 

Questions

1. The paper discusses the framework’s application in Question-Answering. What are the other possible applications in other domains? Do you see an example application in a non-AI domain?
2. I see that the proposed framework is only applicable to collection of existing information. Is there another possible application? Is there a way we can create new information through logical reasoning processes such as deduction, induction and abduction?
3. The paper mentions that some crowd-work platforms allow complex tasks but require a whetting period between workers and task providers. Do you think a change in these platforms would help? Also, in traditional jobs, interviews enable similar whetting. Is it a waste of time if quality of work improves?
4. I found my project similar to the framework in terms of task distribution. Are you using a similar framework in your projects? How are you using the given ideas? Will you be able to integrate this in your project?

Word Count: 524

Read More

Summary

In this paper, the authors try to help crowdsource workers to have a big picture of the system that the small tasks assigned to them try to accomplish which can help in executing the task more efficiently to have a better contribution on achieving the big goal. The work also tries to help companies to remove the bottleneck caused by a small number of people who normally know or maintain the big picture and can cause serious risks if these individuals leave the company. The authors designed and developed a system known as Knowledge Accelerator that can be used by crowdsource worker to answer a given question and allow them to use relevant resources to help them answering the question in a big picture context without the need to a moderator. The system starts by asking the workers to choose different web pages related to the topic then extract the relevant information, then cluster the information on different categories. The system then integrate the information by drafting an article, then allow the editing of the article, and finally add supporting images or videos that are related to the article, and that way the system can help the crowdsource worker to know the big picture of the system and complete the task in a way that can achieve the big picture and goal of the system.

Reflection

I found the study mentioned in the paper to be very interesting. I agree with the authors that a lot of tasks that are done by crowdsource workers are simple and it is hard to divide complex tasks that can require the knowledge of the big picture. Knowing the big picture is very important and often known be very few people who are normally in technical leadership positions and losing them might cause serious issues.

I like the way the system is designed to provide a high-level information about the system while working on a small tasks. The pipeline and multi-stage operations used in the system to generate a cohesive article can help the workers to achieve the goal and also to know more information about the topic.  

This approach can also be used when building large scale systems where many components need to be built by developers, and often these developers don’t know what the system is trying to accomplish or try to solve. Normally developers work on a specific task to add an employee table or building a web service endpoint that can receive a request and send back a response without knowing who will be using the system or what will be the impact of their tasks on the overall system. I think we can use such system to help developers to understand the big picture of the system which can help them to solve problem on a way that can make a greater impact on the big picture or big problem that the system is trying to solve.

Questions

• The system developed by the authors can help with generating articles about a specific topic, can we use the system to solve different problems?
• Can we use this approach in software development to help developers understand the big picture of the system that they are trying to build especially when building large systems?
• Can you think of a way to use a similar approach in your course project?

Read More

Summary

Maintaining a public and open source website can be difficult since the website is supported by individuals that are not paid. This team investigated using a crowd sourcing platform to not only support the platform but create articles. These articles (or tasks) were broken down into micro tasks there were manageable and scalable by the crowd sourced workers. These tasks were integrated throughout other HITs and were given extra contributions in order to relieve any extra reluctance on editing other crowd workers work. 

Reflection

I appreciate the competitive nature of comparing both the supervised learning (SL) and a reinforcement learning (RL) in the same type of game scenario of helping the human succeed by aiding the as best as it can. However as one of their contributions, I have issue with the relative comparison between the SL and RL bots. Within their contributions, they explicitly say they find “no significant difference in performance” between the different models. While they continue to describe the two methods performing approximately equally, their self-reported data describes a better model in most measurements. Within Table 1 (the comparison of humans working with each model), SL is reported as having a better (yet small) increase and decrease in Mean Rank and Mean Reciprocal Rank respectively (lower and then higher is better respectively). Within Table 2 (the comparison of the multitude of teams), there was only one scenario where the RL Model performed better than the SL Model. Lastly even in the participants self-reported perceptions, the SL Model only decreased performance in 1 of 6 different categories. Though it may be a small decrease in performance, they’re diction downplays part of the argument their making. Though I admit the SL model having a better Mean Rank by 0.3 (from Table 1 MR difference or Table 2 Human row) doesn’t appear to be a big difference, I believe part of their contribution statement “This suggests that while self-talk and RL are interesting directions to pursue for building better visual conversational agents…” is not an accurate description since by their own data it’s empirically disproven. 

Questions

• Though I admit I focus on the representation of the data and the delivery of their contributions while they focus on the Human-in-the-loop aspect of the data, within the machine learning environment I imagine the decrease in accuracy (by 0.3 or approximately 5%) would not be described as insignificant. Do you think their verbiage is truly representative of the Machine Learning relevance? 
• Do you think more Turk Workers (they used data from at least 56 workers) or adding requirements of age would change their data? 
• Though evaluating the quality of collaboration is imperative between Humans and AI to ensure AI’s are made adequately, it seems common there is a disparity between comparing that collaboration and AI with AI. Due to this disconnect their statement on progress between the two collaboration studies seems like a fundamental idea. Do you think this work is more idealistic in its contributions or fundamental? 

Read More