Summary
Most of the crowdsourcing tasks in the real world are submitted to platforms as one big task due to the difficulty in decomposing tasks into small, independent units. The authors argue that by decomposing and distributing tasks we could utilize more of the resources provided by crowdsourcing platforms at a lower cost than the existing traditional method. They propose a computational system that can frame interdependent, small tasks to represent one big picture system. This proposal is difficult, so to investigate its viability, the authors prototype the system to test the distributed information combination after all tasks are done and to evaluate the output across multiple topics. The system compared well to existing top information sources on the web and it exceeded or approached quality ratings for highly curated reputable sources. The authors also suggested some design patterns that should help other researchers/systems when thinking of breaking big picture projects into smaller pieces.
Reflection
This was an interesting paper. I haven’t thought about breaking down a project into smaller pieces to save on costs or that it would get better quality results by doing so. I agree that some of the existing tasks are too big, complex, and time consuming and maybe those need to be broken down to smaller tasks. I still can’t imagine how breaking tasks so small that they can’t cost more than $1 generalizes well to all existing projects that we have on Amazon MTurk.
The authors mention that their system, even though it has a strong performance, was generated by non-expert workers that did not see the big picture, and that it should not be thought of as a replacement to expert creation and curation of content. I agree with that. No matter how good the crowd is, if they’re non-experts and they don’t have full access to the full picture, there would be some information missing which could lead to mistakes and imperfection. That shouldn’t be compared to a domain knowledge expert who would do a better job even if it costs more. Cost should not be a reason we favor the results of this system.
The design patterns suggested were a useful touch and the way they were explained help in understanding the proposed system as well. I think that we should adapt some of these design patterns as best as we could in our projects. Learning about this paper late enough in our experiment design would make it hard to implement breaking our tasks down to simpler tasks and test that theory on different topics. I would have loved to see how we each reported since we have an array of different experiments and simplifying some tasks could be impossible.
Discussion
- What are your thoughts on breaking down tasks to such a small size?
- Do you think that this could be applicable to all fields and generate similarly good quality? If not, where do you think this might not perform well?
- Do you think that this system could replace domain experts? Why? Why not?
- What applications is this system best suited for?
