This week’s readings on crowdsourcing continue our discussion on ghost work last week. With the vision of what will the future crowd work be like, Kittur et al.’s paper “The Future of Crowd Work” discusses the benefits and drawbacks of crowd work and addresses the major challenges current crowdsourcing is facing up with. The researchers calls for a new framework that can potentially bring more complex, collaborative, and sustainable crowd work. The framework lays out major research challenges in 1) crowd work processes, including designing workflows, assigning tasks, supporting hierarchical structure, enabling real-time response, supporting synchronous collaboration, controlling quality; 2) crowd computation, including crowds guiding AIs, AIs guiding crowds, platforms; and 3) crowd workers, including job design, reputation, and motivation.
I feel this paper opens more questions than it answers. The vision for the future of crowd work is promising, however, with the high-level ideas provided by the researchers, how to achieve the goal is still unclear. I think there are two key questions worthy of discussion. Firstly, is complex crowd work really needed at the current stage of AI development or what type of complex and collaborative crowd work is in need and to what extent? This question links me to a recent talk provided by Yoshua Bengio, one of the “Godfathers of AI,” on NeurIPS 2019. Entitled “From System 1 Deep Learning to System 2 Deep Learning,” his talk addressed some problems of current AI development — System 1 deep learning — including but not limited to 1) require a large volume of training data to complete naive tasks; 2) poor in generalization among different datasets. It seems the current development of AI is in System 1 and there is still a long way to reach System 2 which requires higher level of cognition, out-of-generation and transferring ability. I think this can partially explain why a large portion of crowd work tasks are labeling or pattern recognition. For simple tasks like this, there seems no need to decompose the work. Currently, it is difficult for us to foresee how fast the AI development and how complex the required crowdsourcing tasks will be. In my opinion, a quantitative study on what portion of current tasks are considered as complex and an analysis of the trend would be useful for a better understanding of the crowd work at the current stage.
Secondly, complex, collaborative, and sustainable crowd work highly depends on the platforms. How to modify the existing crowd work platforms to support the future of crow works remains unclear. The organization and coordination of crowd workers across varying task types and complexity is still lack of consideration in the design and operation of existing platforms, even in large ones, such as AMT, ClickWorker, CloudFactory, and so forth. Based on the observations above, the following questions are worthy of further discussion.
- When do we need more complex, collaborative, and sustainable crowd work?
- How can existing crowd work platforms support the future of crowd work?
- What organizational and coordination structures can facilitate the crowd work across varying task types and complexity?
- How can existing platforms boost effective communication and collaboration on crowd work?
- How can existing platform support for effective decomposition and recombination of tasks, or design interfaces/tools for efficient workflow for complex work?
