This week’s readings on crowdsourcing continue our discussion on ghost work last week. With the vision of what will the future crowd work be like, Kittur et al.’s paper “The Future of Crowd Work” discusses the benefits and drawbacks of crowd work and addresses the major challenges current crowdsourcing is facing up with. The researchers calls for a new framework that can potentially bring more complex, collaborative, and sustainable crowd work. The framework lays out major research challenges in 1) crowd work processes, including designing workflows, assigning tasks, supporting hierarchical structure, enabling real-time response, supporting synchronous collaboration, controlling quality; 2) crowd computation, including crowds guiding AIs, AIs guiding crowds, platforms; and 3) crowd workers, including job design, reputation, and motivation.

I feel this paper opens more questions than it answers. The vision for the future of crowd work is promising, however, with the high-level ideas provided by the researchers, how to achieve the goal is still unclear. I think there are two key questions worthy of discussion. Firstly, is complex crowd work really needed at the current stage of AI development or what type of complex and collaborative crowd work is in need and to what extent? This question links me to a recent talk provided by Yoshua Bengio, one of the “Godfathers of AI,” on NeurIPS 2019. Entitled “From System 1 Deep Learning to System 2 Deep Learning,”  his talk addressed some problems of current AI development — System 1 deep learning — including but not limited to 1) require a large volume of training data to complete naive tasks; 2) poor in generalization among different datasets. It seems the current development of AI is in System 1 and there is still a long way to reach System 2 which requires higher level of cognition, out-of-generation and transferring ability. I think this can partially explain why a large portion of crowd work tasks are labeling or pattern recognition. For simple tasks like this, there seems no need to decompose the work. Currently, it is difficult for us to foresee how fast the AI development and how complex the required crowdsourcing tasks will be. In my opinion, a quantitative study on what portion of current tasks are considered as complex and an analysis of the trend would be useful for a better understanding of the crowd work at the current stage.

Secondly, complex, collaborative, and sustainable crowd work highly depends on the platforms. How to modify the existing crowd work platforms to support the future of crow works remains unclear. The organization and coordination of crowd workers across varying task types and complexity is still lack of consideration in the design and operation of existing platforms, even in large ones, such as AMT, ClickWorker, CloudFactory, and so forth. Based on the observations above, the following questions are worthy of further discussion. 

• When do we need more complex, collaborative, and sustainable crowd work?
• How can existing crowd work platforms support the future of crowd work?
• What organizational and coordination structures can facilitate the crowd work across varying task types and complexity?
• How can existing platforms boost effective communication and collaboration on crowd work?
• How can existing platform support for effective decomposition and recombination of tasks, or design interfaces/tools for efficient workflow for complex work?

Read More

Summary:

This paper surveys existing literature in crowdsourcing and human computation and outlines a framework consisting of 12 major areas of future work. The paper focuses on paid crowd work, as opposed to volunteer crowd work. Envisioning a future where crowd work is attractive to both requesters and workers requires considering work processes, crowd computation, and what crowd workers want. Work processes involves the various workflows, quality control, and task assignment techniques, as well as the synchronicity involved in doing the work itself. Crowd computation can involve crowds guiding AIs, or vice versa. Crowd workers themselves may have different motivations, require additional job support through tools, want ways to maintain a reputation as a “good worker”, and ways to build a career out of doing crowd work. To improve crowd work, it requires re-establishing career ladders for workers, improving task quality and design, and facilitating learning opportunities. The paper ends with a call for more research on several fronts to shape the future of crowd work: observational, experimental, design, and systems-related.

Reflection:

The distributed nature of crowd work theoretically allows anyone to do work from anywhere, at any time, and there are clear benefits to this freedom. On the other hand, this distributed nature also enforces existing power structures and facilitates the abstraction of human labor. This paper addresses some of these concerns with crowd work, and highlights the need for enabling on-the-job training and re-establishing career ladders. However, recent work has highlighted the long-term physical and psychological effects of doing crowd work [1,2]. For example, content moderators are often traumatized by the work that they do. Gray and Suri [3] also point out the need for a “commons” that provides a pool of shared resources for workers, along with a retainer model that values workers’ 24/7 availability. Yet, very few platforms do so, mostly due to weak labor laws. More work needs to be done investigating the broader, long-term and secondary effects of doing crowd work. 

Second, the paper highlights the need for human creativity and thought in guiding AI, but states that crowd work is analogous to a processor. This is not entirely correct, since a processor always produces the same output for a given input. On the other hand, the same (or different) human may not. This poses the potential for human biases to be introduced into the work that they do. For example, Thebault-Spieker et al. found that crowd workers are biased in some regards [4], but not others [5]. More work needs to be done to understand the impact of introducing creative, insightful, and—most importantly—unique human thought “in the loop.”

Finally, there is a tension between how society values those who do complex work (such as engineers, plumbers, artists, etc.), and the constant push towards the taskification, or “Uberization” of complex work (Uber drivers, contractors on Thumbtack and UpWork, crowd workers, etc.), where work is broken down into the smallest possible unit to increase efficiency and decrease costs. What does it mean for work to be taskified? Who benefits, and who loses? How do we value microwork? Can we value microwork the same as “skilled” work?

Questions:

1. Seven years later, is this the type of work you would want your children to do?
2. How do we incorporate human creativity into ML systems, without also incorporating human biases?
3. How has crowd work changed since this paper first came out?

References:

[1] Roberts, Sarah T. Behind the screen: Content moderation in the shadows of social media. Yale University Press, 2019.

[2] Newton, Casey. Bodies in Seats: At Facebook’s Worst-Performing Content Moderation Site in North America, one contractor has died, and others say they fear for their lives. The Verge. June 19, 2019. 

[3] Mary L. Gray and Siddharth Suri. Ghost Work.

[4] Jacob Thebault-Spieker, Daniel Kluver, Maximilian A. Klein, Aaron Halfaker, Brent Hecht, Loren Terveen, and Joseph A. Konstan 2017. Simulation Experiments on (the Absence of) Ratings Bias in Reputation Systems. Proceedings of the ACM on Human-Computer Interaction 1, CSCW: 101:1–101:25. https://doi.org/10.1145/3134736

[5] Jacob Thebault-Spieker, Loren G. Terveen, and Brent Hecht 2015. Avoiding the South Side and the Suburbs: The Geography of Mobile Crowdsourcing Markets. In Proceedings of the 18th Acm Conference on Computer Supported Cooperative Work & Social Computing (CSCW ’15), 265–275. https://doi.org/10.1145/2675133.2675278

Read More

While crowd work has the potential to support a flexible workforce and leverage expertise distributed geographically, the current landscape of crowd work is often associated with negative attributes such as meager pay and lack of benefits. This paper proposes potential changes to better the entire experience in this landscape. The paper draws inputs from organizational behavior, distributed computing and feedback from workers to create a framework for future crowd work. The aim is to provide a framework that would help build a culture of crowd work that is more attractive for the requesters as well as the workers and that can support more complex, creative and highly valued work. The platform should be capable of decomposing tasks, assigning them appropriately, motivating workers and should have a structured workflow that enables a collaborative work environment. Quality assurance is also a factor that needs to be ensured. Creating career ladders, improving task design for better clarity and facilitating learning are key themes that emerged from this study. Improvements along these themes would enable create a work environment conducive for both the requesters as well as workers. Motivating the workers, creating communication channels between requesters and workers, providing feedback to workers are all means to achieve this goal.

Since the authors were requesters themselves, it was nice to see that they sought to get the perspectives of the current workers in order to take into account both the parties’ viewpoints before constructing the framework. An interesting comparison of the crowdsourcing market has been made to a loosely coupled distributed computing system and this helped build the framework by drawing an analogy to solutions developed to similar problems in the distributed computing space. I liked the importance given to feedback and learning which are components of the framework. I feel that feedback is of extreme importance when it comes to improving one’s self and this is not prevalent in the current ecosystem. As for learning, I feel that personal growth is very essential in any working environment and a focus on learning would facilitate self-improvement which in turn would help them perform subsequent tasks better. As a result, the requesters are benefitted since the crowd workers are more proficient in their work. I particularly found the concept of intertwining AIs guiding crowds and crowds guiding AIs extremely interesting. The thought of leveraging the strengths of both AI and humans to strengthen the other is intriguing and has great potential if utilized meaningfully.

• How can we create a shift in the mindset of the current requesters who get their work done for meager pay to actually change their viewpoint and invest in the workers by giving valuable feedback/spend time ensuring the requirements are well understood?
• What are some interesting ways that can be employed to leverage AIs guiding crowds?
• How can we prevent the disruption of quality by a handful of malicious users who collude to agree on wrong answers to cheat the system? How can we build a framework of trust that is resistant to malicious workers and requesters who can corrupt the system?

Read More

Reading: Aniket Kittur, Jeffrey V. Nickerson, Michael Bernstein, Elizabeth Gerber, Aaron Shaw, John Zimmerman, Matt Lease, and John Horton. 2013. The Future of Crowd Work. In Proceedings of the 2013 Conference on Computer Supported Cooperative Work (CSCW ’13), 1301–1318. https://doi.org/10.1145/2441776.2441923

What can we do to make crowd work better than the current state of simple tasks, to allow more complexity and satisfaction for the workers? The paper tries to provide a framework to improve crowd work in that direction. It does this through framing it in terms of 12 research directions that need to be studied so that they can be improved upon. The research foci are envisioned to promote the betterment of the current, less than stellar, sometimes exploitative nature of crowd work and make it into something “we would want our children to participate” in.

I like their parallels to distributed computing because it really is like that, trying to coordinate a bunch of people to complete some larger task by combining the results of smaller tasks. I work on distributed things so I appreciate the parallel they make because it fits my mental framework. I also find it interesting that one of the ways of quality control is to observe the worker’s process rather than just evaluating the output but it makes sense that evaluating the process allows the requester to maybe give guidance on what the worker is doing wrong and help improve the processes, whereas with just looking at the output, you can’t know where things went wrong and can only guess. I also think that their suggestion that crowd workers can move up to be full employees as somewhat dangerous because it seems to incentivize the wrong things for companies. I’m imagining a scenario where a company is built entirely on utilizing high level crowd work where they’re advertising that you have opportunities to “move up”, “make your own hours”, “hustle will reach the top”, where the reward is job security. I realize I just described what tenure track may be like for an academic. But that kind of incentive structure seems exploitative and wrong to me. This kind of set up seems normal because it may have existed for a long time in academia and prospective professors accept it because they are single mindedly determined (and somewhat insane) that they are willing to see this through. But I would hate for something like that to become the norm everywhere else.

1. Did anyone feel like there was any avenue that wasn’t addressed? Or did the 12 research foci fully cover every aspect of potential crowd work research?
2. Do you think the idea of moving up to employee status on crowd work platforms as a reward for doing a lot of good work is a good idea?
3. What kind of off-beat innovations can we think of for new kinds of crowd platforms? Just as a random example – a platform for crowds to work with other crowds, like one crowd assigns tasks for another crowd and they go back and forth.

Read More

SUMMARY

This paper aims to define the future of crowd work in an attempt to ensure that future crowd workers will share the same benefits as those currently shared by full-time employees. The authors define a framework keeping in mind various factors such as workflow, assignment of tasks, real-time response to tasks, etc. The future that the paper envisions includes worker considerations such as providing timely feedback, and job motivation as well as requester considerations such as quality assurance and control, task decomposition. The research foci mentioned in the paper broadly consider the future of work processes, integration of crowd work and computation, supporting the crowd workers of the future in terms of job design, reputation and credentials, motivation and rewards. With respect to the future of crowd computation, the paper suggests a hybrid human-computer system that would capitalize on the best of both human intelligence and machine intelligence. The authors mention two such strategies – crowds guiding AI and AIs guiding crowds.  As a set of future steps that can be undertaken to ensure environment for crowd workers, the authors describe three design goals – creation of career ladders, improving task design through better communication, facilitating learning.

REFLECTION

I found it interesting to learn about the framework proposed by the authors in order to ensure a better working environment in the future for crowd workers. I like the structure of paper wherein the authors mentioned a brief description about the research foci followed by some prior work and then some potential research that can be performed in each of these foci.

I particularly liked the set of steps that the authors proposed – such as the creation of a career ladder. I believe that the creation of such a ladder, will help workers stay motivated as they will have the ability to work towards a larger goal as promotions can be a good incentive to foster a better and more efficient working environment. I also found it interesting to learn how often times, the design of the tasks cause ambiguity which makes it difficult for the crowd workers to perform their tasks well. I think that having sample tests of these designs with some of the better performing workers (as indicated in the paper) is a good idea as it will allow the requesters to gain feedback on their task design since many of the requesters may not realize that these tasks are not as easy to understand as they might believe.

QUESTIONS

1. While talking about crowd-specific factors, the authors mention how crowd workers can leave tasks incomplete with fewer repercussions as compared to traditional organizations. Perhaps having a common reputation system in order to maintain a history of employment (associated with some common ID) in order to maintain recommendation letters, work histories might help to keep track of all the platforms with which a crowd worker was associated as well as their performance?
2. Since the crowd workers interviewed were from Amazon Mechanical Turk alone, wouldn’t the responses collected from the workers as part of this study be biased? The opinion these workers would give would be specific to AMT alone and these opinions might be different among workers that are part of different platforms.
3. Do any of these platforms perform a thorough vetting for the requesters? Have any measures been taken to move towards the development of a better system in order to ensure that the tasks posted by requesters are not harmful/abusive in nature (CAPCTHA solving, reputation manipulation, etc)?

Read More

Summary

The paper discusses crowd work and was presented at CSCW (Computer-supported cooperative work) in 2013. It proposes a framework that takes ideas from organizational behavior and distributed computing along with workers’ feedback. The authors of the paper consider the crowd sourcing platform to be a distributed system’s platform where each worker is considered to be analogous to a node in distributed system. This would help in partitioning tasks like parallel computing does. The ways shared resources can be managed, and allocation is also discussed well in this paper. The paper provides deep analysis on the kind of work crowd workers end up doing, the positives and the negatives of such work.

The paper outlines and identifies 12 research areas that form their model. This takes into account broadly, the future of crowd work processes, crowd computation and the crowd workers. Each of the broad topics addressed various subtopics from quality control to collaboration between workers. The paper also talks about how to create leaders in such systems, the importance of better communication and that learning, and assessment should be an integral part of such systems.

Reflection

It was an interesting read on the future of the crowd work. The approach to define the system as a distributed system was fascinating and a novel way to look at the problem. Workers do have a capability to act as “parallel processors” which make the system more efficient and would enable to do intensive tasks (like application development) effectively. Implementing theories from organizational behavior is interesting that it allows the system to better manage and allocate resources. The authors address various subtopics that talk about various issues in depth. It was a very informative read on where they incorporated background work on each of the research areas. I will be discussing some of the topics or problems that stood out to me.

Firstly, they spoke about processes. Assignment of work, management turns out to be a challenging task. In my opinion, a universal structure or hierarchy is not the way to go. In certain kinds of work or tasks it is needed to have a structure where hierarchy would prove to be useful. Work like software development, would benefit from a structure where the code is reviewed, and the quality is assessed by a separate person. Such work also needs a synchronous as people might have tasks dependent on each other.

Secondly, the paper discussed the future of crowd-computation. This included the discussion of AIs and how they can be used in the future to guide crowd working. AI in recent years have proved to be an important tool. Automatic text summarization can be used to help create “Gold standards”. Similarly, other NLP techniques could very well be used to extract information, annotate, summarize and provide other automatic services that can be used to integrate with the current human framework. This would create a human-in the loop system.

Lastly, the future of crowd workers is also an important topic to ponder. Crowd workers are often not compensated well. Similarly, requesters are often delivered sub-par work. The paper did mention that background verification is not always done properly for such “on-demand worker” as it is done for full-time employees from transcripts to interviews. This is a challenge. However, on-demand workers can be validated like Coursera does to validate students. They can be asked to upload documents for tasks that require specialization. This is in itself a task that can be carried out by contractors who verify documentation or create a turk job for the same.

Overall, this was an interesting read and research should be conducted in each of the areas to see how the system and work improves. It has the potential to create more jobs in the future with recruiters being able to hire people instantaneously.

Questions

1. The authors only considered AMT and ODesk to define the framework. Would other platforms (like Amara, LeadGenuis) have greater/lesser issue which differ from the current needs?
2. They mentioned about “oDesk Worker Diary” which takes snapshots of workers’ computer screen. How is the privacy and security addressed?
3. Can’t credentials be verified digitally for specialized tasks?

Read More