Summary of the Reading

This paper looks to build a model of crowd work to better understand some of the major challenges that the future of crowd work will face. The paper also collects data and makes suggestions on how to make the future of crowd work better for all those involved. 

One of the main contributions of this paper is building a model to represent crowd work. They do this by using not only theoretical data, but also empirical data that is gathered from both requesters and workers on crowd work websites. This model helps the authors to visualize some of the problems that come from crowd work and allow them to better understand possible solutions.

The paper then goes into depth on 12 research foci that are part of the model. They consider the future of the crowd work processes, crowd computation, and crowd workers. After this, they synthesize these foci into concrete recommendations for the future of the industry, including creating career ladders, improving task design and communication, and facilitating learning.

Reflections and Connections

I think that this paper looks at a very important part of crowd work. Crowd work is a very powerful tool, but if we don’t do it right, it could lead to a terrible future where work is cheepened and workers are not respected. We need to make sure that while this industry is still young that it is shaped in a way that will allow it to be sustainable well into the future. Articles like these are important so that we can know what we need to change now, when the industry is still young so that when it grows up, it can be a great part of the economy. 

I also like that the article has a very optimistic view of crowd work. The technology is very powerful and can allow people to do things that never could have been done without it, or could only have been done at a tremendous cost. If we put in the work to make this industry work well for its users, then we will be able to use crowd work to accomplish many great things and many people will be able to use it to make ends meet or to get an extra income on their own time. Crowd work unlocks new, more efficient ways to accomplish many old goals. 

I also think that the goals laid out in this article will be very good for the industry. The idea of creating career ladders would be extremely beneficial to everyone involved. Many people in traditional jobs only try their best because of the possibility of moving up in the company. So, a job where you can’t move up will make people less invested and less motivated. The idea of introducing better communication would also be great for the industry. There will always be times when a task is poorly worded and a quick email can save hours of wasted time and effort. The facilitation of learning also seems extremely beneficial to everyone involved and would also increase the quality of tasks and results. 

Questions

1. Considering this article was published in 2013, do you think the industry is moving in the right direction?
2. How practical do you think these recommendations are? Are they too ambitious?
3. Which of the recommendations do you think is most important for the industry to implement? 

Read More

SUMMARY

‘Ghost work’ is a new world of employment that encompasses the work done behind-the-scenes by various unnamed online workers that help to power mobile apps, websites, and AI systems. These workers are the ‘humans-in-the-loop’ contributing their creativity, innovation, and rational judgement to cases where machines are unable to make decisions accurately. The importance of these ‘gigs’ has increased exponentially over the past few years – especially in order to provide better training data for Machine Learning systems (such as the ImageNet challenge that involved annotating millions of images), cleaning-up social media pages (ensuring that there is little to no abusive content), as well as providing accurate descriptions for product and restaurant reviews. This work covers various online platforms like Amazon’s Mechanical Turk, Microsoft’s UHRS, LeapGenius, and Amara – each catering to slightly varied tasks ranging from micro-tasks (tasks that can be done quickly, but require many people) to macro-tasks (larger projects such as copyediting a newsletter, linking video to captions). The work also illustrates the lives of four such ghost workers, and their experiences on these platforms, giving some insight into the ‘human’ behind the seemingly interchangeable worker represented by a pre-defined worked ID.

REFLECTION

As a researcher working with Machine Learning, I think that I often forget the ‘human-side’ of Machine Learning – especially with respect to the generation of labels for supervised tasks. I found the description about the lives of Joan, Kala, Zaffar, and Karen to be particularly interesting and insightful. This was interesting to me as it helped me better understand the motivation of people engaging in crowd work and helped me better appreciate the efforts that need to be taken in order to gain a good ‘gold standard’. I also found it interesting to learn that on-demand workers produce higher-quality work when compared to the quality of work produced by full-time employees. While the reading posits that a potential reason for this is the competitive nature of on-demand jobs, I wonder if incentive-based bonuses in the case of full-time employees could have an impact on the quality of work provided by these employees.

The reading reminded me of another paper entitled ‘Dirty Jobs: The Role of Freelance Labor in Web Service Abuse’ which focusses on freelance work done via Freelancer.com. This paper discusses the abuse of various web services including solving CAPTCHAs, Online Social Networks Linking, etc. This reading talks about vetting processes done for the workers, but it made me wonder about the type of vetting done for ‘requesters’ and the moderation of the type of tasks that are posted on such ghost work platforms.

QUESTIONS

• What is the main motivation for ghost workers with a Master’s degree to work on these platforms? Is it generally due to an ailing family member? Additionally, Amara has a larger percentage of women, is there any reason behind this?
• In the case of a platform such as Amara, where workers perform tasks in teams, how do they handle cases of harassment (if any were to occur)? Are they any policies that exist to deal with such situations?
• Is there any form of moderation of the type of tasks that are posted on these sights? Are the ghost workers allowed to flag tasks that might seem to contribute to web service abuse? (For example, multiple account creation by using ghost workers to solve CAPTCHAs in real-time).

Read More

Summary

            Ghost work’s introduction and first chapter cover a birds-eye look at the new gig economy of crowd intelligence tasks. They cover several anecdotes of people working on these tasks in various situations, mostly in the United States and India. The introduction also wants to get across that these types of tasks are for problems that AI can’t solve or needs training to be able to solve. The text also tries to impart that there is nothing to fear from AI – automation has always happened through new technologies and that there will always be more work generated by the blind spots of the newer technologies. The first chapter then goes through several different scenarios for these workers, starting with the worst working conditions and then working up to the “best.” Lastly, it pointed out that there are possible moral issues with the whole setup, using a lawsuit of workers for a specific company arguing that they were essentially being paid minimum wage working full time with no benefits.

Personal Reflection

            I thought it was interesting to better understand how these gig workers came into being and why they’re needed.  However, I couldn’t stop thinking about the human element. Yes, the text seems to drive you towards that direction, but it’s not until the last 2-3 pages of the book that it ever actually asks the question “Is this right?” The first few specific anecdotes in the first chapter were chilling – these were people with not just undergraduate degrees but post-graduate degrees who were working for a paycheck that put them below the poverty line. Arguably only the last 2 companies mentioned, Upwork and Amara, were even close to acceptable living conditions. LeadGenius was close, as there were tiers and “promotions” that could be earned through working, but they still seemed to pay very little for quality work. MTurk being the worst really outshone the others.  A talented worker as the example was, she was only earning $16,000 a year working (according to the intro) 10 hours a day, and she was happy it was better than the $4,400 she earned the first year. The text even (insultingly) points out $4400 is more than earning $0. Futhermore, working as an mTurk required additional work to figure out what good HITs would open and what requesters to avoid as well as learn the tips and tricks of the trade. At least in a Starbucks you get paid the full amount during your training. This text and the whole ghost work gig-economy industry feels like share-cropping, where the workers are cheated out of their proper valuation.

Questions

1. How could the ghost-work/gig-economy be regulated? Is self-regulation as shown by the mechanical turk forums and reddits enough?
2. Now knowing what life is like for these workers, could you ethically use this service? Are the rosy-stories of “I can fill in gaps on my resume” or “I can’t work standard hours so the flexibility is nice” or “I have another source of income so this is just free money” enough to counteract the underpayment of these workers?
3. Which of the businesses that setup gig-workers seems like the best tradeoff for requesters and workers? Why?
4. What do you think about the idea of having to screen “employers” on Mechanical Turk? How can this impact the pay rate?

Read More

Paper: Ghost Work, How to Stop Silicon Valley from Building a New Global Underclass

Summary: 

The book highlights one of the major players of the artificial intelligence realm that is the “people.” By people, we often think about the customers. However, ghost workers are people in the system who work behind the scenes. These are on-demand workers who are working to keep the system running without any hiccups. The work is unique for these people. Any company who needs a certain task that requires humans to complete, they can request for such services. The task can be anything from flagging adult content, verifying human credentials, labeling images to create training data. On-demand workers are individuals who are not considered full-time or hourly wage workers. They are paid according to the task at hand. Hence, they need to be vigilant to take up tasks as soon as they appear and not lose the job to others who are looking. Tasks can be classified as a micro and macro task. Micro-tasks are small tasks that take less amount of time, but lots of humans to complete the task. Macro-task, on the other hand, are larger projects like developing webpages or building apps. Some common platforms of on-demand works are Amazon Mechanical Turks, Universal Human Relevance System, Lead genius, and Amara.

Reflection 

The topic was fascinating and gave a unique perspective on “gig-workers.” I was aware of Amazon Mechanical Turks and that humans were an integral part of the automation process. However, this gave a more in-depth idea of what role humans play in this. It was intriguing to know how such individuals support themselves and their families. However, the lack of regulated law makes it difficult to accurately estimate how much such people earn and how exactly do they support their lives. 

One important thing to note here is for all of the companies about most of the workers were in the age-group 18-37, and most had a college degree. This means that younger people find it comfortable to use modern technology to depend on such a unique method of earning money. The workers are generally not paid well enough, and they often need to be extremely skilled to make this their sole livelihood. 

The book-chapters made it very clear that in various ways, such employees are an integral part of the technical system, and they are, at times, better performers than regular employees. Does this mean that companies can shift their work model from hiring more and more full-time employees to move to the on-demand model completely? That is still debatable. I do not think that such a model would be liable when it comes to long term commitments to deliver projects. Although, for full-time employees, job security might be the reason they are comfortable. However, having no financial guarantee, in the long run, is bound to be detrimental for a person. In my opinion, the need to perform “flawlessly” would cause people to reach their breaking point eventually. I firmly believe that such roles make people live an unhealthy work-life balance as they would continuously search for tasks to get more gigs.

Questions

1. Can use more “gig workers” prove to be efficient/profitable for companies?
2. Other than personal reasons, what makes people take up such jobs? Can someone aspire for such roles, or necessity drives them?
3. Would older people be able to do such tasks? If yes, how can we measure the efficiency and compare the productivity against the average workforce age of 18-37?

Read More

Summary

In the opening chapter of this book, the authors introduce the world of ghost workers, or gig workers, who have been thriving, almost unnoticed, in the shadows of modern-day software. Here, the authors give us an insight about how the gig economy has grown over the years, in terms of number of workers, where they come from, the different kinds of on-demand platforms and the features they offer, and the different kinds of jobs on these platforms. The highlight of these chapters is how these gig workers sit at the core of Artificial Intelligence (AI), either by providing training labels to build the AI models, or by filling when AI falls short of the job. The authors break down the hype of robots rising, and present the last-mile paradox of AI i.e. the race towards automation will never converge, nor take away all our jobs, but will result in the shifting of jobs from full-time employment towards gig work. They also draw historical analogies to factory work, piece work and outsourcing. Posting tasks for gig workers, instead of hiring people full-time, seems more appealing to companies and requesters for multiple reasons including automated hiring, evaluation and payment, low costs and overhead, and higher-quality work in a short time. The authors discuss the future of employment, which would look more like a world of ghost workers and not robots, and therefore, make the case for focusing our attention on improving this world of on-demand work.

Reflection

1. The hype about AI has been majorly propelled by our media [3]. Companies, especially the rising number of AI start-ups, are partly to blame, as they have to portray themselves as an AI start-up for getting investors onboard. As they explore complex open-ended problems, these standard AI systems are almost certainly bound to fall short, leaving room for the “last-mile” to be filled in by on-demand online workers [1].
   1. Essentially, the efforts of these ghost workers are passed off as the genius of AI, without any credit, and thus propels the hype further. This adds on to the “algorithmic cruelty” of the ghost work platform APIs. 
   2. Sometimes, companies acknowledge the contributions of the experts who are responsible for training and shaping their AI models, but fall short of acknowledging the inevitable presence of these human experts in a predominantly AI workflow [2]. This, once again, can be attributed towards a need to portray themselves as an AI company. 
2. Instead of the usual rhetoric about AIs taking over jobs, this book paints a realistic portrait about the future of employment, which involves a shift towards on-demand crowd work. This necessitates the need to redesign on-demand infrastructure with a goal of enhancing the quality of workers’ lives (illustrated in great detail in the later chapters). The chapters briefly discuss organizational hierarchy and worker collaboration, two important factors contributing towards LeadGenius’s success. These factors are also echoed by Kittur et al [4] as crucial for enhancing the value and meaning of crowd work. 
   1. It’s good to see politicians addressing issues related to gig economy [6, 7] and AI disrupting the traditional workplace [8]. 
3. The gig workers need to be hypervigilant in order to get more work, and earn more money. Employers no longer need to actively assign jobs to employees. Therefore, the cost is now borne through a worker’s time and effort to seek and eventually get the job. This is one area where algorithms could help by connecting workers to jobs on these on-demand platforms [9]. Going one step further, ML models can be used to automatically assign workers to tasks where they can contribute [4].  
4. The AI hype will inevitably blow up, and result in AI-infused systems to invariably rely on human intelligence for complementing (and completing) the shortcomings of AI. Human intelligence, employed either by on-demand labor or any other source, is an invaluable resource, and therefore, shouldn’t suffer the brunt of algorithmic cruelty. To address this, we need human-centered design of tools and platforms. Jeff Bigham, in his “The Coming AI Autumn” post, points out some areas where HCI research and practice could help in designing intelligent systems and making them useful for people. 
5. These chapters illustrate many real-world examples where humans are needed to “backfill decision-making with their broad knowledge of the world” to make up for the AI’s limitations e.g. spike in search terms during a disaster, identifying hate speech, finding a great springtime wedding venue, face verification. Partially to blame is the predominant use of artificial/synthetic datasets for training AIs, which calls for designing problems rooted in the real world. 

Questions

1. Let’s say, we have an organization with a human workforce using an AI system for solving a certain problem. The whole workflow involves a feedback channel from the human experts that is also used to train certain aspects of the AI systems, which going forward, may reduce the role of these human experts. Some of these experts may be AI critics as well. Should this organization be collecting data from these experts? If so, how should the workflow be designed? What are some of trade-offs?
2. The book chapters raises some interesting points about global labor arbitrage and localization of data. Most of the AI systems being built are almost always deficient of data coming in from places/regions/countries with lower labor costs, and therefore, may be biased towards non-US data (face recognition, speech translation, etc.). Why is that the case? How should this be addressed?
3. “The API isn’t designed to listen to Ayesha”. (How does Ghost Work Work?) Has anyone been on the receiving end of algorithmic cruelty? What kind of systems or intelligent user interfaces did you wish for, if any? 
4. How should journalists cover AI? How should AI claims be fact-checked? 

References

1. The rise of ‘pseudo-AI’: how tech firms quietly use humans to do bots’ work https://www.theguardian.com/technology/2018/jul/06/artificial-intelligence-ai-humans-bots-tech-companies
2. AI at the Speed of Real Time: Applying Deep Learning to Real-time Event Summarization https://www.dataminr.com/blog/ai-at-the-speed-of-real-time-applying-deep-learning-to-real-time-event-summarization
3. An Epidemic of AI Misinformation https://thegradient.pub/an-epidemic-of-ai-misinformation/
4. Kittur, A., Nickerson, J. V., Bernstein, M., Gerber, E., Shaw, A., Zimmerman, J., … & Horton, J. (2013, February). The future of crowd work. In Proceedings of the 2013 conference on Computer supported cooperative work (pp. 1301-1318).
5. AI and automation will disrupt our world — but only Andrew Yang is warning about it https://thehill.com/opinion/technology/469750-ai-and-automation-will-disrupt-our-world-but-only-andrew-yang-is-warning
6. Elizabeth Warren Takes On the ‘Gig Economy’ https://www.thenation.com/article/elizabeth-warren-takes-on-the-gig-economy/
7. Pete Buttigieg just called out Uber and McDonald’s for their treatment of workers — and said beefing up unions is the best way to protect them https://www.businessinsider.com/pete-buttigieg-plan-to-overhaul-the-gig-economy-2019-7
8. The Coming AI Autumn https://jeffreybigham.com/blog/2019/the-coming-ai-autumnn.html
9. Prolific. https://www.prolific.co/

Read More