Ross, Joel, et al. “Who are the crowdworkers?: shifting demographics in mechanical turk.” CHI’10 Extended Abstracts on Human Factors in Computing Systems. ACM, 2010.
Discussion Leader: Ananya Choudhury
Summary
This paper focusses on how demography of turkers is gradually shifting over a period of time. While previous research by Ipeirotis (2008) suggests worker population based primarily in United States, the study conducted in this paper reveals AMT marketplace becoming significantly international with Indians making up more than one-third of turking population. Based on different criteria like age, annual income, gender, and education this paper gives a comparison between Indian and US turkers. The study shows an increase in the number of highly educated, young, male Indian turkers compared to that in US. The results also tell that for most turkers (both in India and US) turking is an extra source of income, while for a significant number of Indian turkers turking is sometimes or always necessary to meet basic needs. Finally, based on these results the paper raises a few open-ended questions on ethics and authenticity of the data collected.
Reflection
This paper paints a pretty clear picture on who these turkers are. As mentioned in the paper, knowing turkers will help analyze survey results better. However I feel when data is collected in exchange of monetary benefits, the goal of a responder may shift from providing honest opinions to providing responses that will guarantee maximum benefits. Cultural background of turkers also play an important role. As mentioned in the paper, Indians are culturally more reluctant to present themselves as unemployed. If there is a tendency among Indians to not reveal their actual employment status, then the statistics provided in Figure 5 and 6 might not be accurate. This questions the credibility of turkers or data collected through mediums like AMT and analysis performed on these datasets. So knowing turkers may help analyzing data better, but if the knowledge per se is not accurate then how will it help the analysis?
Questions
Do you think knowing backgrounds of the workers will help researchers analysis data better?
Do you think collecting data in exchange of monetary benefits is the best way? What other ways can we devise so that we gather more genuine responses?
Is AMT or similar crowdsourcing mediums the right channel to collect survey data that cannot be validated? Can micro-volunteering be a better way to collect such information?
What is the reason that worker demography comprises of mostly Indians and Americans? Why is rest of the world still under-represented?
