Technology: Amazon Mechanical Turk (MTurk)
Demo Leader: Leanna Ireland
Summary:
Amazon Mechanical Turk (MTurk) is a crowdsource platform which connects requesters (researchers, etc.) to a human workforce to complete tasks in exchange for money. Requesters as well as workers can be located all over the world.
Requesters provide tasks and compensation for the workers. The tasks, or human intelligence tasks (HITs), can range from identifying photos and transcribing interviews to writing reviews and taking surveys. When creating a task, requesters can specify worker requirements, such as the number of HITs undertaken, percentage of HITs approved, or a worker’s location. Other qualifications can be specified but for a fee. These options include: US political affiliation, education status, gender, and even left handedness.
Requesters set the monetary reward. Many HITs on MTurk are actually set to a relatively low reward (e.g., US $0.10). Some workers choose to pass over work with low payment; however, others will complete the low paying rewards to increase their HIT approval rates. Requesters pay workers based on the quality of their work. They approve or reject the work completed by workers and thus, if a worker’s work is rejected, the monetary reward is not given.
Overall, MTurk is an inexpensive rapid form of data collection, often resulting in participants more representative of the general population than other Internet and student samples (Buhrmester et al. 2011). However, MTurk participants can vary from the general population. Goodman and colleagues (2013) found that compared to community samples MTurk participants pay less attention to experimental materials, for example. In addition, MTurk raises some ethical issues with the often low rewards for workers. Completing three twenty-minute tasks for 1.50 a piece, for example, does not allow workers to meet many mandated hourly minimum wages.
Because MTurk is a great source for data collection, MTurk can also be used in nefarious ways. This could include being asked to take a geotagged photo of the front counter of your local pharmacy. This innocent-enough task could help determine local tobacco prices or could discover the location and front counter security measures of a store. In addition, requesters could crowdsource paid work for lower value or even crowdsource class assignments to the US population, such as the demo below…
Research about MTurk:
Buhrmester, M., Kwang, T., & Gosling, S. D. (2011). Amazon’s Mechanical Turk a new source of inexpensive, yet high-quality, data?. Perspectives on psychological science, 6(1), 3-5.
Goodman, J. K., Cryder, C. E., & Cheema, A. (2013). Data collection in a flat world: The strengths and weaknesses of Mechanical Turk samples. Journal of Behavioral Decision Making, 26(3), 213-224.
Demo:
- Go to the MTurk
- You will be presented with two options: to either become a worker or a requester. Click ‘Get Started’ under the ‘Get Results from Mechanical Turk Workers’ to become a requester.
- To become a requester, for the purposes of the demo, click the ‘get started’ button under the ‘Work Distribution Made Easy’ option.
- You will be asked to sign-in to create a project on this page. Browse through the various available options on the left column. Requesters can launch surveys, ask workers to choose an image they prefer, or ask them their sentiments of a tweet. Simply click ‘sign in to create project’.
- You will then be asked to sign in or set up an amazon account.
- After signing into your account, you will be brought to a page with the following tabs: home, create, manage, developer, and help tabs.
- To create a new project, click ‘create’ and then ‘new project’ directly below. You will now need to select the type of project you wish to design from the left column. I choose ‘survey link’ as I will be embedding a link from Qualtrics (web-based survey tool) for the purposes of this demonstration, so the following instructions are for the survey link option. The survey is asking a question from our previous week’s discussion: “What do you think is a bigger problem — incorrect analysis by amateurs, or amplifying of false information by professionals?”
- After you have indicated your choice of project, click the ‘create project’ button.
- Under the ‘Enter properties’ tab, provide a project name as well as description for your HIT to the Workers. You will also need to set up your HITs. This includes indicating the reward per assignment, number of assignments per (how many people you want to complete your task), the time allotted per assignment, HIT expiration, and the time window before payments are auto-approved. Lastly, you need to indicate worker requirements (e.g., location, HIT approval rate, number of HITs approved).
- Under the design layout, you can use the HTML editor to layout the HIT (e.g., write up the survey instructions as well as provide the survey link).
- You then can preview the instructions. After you have completed setting up your HIT, you will be taken back to the create tab where your new HIT is listed. To publish the batch, simply click ‘Publish Batch’. You then need to confirm payment and publish.
- To view the results and to allocate payment, click ‘Manage’ and download CSV. To approve payment, place an X under the column ‘Approve’. To reject payment, place an X under the column ‘Reject’. This CVS file is then uploaded to MTurk, where approvals and rejections are processed and payment is disbursed to the workers.
- To download results from MTurk, under the ‘manage’ tab, click ‘results. And download the CSV. OR, you can download the results from the platform you are using (e.g., Qualtrics).
- Lastly, there is an entire community forum for MTurk workers entitled Turker Nation. Workers share tips and techniques and discuss all things MTurk and more (e.g., what HITs to complete but also which HITs or requesters to avoid). This can be a useful site to further advertise your HITs.
