Tech Demo: LegionTools

Sukrit Venkatagiri

Paper:

W.S. Lasecki, M. Gordon, D. Koutra, M.F. Jung, S.P. Dow and J.P. Bigham. Glance: Rapidly Coding Behavioral Video with the Crowd. In Proceedings of the ACM Symposium on User Interface Software and Technology (UIST 2014).

Brief Overview:

When running studies on Amazon Mechanical Turk, we often need a large number of workers in a short amount of time: either to get survey responses quickly or for real-time and synchronous tasks. LegionTools is the answer to this problem. It is a toolkit and interface that can set up a HIT on MTurk through an algorithm that posts and expires a steady stream of HITs (that all point to the same task), to quickly gather a large number of Turkers.

It can also pool these workers in a “waiting room” of sorts, and a selected subset (or all) of these workers can be routed to a real-time synchronous task, with the push of a button.

Download Link: https://github.com/cromaLab/LegionTools or http://rochci.github.io/LegionTools/

Steps (taken from LegionTools GitHub page):

  1. Add a new task by typing a unique session name, title, description, keywords, and clicking Add new task. Remember your session name, you can use it to pull up your session later on.
  2. Set the target number of workers. Set the price range and click “Update price”.
  3. Click Change waiting page instructions to edit the text that workers will be shown while waiting for your task.
  4. Click Start recruiting to beging recruiting. You must click Stop recruiting to end the recruiting process. Note that stopping recruitment will take some time, depending on your target number of workers.
  5. Pull up a previous task using just your task session name. If you closed the UI page and left the recruiting tool running, you may stop that recruiting process by loading the associated session and clicking Stop recruiting.
  6. Modify task title, description, and keywords with Update. Changes automatically affect all new HITs posted by the recruiting tool.
  7. Send workers to a URL with the Fire button. Your chosen URL must be HTTPS.
  8. When you are ready to review completed HITs, click Reload in the Overview section to load all reviewable HITs associated with a given task session.

 

Sukrit Venkatagiri