Greg Little, Lydia B. Chilton, Max Goldman, and Robert C. Miller. 2010. TurKit: human computation algorithms on mechanical turk. In Proceedings of the 23nd annual ACM symposium on User interface software and technology (UIST ’10). ACM, New York, NY, USA, 57-66. DOI=10.1145/1866029.1866040 http://doi.acm.org/10.1145/1866029.1866040
Discussion Leader: Adam Binford
Summary
TurKit is a framework developed by the authors to develop algorithms that incorporate human computation through Amazon’s Mechanical Turk. Mechanical Turk has its own API for requesters to use to integrate with their algorithms, but it’s mostly only conducive to highly independent and parallelizable tasks. TurKit provides a way to develop sequential and iterative algorithms that make use of multiple steps of human computation. It does this through some JavaScript extensions and a new crash-and-rerun programming model.
The crash-and-run model allows a TurKit script to be re-executed without having to rerun the costly Mechanical Turk functions, which are costly in both time and money. During an initial run of the script, the calls to the Mechanical Turk API go through and the results, whenever they are returned, are saved, so that subsequent executions of the script simply use the saved result. This allows you to develop, test, and make changes to your TurKit scripts without having to pay someone to do the HIT each time. This saving of state is achieved through some keywords introduced on top of regular JavaScript syntax. Use of the keyword once means the indicated function should only be called once on the first run, and in subsequent executions the same result is used. Additionally, functions fork and join are added to help facilitate parallelism and execution of multiple HITs at the same time, whose results can then be combined into later logic in the algorithm. The authors discuss some iterative applications that can make use of this paradigm, such as iterative writing or iterative blurry text recognition. Finally, they discuss two experiments by other researchers that make use of TurKit and the performance hit of the crash-and-rerun paradigm.
Reflection
The key contribution of this paper I believe is the crash-and-rerun programming model. It is what enables the key benefit of TurKit, developing and modifying a human computation enabled algorithm without having to pay someone to complete a HIT each time you run it. I think the crash-and-rerun style to achieve this is really clever, and I wonder if it could have any use in non-human computation development as well. The idea of retroactive print-line-debugging would be very useful for many different scenarios. Clearly the usability of this model is dependent on the amount of non-deterministic code in your algorithm, and many real world programs would be too large or have too much concurrency to make it feasible.
Crowdsourcing does have some unique properties that make it especially suited for this model. As stated in the paper, compared to the time it takes to complete a HIT, code execution time is almost negligible. So while some overhead with re-executing the program is acceptable, it may not be for other areas. Additionally, most human computation tasks tend to be fairly straightforward. If it’s not completely parallelizable, it is a simple iterative process. It will be interesting to see if toolkits like these enable people to come up with more complex human computation algorithms that stretch the limits of the crash-and-rerun model.
Questions
– Do you think most human computation implementations could make use of TurKit or are the majority single parallelizable task based?
– What human computation algorithms might be too complex for TurKit?
– What other applications would there be for the crash-and-rerun model?
– Do you think the example of writing an article would be possible through a pool of generally unskilled and unknowledgeable workers?
