How to detect if a crowd worker has returned an assignment of your HIT on MTurk

When using your own web app to deploy HITs via MTurk API (ExternalQuestion), it’s important sometimes to detect the status of HIT assignments — whether it’s being worked on or returned or abandoned. This might influence what data should be rendered in your task interface, or what tasks to release the next, or other specific requirements.

However, MTurk doesn’t provide such information in the API. Yet, with a good understanding of how MTurk assign unique IDs and some javascript tricks,

Method 1: Check Assignment ID

MTurk usually assign the returned Assignment ID to the next worker taking the HIT. This is very useful information because some requesters might use assignment id as primary key in their tables. Bad idea.

By checking if the assignment ID has been recorded in your system, you can tell the previous task with this assignment ID was returned/abandoned by previous worker.

However, this is not enough. When the assignment ID is not reused, we need to help ourselves.

Method 2: sending AJAX POST request repeatedly

You can send an AJAX post every few minutes to your server, to make sure the worker is still there. When all your tasks are recorded taken in your database, but there are still new worker taking a new assignment, that means some previous workers returned their tasks but you don’t know yet. By check the last time you get an AJAX post for a task, you will know which one is no longer active and mark it as returned/abandoned.

Following is my example implementation

var mytimeout = 180000;
function working(){
 $.ajaxSetup({
 beforeSend: function(xhr, settings) {
 if (!csrfSafeMethod(settings.type) && !this.crossDomain) {
 xhr.setRequestHeader("X-CSRFToken", csrftoken);
 }
 }
 });
 $.ajax({
 url : "/pipeline/worker/save", // the endpoint
 type : "POST", // http method
 async: true,
 data : {stillhere: true, assignmentId: $("input[name=assignmentId]").val(), hitId: $("input[name=hitId]").val(), workerId: $("input[name=workerId]").val()}, // data sent with the post request
 // handle a successful response
 success : function(json) {
 },

// handle a non-successful response
 error : function(xhr,errmsg,err) {
 // alert(errmsg); // add the error to the dom
 console.log(xhr.status + ": " + xhr.responseText); // provide a bit more info about the error to the console
 },
 complete: function (data) {
 // Schedule the next
 setTimeout(working, mytimeout);
 }
 }); 
 }
 setTimeout(working, mytimeout);

Usually these two methods together should take care of all the returned HIT. If you are still worried, you can always use MTurk API to retrieve submitted assignment IDs and compare with your database records.

Leave a Reply

Your email address will not be published. Required fields are marked *