{"id":276,"date":"2020-01-30T13:38:37","date_gmt":"2020-01-30T13:38:37","guid":{"rendered":"http:\/\/wordpress.cs.vt.edu\/cs6724s20\/?p=276"},"modified":"2020-01-30T13:38:38","modified_gmt":"2020-01-30T13:38:38","slug":"mohannad-al-ameedi-beyond-mechanical-turk","status":"publish","type":"post","link":"https:\/\/wordpress.cs.vt.edu\/cs6724s20\/2020\/01\/30\/mohannad-al-ameedi-beyond-mechanical-turk\/","title":{"rendered":"Mohannad Al Ameedi – Beyond Mechanical Turk"},"content":{"rendered":"\n

Summary<\/strong><\/h2>\n\n\n\n

In this paper, the authors aim\nto highlight and explore the features of online crowd works other than Amazon Mechanical\nTurk (AMT) that is not been investigated by many researchers. They recognize that\nAMT as a system that made a revolution on data processing and collection, but\nalso lack very crucial features like quality control, automation, and integration.\n<\/p>\n\n\n\n

The paper poses many\nquestions about human computation and presents some solutions. The questions\nrelated to the current problems with AMT, features of other platforms, and a\nway to measure each system. <\/p>\n\n\n\n

The authors discuss the limitation\nthe AMT system like lacking a good quality control, lacking a good management tools,\nand missing a process to prevent fraud, and lacking of a way to automate repeated\ntasks. <\/p>\n\n\n\n

The paper defines criteria to\nevaluate or asses a crowd work platform. The assessment uses different categories\nlike incentive program, quality measures used in the system, the worker\ndemographic and identity information, worker skills or qualifications, and\nother categories that they have used to compare\/contrast different systems including\nAMT. <\/p>\n\n\n\n

The authors also reviewed like\nseven AMT-alternatives like ClickWorker, CloudFactory, CrowdComputing Systems,\nCrowdFlower, CrowdSource, MobileWorks, and oDesk. They show the benefits of\neach system over AMT using the criteria mentioned above, and show that there is\na significant improvement on these systems that make the entire process much\nbetter and enable the worker and the requester to interact in a better way. <\/p>\n\n\n\n

The crows-platforms analysis\ndone by the authors was the only work at the time the paper was written to compare\ndifferent systems on a defined criterion, and they hope that this work offer a\ngood foundation for other researchers to get better results. <\/p>\n\n\n\n

<\/p>\n\n\n\n

All platforms are still missing features like analytics data about each worker to provide visibilities of the work that is done by each worker. There is also a lack of security measure to make sure that the system is robust and can respond to any adversarial attack. <\/p>\n\n\n\n

Reflection<\/strong><\/h2>\n\n\n\n

I found that the features\nthat are missing from Amazon Mechanical Turk interesting giving the volume of\nthe system usage and also given the work that Amazon do today on the cloud computing,\nmarketplace and other area where the quality of the work is well known. <\/p>\n\n\n\n

I also found that the\ntechnical details mentioned in the paper interesting. It seems to me that the\nauthors go lots of feedback from everybody get involved in the system. <\/p>\n\n\n\n

I agree with authors on the criteria\nmentioned in the paper to asses crowdsourcing systems like pay motivation,\nquality control, and automation and system integration. <\/p>\n\n\n\n

<\/p>\n\n\n\n

The authors didn\u2019t specify which system is the best on their opinion and which system meet all of their criteria. <\/p>\n\n\n\n

Questions<\/strong><\/h2>\n\n\n\n
  • Are there any other criteria\nthat we can use to assess the crowdsourcing systems? <\/li>
  • The author didn\u2019t mention\nwhich system is the best. Is there a system that can outperform others? <\/li>
  • Is there a reason why Amazon\ndidn\u2019t address these findings? <\/li><\/ul>\n","protected":false},"excerpt":{"rendered":"

    Summary In this paper, the authors aim to highlight and explore the features of online crowd works other than Amazon Mechanical Turk (AMT) that is not been investigated by many researchers. They recognize that AMT as a system that made a revolution on data processing and collection, but also lack very crucial features like quality […]<\/p>\n","protected":false},"author":294,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[43,28],"class_list":["post-276","post","type-post","status-publish","format-standard","hentry","category-uncategorized","tag-beyondmturk","tag-class2"],"jetpack_featured_media_url":"","_links":{"self":[{"href":"https:\/\/wordpress.cs.vt.edu\/cs6724s20\/wp-json\/wp\/v2\/posts\/276","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/wordpress.cs.vt.edu\/cs6724s20\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/wordpress.cs.vt.edu\/cs6724s20\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/wordpress.cs.vt.edu\/cs6724s20\/wp-json\/wp\/v2\/users\/294"}],"replies":[{"embeddable":true,"href":"https:\/\/wordpress.cs.vt.edu\/cs6724s20\/wp-json\/wp\/v2\/comments?post=276"}],"version-history":[{"count":1,"href":"https:\/\/wordpress.cs.vt.edu\/cs6724s20\/wp-json\/wp\/v2\/posts\/276\/revisions"}],"predecessor-version":[{"id":277,"href":"https:\/\/wordpress.cs.vt.edu\/cs6724s20\/wp-json\/wp\/v2\/posts\/276\/revisions\/277"}],"wp:attachment":[{"href":"https:\/\/wordpress.cs.vt.edu\/cs6724s20\/wp-json\/wp\/v2\/media?parent=276"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/wordpress.cs.vt.edu\/cs6724s20\/wp-json\/wp\/v2\/categories?post=276"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/wordpress.cs.vt.edu\/cs6724s20\/wp-json\/wp\/v2\/tags?post=276"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}