There are multiple challenges that exist while ensuring quality control of crowdworkers that are not always easily resolved by employing simple methods such as the use of gold standards or worker agreement. Thus, the authors of this paper propose a new technique to ensure quality control in crowdsourcing for more complex tasks. By utilizing features from worker behavioral traces as well as worker outputs, they aid researchers to better understand the crowd. As part of this research, the authors propose novel visualizations to illustrate user behavior, new techniques to explore crowdworker products, tools to group as well as classify workers, and mixed initiative machine learning models that build on a user\u2019s intuition about the crowd. They created CrowdScape \u2013 built on top of MTurk which captures data from the MTurk API as well as a Task Fingerprinting system in order to obtain worker behavioral traces. The authors discuss various case studies such as translation, picking a favorite color, writing about a favorite place, and tagging a video and describe the benefits of CrowdScape in each case. <\/p>\n\n\n\n
I found that CrowdScape is a very\ngood system especially considering the difficulty in ensuring quality control\namong crowdworkers in case of more complex tasks. For example, in case of a\nsummarization task, particularly for larger documents, there is no single gold\nstandard that can be used and it would be rare that the answers of multiple\nworkers would match for us to use majority vote as a quality control strategy.\nThus, for applications such as this, I\nthink it is very good that the authors proposed a methodology that combines\nboth behavioral traces as well as worker output and I agree that it provides\nmore insight that using either alone. I found that the example of the requester\nintending to have summaries written for YouTube physics tutorials was an\nappropriate example.<\/p>\n\n\n\n
I also liked the visualization\ndesign that the authors proposed. They aimed to combine multiple views and made\nthe interface easy for requesters to use. I especially found the use of 1-D and\n2-D matrix scatter plots showing distribution of features over the group of\nworkers that also enabled dynamic exploration to be well thought out. <\/p>\n\n\n\n
I found the case study on\ntranslation to be especially well thought out \u2013 given that the authors\nstructured the study such that they included a sentence that did not parse well\nin computer generated translations. I feel that such a strategy can be used in\nmultiple translation related activities in order to more easily discard\nsubmissions by lazy workers. I also liked the case study on \u2018Writing about a\nFavorite Place\u2019 as it indicated the performance of the CrowdScape system in a\nsituation wherein no two workers would provide the same response and\ntraditional quality control techniques would not be applicable. <\/p>\n\n\n\n
SUMMARY There are multiple challenges that exist while ensuring quality control of crowdworkers that are not always easily resolved by employing simple methods such as the use of gold standards or worker agreement. Thus, the authors of this paper propose a new technique to ensure quality control in crowdsourcing for more complex tasks. By utilizing […]<\/p>\n","protected":false},"author":288,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[93,92],"class_list":["post-965","post","type-post","status-publish","format-standard","hentry","category-uncategorized","tag-class12","tag-crowdscape"],"jetpack_featured_media_url":"","_links":{"self":[{"href":"https:\/\/wordpress.cs.vt.edu\/cs6724s20\/wp-json\/wp\/v2\/posts\/965","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/wordpress.cs.vt.edu\/cs6724s20\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/wordpress.cs.vt.edu\/cs6724s20\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/wordpress.cs.vt.edu\/cs6724s20\/wp-json\/wp\/v2\/users\/288"}],"replies":[{"embeddable":true,"href":"https:\/\/wordpress.cs.vt.edu\/cs6724s20\/wp-json\/wp\/v2\/comments?post=965"}],"version-history":[{"count":2,"href":"https:\/\/wordpress.cs.vt.edu\/cs6724s20\/wp-json\/wp\/v2\/posts\/965\/revisions"}],"predecessor-version":[{"id":967,"href":"https:\/\/wordpress.cs.vt.edu\/cs6724s20\/wp-json\/wp\/v2\/posts\/965\/revisions\/967"}],"wp:attachment":[{"href":"https:\/\/wordpress.cs.vt.edu\/cs6724s20\/wp-json\/wp\/v2\/media?parent=965"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/wordpress.cs.vt.edu\/cs6724s20\/wp-json\/wp\/v2\/categories?post=965"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/wordpress.cs.vt.edu\/cs6724s20\/wp-json\/wp\/v2\/tags?post=965"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}