{"id":932,"date":"2020-04-08T12:32:38","date_gmt":"2020-04-08T12:32:38","guid":{"rendered":"http:\/\/wordpress.cs.vt.edu\/cs6724s20\/?p=932"},"modified":"2020-04-08T12:32:38","modified_gmt":"2020-04-08T12:32:38","slug":"04-08-2020-mohannad-al-ameedi-crowdscape-interactively-visualizing-user-behavior-and-output","status":"publish","type":"post","link":"https:\/\/wordpress.cs.vt.edu\/cs6724s20\/2020\/04\/08\/04-08-2020-mohannad-al-ameedi-crowdscape-interactively-visualizing-user-behavior-and-output\/","title":{"rendered":"04\/08\/2020 – Mohannad Al Ameedi – CrowdScape Interactively Visualizing User Behavior and Output"},"content":{"rendered":"\n

Summary<\/strong><\/h2>\n\n\n\n

In this paper, the authors propose a system that can evaluate complex tasks based on both workers output and behaviors. Other available systems are focus on once aspect of evaluation on either the worker output or behavior which can give poor results especially with complex or creative work. The proposed system combine works through interactive visualization and mixed initiative machine learning. The proposed system, CrowdScape, offers visualization to the users that allow them to filter out poor output to focus on a limited number of responses and use machine learning to measure the similarity of response with the best submission and that way the requester can get the best output and best behavior at the same time. The system provides time series data about for user actions like mouse move or scroll up and down to generate visual timeline for tracing user behavior. The system can work only with web pages and has some limitation, but the value that can give to the customer is high and can enable users to navigate through workers results easily and efficiently. <\/p>\n\n\n\n

Reflection<\/strong><\/h2>\n\n\n\n

I found the method\nused by the authors be very interesting. Requesters receive too much information\nabout the workers and visualizing that data can help the requesters to know more\nabout the data, and the use of machine learning can help a lot on classifying or\nclustering the optimal workers output and behaviors. Other approaches mentioned\nin the paper are also interesting especially for simple tasks that don\u2019t need complex\nevaluation. <\/p>\n\n\n\n

I also didn\u2019t\nknow that we can get detailed information about the workers output and behavior\nand found YouTube example mentioned in the paper to be very interesting. The\nexample mentioned shows that MTurk returns everything related to the user\nactions using the help of JavaScript while working on the YouTube video which\ncan be used in many scenarios. \n\nI agree with the authors about the approach which\ncan combine the best of the two approaches. I think it will be interesting to\nknow how many worker response are filtered out in the first phase of the\nprocess because that can tell us if sending the request even worthwhile. If too\nmany responses are not considered, then it is possible that task need to be\nevaluated again. \n\n\n\n<\/p>\n\n\n\n

<\/p>\n\n\n\n

Questions<\/strong><\/h2>\n\n\n\n
  • The authors mentioned that their proposed system can\nhelp to filter out poor outputs on the first phase. Do you think if to many responses\nare filtered out means that the guidelines are the selection criteria needs to\nbe reevaluated?<\/li>
  • The authors depend on JavaScript to track information\nabout the workers behaviors do you think MTurk needs to approve that or it is\nnot necessary? And do you think that the workers also need to be notified before\naccepting the task? <\/li>
  • The authors mention that CrowdScape can be used to\nevaluate complex and creative tasks, do you think that they need to add some process\nto make sure that the task really need to be evaluated by their system, or you\nthink the system can also work with simple tasks? <\/li><\/ul>\n\n\n\n

    <\/p>\n","protected":false},"excerpt":{"rendered":"

    Summary In this paper, the authors propose a system that can evaluate complex tasks based on both workers output and behaviors. Other available systems are focus on once aspect of evaluation on either the worker output or behavior which can give poor results especially with complex or creative work. The proposed system combine works through […]<\/p>\n","protected":false},"author":294,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[93,92],"class_list":["post-932","post","type-post","status-publish","format-standard","hentry","category-uncategorized","tag-class12","tag-crowdscape"],"jetpack_featured_media_url":"","_links":{"self":[{"href":"https:\/\/wordpress.cs.vt.edu\/cs6724s20\/wp-json\/wp\/v2\/posts\/932","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/wordpress.cs.vt.edu\/cs6724s20\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/wordpress.cs.vt.edu\/cs6724s20\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/wordpress.cs.vt.edu\/cs6724s20\/wp-json\/wp\/v2\/users\/294"}],"replies":[{"embeddable":true,"href":"https:\/\/wordpress.cs.vt.edu\/cs6724s20\/wp-json\/wp\/v2\/comments?post=932"}],"version-history":[{"count":1,"href":"https:\/\/wordpress.cs.vt.edu\/cs6724s20\/wp-json\/wp\/v2\/posts\/932\/revisions"}],"predecessor-version":[{"id":933,"href":"https:\/\/wordpress.cs.vt.edu\/cs6724s20\/wp-json\/wp\/v2\/posts\/932\/revisions\/933"}],"wp:attachment":[{"href":"https:\/\/wordpress.cs.vt.edu\/cs6724s20\/wp-json\/wp\/v2\/media?parent=932"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/wordpress.cs.vt.edu\/cs6724s20\/wp-json\/wp\/v2\/categories?post=932"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/wordpress.cs.vt.edu\/cs6724s20\/wp-json\/wp\/v2\/tags?post=932"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}