{"id":1060,"date":"2020-04-15T10:40:27","date_gmt":"2020-04-15T14:40:27","guid":{"rendered":"http:\/\/wordpress.cs.vt.edu\/cs6724s20\/?p=1060"},"modified":"2020-04-15T10:40:27","modified_gmt":"2020-04-15T14:40:27","slug":"04-15-2020-palakh-mignonne-jude-believe-it-or-not-designing-a-human-ai-partnership-for-mixed-initiative-fact-checking","status":"publish","type":"post","link":"https:\/\/wordpress.cs.vt.edu\/cs6724s20\/2020\/04\/15\/04-15-2020-palakh-mignonne-jude-believe-it-or-not-designing-a-human-ai-partnership-for-mixed-initiative-fact-checking\/","title":{"rendered":"04\/15\/2020 – Palakh Mignonne Jude – Believe it or not: Designing a Human-AI Partnership for Mixed-Initiative Fact-Checking"},"content":{"rendered":"\n

SUMMARY<\/h2>\n\n\n\n

The authors of this paper design and evaluate a mixed-initiative fact-checking approach that blends prior human knowledge with the efficiency of automated ML systems. The authors found that users tend to over-trust the model which could degrade human accuracy. They conducted three randomized experiments \u2013 the first, compares user who perform the task with or without viewing ML predictions, the second, compares a static interface with an interactive one (that enables users to fix model predictions), and the third, compares a gamifies task design to a non-gamified one. The authors designed an interface that displays the claim, the predicted correctness, and relevant articles. For the first experiment, the authors considered responses from 113 participants with 58 assigned to Control and 55 to System. For the second experiment, the authors considered responses from 109 participants with 51 assigned to Control and 58 to Slider. For the third experiment, the authors considered responses from 106 participants, and found no significant differences between the two groups. <\/p>\n\n\n\n

REFLECTION<\/h2>\n\n\n\n

I liked the idea of a mixed-initiative\napproach to fact checking that builds on the affordances of both humans and AI.\nI found that it was good that the authors designed the experiments such that\nthe confidence scores (and therefore the fallibility) of the system was openly\nshown to the users. I also felt that the interface design was concise and appropriate\nwithout being overly complex. I also liked the design of the gamified approach\nand was surprised to learn that the game design did not impact participant\nperformance.<\/p>\n\n\n\n

I agree that for this case in\nparticular, participant demographics may affect the results. Especially since\nthe news articles considered were mainly related to American news. I wonder how\nmuch if a difference in the results would be observed in a follow-up study that\nconsiders different demographics as compared to this study. I also agree that\ncaution must be exercised with such mixed-initiative systems as imperfect data\nsets would have a considerable impact on model predictions and that the humans\nshould not blindly trust the AI predictions). It would definitely be\ninteresting to see the results obtained when users check their own claims and\ninteract with other user\u2019s predictions. <\/p>\n\n\n\n

QUESTIONS<\/h2>\n\n\n\n
  1. The\nauthors explain that the incorrect statement on Tiger Woods was due to the\nmodel having learnt the bi-gram \u2018Tiger Woods\u2019 incorrectly \u2013 something that a\nmore sophisticated classifier may have avoided. How much of an impact would\nsuch a classifier have made on the results obtained overall? Have other\ncomplementary studies been conducted?<\/li>
  2. The\nauthors found that a smaller percentage of users used the sliders than expected.\nThey state that while the sliders were intended to be intuitive, they may require\na learning curve causing lesser users to adopt it. Would the use of a tutorial that\nenabled users to familiarize themselves have helped in this case?<\/li>
  3. Were\nthe experiments conducted in this study adequate? Are there any other\nexperiments that the authors should have conducted in addition to the ones\nmentioned?<\/li><\/ol>\n","protected":false},"excerpt":{"rendered":"

    SUMMARY The authors of this paper design and evaluate a mixed-initiative fact-checking approach that blends prior human knowledge with the efficiency of automated ML systems. The authors found that users tend to over-trust the model which could degrade human accuracy. They conducted three randomized experiments \u2013 the first, compares user who perform the task with […]<\/p>\n","protected":false},"author":288,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[100,98],"class_list":["post-1060","post","type-post","status-publish","format-standard","hentry","category-uncategorized","tag-believeitornot","tag-class13"],"jetpack_featured_media_url":"","_links":{"self":[{"href":"https:\/\/wordpress.cs.vt.edu\/cs6724s20\/wp-json\/wp\/v2\/posts\/1060","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/wordpress.cs.vt.edu\/cs6724s20\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/wordpress.cs.vt.edu\/cs6724s20\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/wordpress.cs.vt.edu\/cs6724s20\/wp-json\/wp\/v2\/users\/288"}],"replies":[{"embeddable":true,"href":"https:\/\/wordpress.cs.vt.edu\/cs6724s20\/wp-json\/wp\/v2\/comments?post=1060"}],"version-history":[{"count":1,"href":"https:\/\/wordpress.cs.vt.edu\/cs6724s20\/wp-json\/wp\/v2\/posts\/1060\/revisions"}],"predecessor-version":[{"id":1063,"href":"https:\/\/wordpress.cs.vt.edu\/cs6724s20\/wp-json\/wp\/v2\/posts\/1060\/revisions\/1063"}],"wp:attachment":[{"href":"https:\/\/wordpress.cs.vt.edu\/cs6724s20\/wp-json\/wp\/v2\/media?parent=1060"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/wordpress.cs.vt.edu\/cs6724s20\/wp-json\/wp\/v2\/categories?post=1060"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/wordpress.cs.vt.edu\/cs6724s20\/wp-json\/wp\/v2\/tags?post=1060"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}