{"id":1064,"date":"2020-04-15T10:40:27","date_gmt":"2020-04-15T14:40:27","guid":{"rendered":"http:\/\/wordpress.cs.vt.edu\/cs6724s20\/?p=1064"},"modified":"2020-04-15T10:40:27","modified_gmt":"2020-04-15T14:40:27","slug":"04-15-20-lee-lisle-believe-it-or-not-designing-a-human-ai-partnership-for-mixed-initiative-fact-checking","status":"publish","type":"post","link":"https:\/\/wordpress.cs.vt.edu\/cs6724s20\/2020\/04\/15\/04-15-20-lee-lisle-believe-it-or-not-designing-a-human-ai-partnership-for-mixed-initiative-fact-checking\/","title":{"rendered":"04\/15\/20 – Lee Lisle – Believe it or Not: Designing a Human-AI Partnership for Mixed-Initiative Fact-Checking"},"content":{"rendered":"\n

Summary<\/strong><\/p>\n\n\n\n

Ngyuen et al\u2019s paper discusses the\nrise of misinformation and the need to combat it via tools that can verify\nclaims while also maintaining users\u2019 trust of the tool. They designed an\nalgorithm that finds sources that are similar to a given claim to determine\nwhether or not the claim is accurate. They also weight the sources based on\nesteem. They then ran 3 studies (with over 100 participants in each) where\nusers could interact with the tool and change settings (such as source\nweighting) in order to evaluate their design. The first study found that the\nparticipants trusted the system too much \u2013 when it was wrong, they tended to be\ninaccurate, and when it was right, they were more typically correct. The second\nstudy allowed participants to change the inputs and inject their own expertise\ninto the scenario. This study found that the sliders did not significantly\nimpact performance. The third study focused on gamification of the interface,\nand found no significant difference.<\/p>\n\n\n\n

Personal Reflection<\/strong><\/p>\n\n\n\n

               I\nenjoyed this paper from a 50,000 foot perspective, as they tested many\ndifferent interaction types and found what could be considered negative results.\nI think papers that show that all work is not necessarily good have a certain\namount of extra relevance \u2013 they certainly show that there\u2019s more at work than\njust novelty. <\/p>\n\n\n\n

I especially appreciated the study\non the effectiveness of gamification. Often, the prevailing theory is that\ngamification increases user engagement and increases the tools\u2019 effectiveness. While\nthe paper is not conclusive that gamification cannot do this, it certainly\nlends credence to the thought that gamification is not a cure-all.<\/p>\n\n\n\n

However, I took some slight issue\nwith their AI design. Particularly, the AI determined that the phrase \u201cTiger\nWoods\u201d indicated a supportive position. While their stance was that AIs are\nflawed (true), I felt that this error was quite a bit more than we can expect\nfrom normal AIs, especially ones that are being tweaked to avoid these\nscenarios. I would have liked to see experiment 2 and 3 improved with a better\nAI, as it does not seem like they cross-compared studies anyway.<\/p>\n\n\n\n

Questions<\/strong><\/p>\n\n\n\n

  1. Does the interface design including a slider to\nadjust source reputations and user agreement on the fly seem like a good idea?\nWhy or why not?<\/li>
  2.  What do\nyou think about the attention check and its apparent failure to accurately\ncheck? Should they have removed the participants with incorrect answers to this\ncheck?<\/li>
  3. Should the study have included a pre-test to\ndetermine how the participants\u2019 world view may have affected the likelihood of them\nagreeing with certain claims? I.E., should they have checked to see if the\nparticipants were impartial, or tended to agree with a certain world view? Why\nor why not?<\/li>
  4. What benefit do you think the third study\nbrought to the paper? Was gamification proved to be ineffectual, or is it a\ndesign tool that sometimes doesn\u2019t work?<\/li><\/ol>\n","protected":false},"excerpt":{"rendered":"

    Summary Ngyuen et al\u2019s paper discusses the rise of misinformation and the need to combat it via tools that can verify claims while also maintaining users\u2019 trust of the tool. They designed an algorithm that finds sources that are similar to a given claim to determine whether or not the claim is accurate. They also […]<\/p>\n","protected":false},"author":105,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[100,98],"class_list":["post-1064","post","type-post","status-publish","format-standard","hentry","category-uncategorized","tag-believeitornot","tag-class13"],"jetpack_featured_media_url":"","_links":{"self":[{"href":"https:\/\/wordpress.cs.vt.edu\/cs6724s20\/wp-json\/wp\/v2\/posts\/1064","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/wordpress.cs.vt.edu\/cs6724s20\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/wordpress.cs.vt.edu\/cs6724s20\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/wordpress.cs.vt.edu\/cs6724s20\/wp-json\/wp\/v2\/users\/105"}],"replies":[{"embeddable":true,"href":"https:\/\/wordpress.cs.vt.edu\/cs6724s20\/wp-json\/wp\/v2\/comments?post=1064"}],"version-history":[{"count":1,"href":"https:\/\/wordpress.cs.vt.edu\/cs6724s20\/wp-json\/wp\/v2\/posts\/1064\/revisions"}],"predecessor-version":[{"id":1065,"href":"https:\/\/wordpress.cs.vt.edu\/cs6724s20\/wp-json\/wp\/v2\/posts\/1064\/revisions\/1065"}],"wp:attachment":[{"href":"https:\/\/wordpress.cs.vt.edu\/cs6724s20\/wp-json\/wp\/v2\/media?parent=1064"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/wordpress.cs.vt.edu\/cs6724s20\/wp-json\/wp\/v2\/categories?post=1064"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/wordpress.cs.vt.edu\/cs6724s20\/wp-json\/wp\/v2\/tags?post=1064"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}