{"id":786,"date":"2020-03-22T00:24:03","date_gmt":"2020-03-22T00:24:03","guid":{"rendered":"http:\/\/wordpress.cs.vt.edu\/cs6724s20\/?p=786"},"modified":"2020-03-24T23:32:49","modified_gmt":"2020-03-24T23:32:49","slug":"3-25-2020-mohannad-al-ameedi-evaluating-visual-conversational-agents-via-cooperative-human-ai-games","status":"publish","type":"post","link":"https:\/\/wordpress.cs.vt.edu\/cs6724s20\/2020\/03\/22\/3-25-2020-mohannad-al-ameedi-evaluating-visual-conversational-agents-via-cooperative-human-ai-games\/","title":{"rendered":"3\/25\/2020 – Mohannad Al Ameedi – Evaluating Visual Conversational Agents via Cooperative Human-AI Games"},"content":{"rendered":"\n

Summary<\/strong><\/h2>\n\n\n\n

Improvements in artificial intelligence systems are normally\nmeasured alone without taking into consideration the human element. In this\npaper, the authors try to measure and evaluate the human-AI team performance by\ndesigning an interactive visual conversational agent that involve both human\nand AI to solve a specific problem. The conversational agent assigns the AI\nsystem a secret image with caption which is not known by the human, and the\nhuman start rounds of questions to guess the correct image from pool of images.\nThe agent maintains an internal memory of questions and answers to help maintaining\nthe conversation.<\/p>\n\n\n\n

The authors use two version of AI systems, the first one is trained using supervised learning and the second is trained using reinforcement learning. The second system outperforms the first, but the improvement doesn\u2019t translate well when interacting with human which proves that advances in AI system doesn\u2019t necessarily means advances in the human-AI team performance. <\/p>\n\n\n\n

Reflection<\/strong><\/h2>\n\n\n\n

I found the idea of running two AI systems with the same human to be very interesting. Normally we think that advances in AI system can lead to better usage by the human, but the study shows that this is not the case. Putting the human in the loop while improving the AI system will give us the real performance of the system. <\/p>\n\n\n\n

I also found the concept of reinforcement learning in conversational\nagents to be also very interesting. Using online learning by assigning a positive\nand negative rewards can help to improve the conversation between human and AI\nsystem, which can prevent the system from getting stuck on the same answer if\nthe human ask the same question.<\/p>\n\n\n\n

The work in somehow like the concept of compatibility. When human makes a mental model about the AI system. Advances in AI system might not be translated into a better usage by the human, and this is what was proven by the authors when they use two AI systems and one is better than the other, but improvement not necessarily translate to better performance by the users. <\/p>\n\n\n\n

Questions<\/strong><\/h2>\n\n\n\n
  • The authors proved that\nimprovement in AI system alone doesn\u2019t necessarily leads to a better\nperformance when using the system by human, can we involve the human in the\nprocess of improving the AI system to lead to a better performance when the AI system\nget improved?<\/li>
  • The authors use a single\nsecret image known by the AI system but not known by the human, can we make the\nimage unknown to the AI system too by providing a pool of images and the AI\nsystem select the appropriate image? And can we do that with acceptable response\nlatency? <\/li>
  • If we have to use a conversational\nagents like bots in production setting, do you think the performance of an AI system\ntrained using a supervised learning can response faster than a system trained using\na reinforcement learning giving that the reinforcement learning will need to adjust\nit is behavior based on the reward or feedback? <\/li><\/ul>\n","protected":false},"excerpt":{"rendered":"

    Summary Improvements in artificial intelligence systems are normally measured alone without taking into consideration the human element. In this paper, the authors try to measure and evaluate the human-AI team performance by designing an interactive visual conversational agent that involve both human and AI to solve a specific problem. The conversational agent assigns the AI […]<\/p>\n","protected":false},"author":294,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[87,81],"class_list":["post-786","post","type-post","status-publish","format-standard","hentry","category-uncategorized","tag-class10","tag-vqagames"],"jetpack_featured_media_url":"","_links":{"self":[{"href":"https:\/\/wordpress.cs.vt.edu\/cs6724s20\/wp-json\/wp\/v2\/posts\/786","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/wordpress.cs.vt.edu\/cs6724s20\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/wordpress.cs.vt.edu\/cs6724s20\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/wordpress.cs.vt.edu\/cs6724s20\/wp-json\/wp\/v2\/users\/294"}],"replies":[{"embeddable":true,"href":"https:\/\/wordpress.cs.vt.edu\/cs6724s20\/wp-json\/wp\/v2\/comments?post=786"}],"version-history":[{"count":2,"href":"https:\/\/wordpress.cs.vt.edu\/cs6724s20\/wp-json\/wp\/v2\/posts\/786\/revisions"}],"predecessor-version":[{"id":788,"href":"https:\/\/wordpress.cs.vt.edu\/cs6724s20\/wp-json\/wp\/v2\/posts\/786\/revisions\/788"}],"wp:attachment":[{"href":"https:\/\/wordpress.cs.vt.edu\/cs6724s20\/wp-json\/wp\/v2\/media?parent=786"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/wordpress.cs.vt.edu\/cs6724s20\/wp-json\/wp\/v2\/categories?post=786"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/wordpress.cs.vt.edu\/cs6724s20\/wp-json\/wp\/v2\/tags?post=786"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}