In this paper, the authors study the effectiveness of vision-to-language systems for automatically generating alt texts for images and the impact of human-in-the-loop for this task. The authors set up four methods for generating alt text. First is a simple implementation of modern vision-to-language alt text generation. The second is a human-adjusted version of the first method. The third method is a more involved one, where a Blind or Vision Impaired (BVI) user chats with a non-BVI user to gain more context about an image. The final method is a generalized version of the third method, where the authors analyzed the patterns of questions asked during the third method to form a structured set of pre-defined questions that a crowdsource worker can directly provide the answer to without having the need for a lengthy conversation. The authors conclude that current vision-to-language techniques can, in fact, harm context understanding for BVI users, and simple human-in-the-loop methods significantly outperform. They also found that the method of the structured questions worked the best.<\/p>\n\n\n\n
This\nwas an interesting study that implicitly pointed out the limitation of\ncomputers at understanding social context which is a human affordance. The\nauthors stated that the results of a vision-to-language system often confused\nthe users because the system did not get the point<\/em>. This made me wonder\nif the current limitation could be overcome in the future.<\/p>\n\n\n\n I\nwas also concerned whether the authors\u2019 proposed methods were even practical.\nSure, the human-in-the-loop method involving Mturk workers greatly enhanced the\ndescription of a Twitter image, but based on their report, it\u2019ll take too long\nto retrieve the description. The paper reports that to answer one of the structured\nquestions, it takes on average, 1 minute. This is excluding the time it takes\nfor a Mturk worker to accept a HIT. The authors suggested pre-generating alt\ntexts for popular Tweets, but this does not completely solve the problem.<\/p>\n\n\n\n I was also skeptical about the way the authors performed validation with the 7 BVI users. In their validation, they simulated their third method (TweetTalk, a conversation between BVI and sighted users). However, they did not do it by using their application, but rather a face-to-face conversation between the researchers and the participants. The authors claimed that they tried to replicate the environment as much as possible, but I think there still can be flaws since the researchers serving as the sighted user already had expert knowledge about their experiment. Also, as stated in the paper\u2019s limitations section, the validation was performed with too fewer participants. This may not fully capture the BVI users\u2019 behaviors.<\/p>\n\n\n\n These\nare the questions that I had while reading this paper:<\/p>\n\n\n\n 1.\nDo you think the authors\u2019 proposed methods are actually practical? What could\nbe done to make them practical if you don\u2019t think so?<\/p>\n\n\n\n 2.\nWhat do you think were the human affordances needed for the human element of\nthis experiment other than social awareness?<\/p>\n\n\n\n 3.\nDo you think the authors\u2019 validation with the BVI users is sound? Also, the\nvalidation was only done for the third method. How can the validation be done\nfor the rest of the methods?<\/p>\n","protected":false},"excerpt":{"rendered":" In this paper, the authors study the effectiveness of vision-to-language systems for automatically generating alt texts for images and the impact of human-in-the-loop for this task. The authors set up four methods for generating alt text. First is a simple implementation of modern vision-to-language alt text generation. The second is a human-adjusted version of the […]<\/p>\n","protected":false},"author":286,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[77,80],"class_list":["post-690","post","type-post","status-publish","format-standard","hentry","category-uncategorized","tag-class7","tag-socialalttext"],"jetpack_featured_media_url":"","_links":{"self":[{"href":"https:\/\/wordpress.cs.vt.edu\/cs6724s20\/wp-json\/wp\/v2\/posts\/690","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/wordpress.cs.vt.edu\/cs6724s20\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/wordpress.cs.vt.edu\/cs6724s20\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/wordpress.cs.vt.edu\/cs6724s20\/wp-json\/wp\/v2\/users\/286"}],"replies":[{"embeddable":true,"href":"https:\/\/wordpress.cs.vt.edu\/cs6724s20\/wp-json\/wp\/v2\/comments?post=690"}],"version-history":[{"count":1,"href":"https:\/\/wordpress.cs.vt.edu\/cs6724s20\/wp-json\/wp\/v2\/posts\/690\/revisions"}],"predecessor-version":[{"id":691,"href":"https:\/\/wordpress.cs.vt.edu\/cs6724s20\/wp-json\/wp\/v2\/posts\/690\/revisions\/691"}],"wp:attachment":[{"href":"https:\/\/wordpress.cs.vt.edu\/cs6724s20\/wp-json\/wp\/v2\/media?parent=690"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/wordpress.cs.vt.edu\/cs6724s20\/wp-json\/wp\/v2\/categories?post=690"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/wordpress.cs.vt.edu\/cs6724s20\/wp-json\/wp\/v2\/tags?post=690"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}