Authors: Ting-Hao (Kenneth) Huang, Joseph Chee Chang, and Jeffrey P. Bigham
Summary
This paper discusses Evorus, a crowd-powered intelligent conversation agent that is targeted towards automation over time. It allows new chatbots to be integrated, reuses prior crowd responses, and learns to automatically approve responses. It demonstrates how automation can efficiently be deployed by augmenting an existing system. Users used Evorus through Google Hangouts.
Reflection
There’s a lot happening in this paper, but then it’s perfectly justified because of the eventual target — fully automated system. This paper is a great example of how to carefully plan the path to an automated system from manual origins. It is realistic in terms of feasibility, and the transition from a crowd-based system to a crowd-AI collaborative system aimed towards a fully automated one seems organic and efficient as seen from the results.
In terms of their workflow, they break down different elements i.e. chatbots and vote bots, and essentially, scope down the problem to just selecting a chatbot and voting on responses. A far-fetched approach would have been to build (or aim for) an end-to-end (God-mode) chatbot that can give the perfect response. Because the problem is scoped down, and depends on interpretable crowd worker actions, designing a learning framework around these actions and scoped down goals seems like a feasible approach. This is a great takeaway from the paper — how to break down a complex goal into smaller goals. Instead of attempting to automate an end-to-end complex task, crafting ways to automate smaller, realizable elements along the path seems like a smarter alternative.
The voting classifier was carefully designed, considering a lot of interpretable and relevant features again such as message, turn and conversation levels. Again, this was evaluated with a real purpose i.e. reducing the human effort in voting.
This paper also shows how we can still build intelligent systems that improve over time on top of AI engines that we cannot (or may, actually do not have to) modify i.e. third-party developer chatbots, off-the-shelf AI APIs. Crowd-AI collaboration can be useful for this aspect, and therefore designing the user interaction(s) remains critical for a learning framework to be augmented to the fixed AI engine e.g. vote bot or the select bot in this paper’s case.
Questions
- If you are working with an off-the-shelf AI engine that cannot be modified, how do you plan on building a system that improves over time?
- What other (interaction) areas in the Evorus system do you see for a potential learning framework that would improve the performance of the system (according to the existing metrics)?
- If you were working a complex task, would you prefer an end-to-end God-mode solution or adopt a slow approach by carefully breaking it down and automating each element?
