This paper tries to take a hard look at how useful conversational agents like Google Now and Siri are in the real world, when in the hands of real users who try to use them in daily life. The authors conduct interviews with 14 users to get their general thoughts about how they use these tools and in some case, get step by step details on how they do specific tasks. The paper is able to get some interesting insights and provide some useful recommendations on how to improve the existing CAs. Recommendations include making design changes to inform users the limitations of what the CAs can do, tone down some of the more personable aspects which gives a false impression that they are equivalent to humans in understanding, and rethinking design for easier use in hands free scenarios.
First thing that I noticed, after having read and focused primarily on papers that had some quantitative aspect to them, was that this paper is entirely focused on evaluating and interpreting the content of their interviews. I suppose this is another important way in which HCI research is done and shared with the world, because it focuses entirely on the human side of it. I think they have some good interpretations and recommendations from it. The general problem I have with these kinds of studies is the small sample size, which rears up here too. But I can look past that because I think they still are able to get some good insights and make some good recommendations, and provide focus on a mode of interaction that is entirely dialogue based. I do think that if they could have a bigger sample size and do some quantitative work, they could maybe show some trends in the failings of CAs. The most interesting insight for me is the fact that CAs seemed to have been designed with the thought that they would be the focus of attention when used, when in reality people were trying to use it while doing something else and were not looking at their phone. So the feedback mechanism was useless for the users because they were trying to be hands free. From my perspective, that seems to be the most actionable change and can probably lead to (or maybe it already has lead to) interesting design research on how to best provide task feedback for different kinds of tasks for hands free usage.
- What kind of design elements can be included to help people understand the limits of what the CA can do, and thereby avoid having unfulfillable expectations?
- Similarly, what kind of design elements would be useful to better suit the hands free usage of the CAs?
- Should CAs aim to be more task oriented like Google Now, or more personable like Siri? What’s your preferred fit?
Hi Subil. I really like your first question. I think the types of design elements that can be included to help people understand the limits of what the CA can do—and thereby avoid having unfulfillable expectations—are similar to what we read about in the last few weeks. For example, Microsoft’s guidelines for Human-AI interaction (https://www.microsoft.com/en-us/research/uploads/prod/2019/01/Guidelines-for-Human-AI-Interaction-camera-ready.pdf). In the guidelines, they state that AI capabilities should be clear and how well it can do it.