Summary
In this paper, the author presents first two research efforts related to the human interaction. The first effort focuses on user direct manipulation and the second focuses on automated services. The author suggests an approach that can integrated both by offering a way to allow the user to directly manipulate user interface elements, and also use automation to decrease the amount of interaction necessary to finish a task. The author presents factors that can make the integration between the automation and direct manipulation more effective. These factors include developing added value automation, considering user goals uncertainty, considering the user attention in the timing of the automated service, considering the cost and benefit of the action uncertainty, involving direct dialog with the user, and other factors.
The author proposes a system called lookout which can help users to schedule meetings and appointments by reading the user emails and extract useful information related to meetings and auto populate some fields, like the meeting time and meeting recipients, which can save the user mouse and keyboard interaction. The uses probabilistic classification system to help reduce the amount of interaction necessary to accomplish scheduling tasks. The system also uses sound recognition feature developed by Microsoft research to offer additional help to the users during the direct manipulation of the suggested information. The lookout system combines both automated services and ability to allow direct manipulation of the system. The system also asses the user interaction to decide when the automated service will not be helpful to the user to make sure that the uncertainty is well considered. The lookout system improves the human-computer interaction through the combination of reasoning machinery and direct manipulation.
Reflection
I found the idea of maintaining of a working memory of user interaction interesting. This approach to learn from the user experience as stated in the paper, but it can also use machine learning methods to predict the next required action for a new user or an existing user by learning from all other users.
I also found the lookout system very interesting since it is integrating the direct user input with automated service. The sound recognition features that was developed by Microsoft research is not only allow the user to interact via mouse and keyboard but it also give another interaction media which is extremely important to users with disabilities.
Cortana, which is a Microsoft intelligent agent, does parsing to the email too and check if the user send an email that contains a keyword like “I will schedule a meeting” or “I will follow up on that later” and then will remind the user to follow up and will present two buttons asking the user to directly interact with the alert by either dismiss the alert or by asking the system to send a follow up again next day.
Questions
- Can we use the human interaction data used in lookout as a labeled data and develop a machine learning algorithm that can predict the next user interaction?
- Can we use LookOut idea in a different domain?
- The author suggests dozen factors to integrate the automation services with direct manipulation, which factors you think can be useful to the crowdsourcing users?
