04/08/2020 – Nurendra Choudhary – CrowdScape: Interactively Visualizing User Behavior and Output

April 7, 2020April 8, 2020 Nurendra Choudhary Leave a comment

Summary

In this paper, the authors solve the problem of large-scale human evaluation through CrowdScape, a system for large-scale human evaluation based on interactive visualizations and mixed-initiative machine learning. They track two major previous approaches of quality control – worker behaviour and worker output.

The contributions of the paper include an interactive interface for crowd worker results, visualization for crowd behavior, techniques for exploration of crowd worker products and mixed initiative machine learning for bootstrapping user intuitions. The previous work includes analyzing the crowd worker behavior and output independently, whereas, CrowdScape provides an interface for analyzing them together. CrowdSpace utilizes mouse movement, scrolling, keypresses, focus events, and clicks to build worker profiles. Additionally, the paper also points out its limitations such as neglected user behaviors like focus of fovea. Furthermore, it shows the potential of CrowdSpace in other experimental setups which are primarily offline or cognitive and don’t require user movement on the system to analyze their behavior.

Reflection

CrowdSpace is a very necessary initiative as the number of users increases for evaluation. Another interesting aspect is that this also increases developers’ creativity and possibilities as he can now evaluate more complex and focus-based algorithms. However, I feel the need for additional compensation here. The crowd workers are being tracked and this is an intrusion to their privacy. I understand that this is necessary for the process to function but given that it makes focus an essential aspect of worker compensation, they should be awarded fairly for it.

Also, the user behaviors tracked here fairly cover most significant problems in the AI community. However, more inputs should cover a better range of problems. Adding more features would not only increase problem coverage but also lead to increase in development effort. There could be several instances when a developer does not build something due to lack of evaluation techniques or popular measures. Increasing features would help get rid of this concern. For example, if we are able to track the fovea of user’s, developers could not study the effect of different advertising techniques or build algorithms to predict and track interest in different varieties of videos (business of YouTube).

Also, I am not sure of the effectiveness of tracking the movements given in the paper. The paper considers effectiveness as a combination of worker’s behavior and output. But in several tasks you need mental models that do not require the movements tracked in the paper. In such cases, the output needs to have more weightage. I think the evaluator should be given the option to change the weights of different parameters, so that he could vary the platform for different problems making it more ubiquitous.

Questions

What kind of privacy concerns could be a problem here? Should the analyzer have access to such behavior? Is it fair to ask the user for his information? Should the user be additionally compensated for such intrusion to privacy?
What other kinds of user behaviors are traceable? The paper mentions fovea’s focus. Can we also track listening focus or mental focus in other ways? Where will this information be useful in our problems?
CrowdSpace uses the platform’s interactive nature and visualization to improve user experience. Should there be an overall focus on improving UX at the development level? Or should we let them be separate processes?
CrowdSpace considers worker behavior and output to analyze human evaluation. What other aspects could be used to analyze the results?

Word Count: 582

04/08/2020 – Yuhang Liu – CrowdScape: Interactively Visualizing User Behavior and Output

April 7, 2020April 8, 2020 yuhang Liu Leave a comment

Summary:

This article proposes that the crowdsourcing platform is a very good tool and can help people solve quite a few problems. It can help people quickly allocate work and complete tasks on a large scale. Therefore, the work quality of the workers on the crowdsourcing platform is very important. In the previous research, other researchers have developed algorithms on the inspection of the workers ’work quality and the workers’ behavior to detect the workers ’work quality, but these algorithms are all It has limitations, especially those complex tasks. The manual assessment of the quality of tasks completed by workers can solve this problem, but when many workers are needed, it cannot be scaled well. Therefore, based on this background, the author created CrowdScape, which can support the inspection of the work quality of crowdsourced workers through interactive visualization and hybrid active machine learning. The system’s approach is to combine information about worker behavior and worker output to better explain the work of the workers. This system mainly completes the exploration of workers’ work quality through the development and application of the following functions. : 1 Build an interface that can interactively browse the results of crowd workers, and explain the performance of workers by combining the information of workers’ behavior and output 2 Can visualize the behavior of mass workers 3 Can explore the products of mass workers 4 Tools for grouping and classifying workers 5 Combine machine learning to guide users’ intuition to the masses.

Reflection:

The system introduced in this article has many innovations, the most important of which is the ability to combine the behavior of workers with the output of workers, which is beneficial to study the behavior of workers with different performances. This is what we often say that the result is determined by the process. The author incorporates this into the research. I think it is a very innovative point, through this vigorous research, we can get the working behavior of workers with different results, and in the subsequent research, we can not only focus on workers with good output. The more important meaning is that the behavior guidance can be summarized through the behavior of workers with good output, and the behavior guidance can be used to guide the workers to complete the task better. And another innovation point I think is to visualize the interactive process. As we all know, people can better receive visual information. I think this is also feasible when evaluating the work effect of crowdsourced workers. Visualizing the interaction process of crowdsourced workers can help people better study the behavior of workers, and at the same time can improve people’s understanding of the performance of crowdsourced workers at work, and it also help us design ideas for crowdsourcing tasks. At the same time, the system’s dynamic query system can quickly analyze large data sets by providing users with immediate feedback. I think Crowddscape is based on these points in order to better discover people’s work patterns and understand the essence of crowdsourcing. And constantly adapt to more complex and innovative crowdsourcing tasks.

Question:

Whether the quality of workers ’work must be related to behavior, and how the system prevents some workers achieving good results with inappropriate behavior?
When workers know their behavior will be recorded, if it will affect their work?
Is there any other method to lead workers to have a better output?

04/08/2020 – Yuhang Liu – Agency plus automation: Designing artificial intelligence into interactive systems

April 7, 2020April 8, 2020 yuhang Liu Leave a comment

Summary:

This article discusses the relationship between artificial intelligence and staff. First of all, in modern society, with the rapid development of technology, the continuous development of artificial intelligence makes people more and more inclined to use the latest artificial intelligence to replace real people, but These ideas are usually based on very optimistic situations and underestimate the challenges of applying artificial intelligence. In general, artificial intelligence cannot complete these tasks without human resources. Therefore, in this case, we need to change our thinking. If we are not able to completely constitute automated technology to liberate human resources, then we can build computing auxiliary functions to strengthen and enrich people’s intellectual work. Therefore, the author of this article proposes to build a system that can realize rich interaction between people and algorithms. The system in this article is not intended to liberate the manpower in some systems. On the contrary, it is hoped that these manpower can be strengthened to better play the role of man in a system. This system balances the advantages of all aspects while promoting human control and action. The author applies this system to three aspects, and integrates the active computing in the interactive system to explain that this system is useful and can achieve the purpose of enhancing manpower. Finally, the author discusses the possibility of its future application.

Reflection:

I think the author ’s idea is very innovative. First of all, I have to admit that it is very difficult to fully implement the automation function. In my recent project, I have obvious feelings. In the natural scene, there is no labeled tweets whether it is a rumor. To implement supervised learning to automatically discover rumors in social networks, it is necessary to have marked data, which also reflects the importance of humans, so I need to use crowdsourcing platforms in my project. The system introduced in the article is to help people to play a greater role in a system and help people find better ways. I think this is very effective. Enhancing the role of people can undoubtedly increase efficiency. At the same time, I think that this system has also contributed to the subsequent research, not only providing new research ideas, when a field encounters major challenges or bottlenecks, we can consider changing the line of defense to study the problem. At the same time, I think that strengthening the role of people in a system can also better understand the role of people in an application, so that it can better study how to use artificial intelligence to replace humans in computer interaction in the future. In the end, the article also raises a deeper question, should we accept computers help us think, the application of such systems has no intention to raise the workers’ level, and to a large extent can help new hands adapt to a system, starting work more quickly. Such a system can largely allow people control system instead of passively accepting algorithms, so how to define the application direction of artificial intelligence in the future is also a problem that we need to think about.

Question:

Do you think the system mentioned in the paper can help new hands adapt a new system?
Do you think extend people’s ability by computer can help us research the role of people in human and computer interaction?
Should we accept the idea that “computers help people think”, will it lead other problems?

04/08/20 – Ziyao Wang – Agency plus automation: Designing artificial intelligence into interactive systems

April 7, 2020April 8, 2020 Ziyao Wang 1 Comment

The author proposed that currently much developers and researchers are too optimistic as they do not see human labor under automated services. This kind of long-standing focus on only the AI side results in the lack of researches about the interaction between AI and humans. The author proposed that it is better to let AI enrich human’s intellectual work rather than let AI replace human. The author introduced interactive systems in three areas, which are data wrangling, exploratory analysis, and natural language translation. The author integrated integrate proactive computational support into these systems and described the performance of these hybrid systems using predictive models. In conclusion, he proposed that only a fluent interleaving of both automated suggestions and direct manipulation can enable more productive and flexible work.

Reflections:

As a student who had only one class about human-computer interaction before attending this course, I kept the wrong view that we should focus on the AI side of the data analytic side. Instead, there is a huge improvement in user experience if the system has a good design of human-computer interaction. The author of this paper reminds me of the importance of the interface and how AI can do for human works.

It is not correct to let a human do what they can do, let AI do want they can do and combine both sides of work simply. Instead, the interaction between humans and AI can be designed to leverage the strength of both sides and make the whole system have better performance. I like the idea about let AI to suggest for human when human is dealing with some work. This can let the results to be highly accurate and let the processing speed to decrease. Human does not need to search for information online and AI does not need to predict what is human thinking. They can just cooperate to let humans ensure accuracy while AI provides quick background searching and complex calculation. They reached harmony in the designed system.

Additionally, the author mentioned that it is not always efficient to let AI support human. For some simple job, AI can complete on itself. In these cases, it would be more efficient to let AI do the task on itself without asking advice from human workers. For a simple job, for example, some simple labeling, even human gives no advice, AI can still reach high accuracy. If we let AI wait for human response, time would be wasted. As a result, a fluent interleaving of both automated suggestions and direct manipulation can reach the highest performance of a system. We should consider both suggestion and automatic operations in our system designs.

Questions:

The author proposed that systems should have a fluent interleaving of both automated suggestions and direct manipulation. Is there any situation in which AI assistants would decrease the performance of humans and humans should complete the tasks on themselves?

What’re the criteria for determining whether the AI should make suggestions to human or it should direct manipulation?

Compared with letting humans correct AI results, what’re the advantages of letting AI suggest humans?

04/08/2020 – Nan LI – Agency plus automation: Designing artificial intelligence into interactive systems

April 4, 2020April 8, 2020 NAN LI 2 Comments

Summary:

The main objective of this paper is to present the significance and benefit of integrating the interactive system with artificial intelligence. The author made a point that the current focus on AI limited on full automation, which may lead to a mislead due to inappropriate assumptions or biased training data. As a result, users may rely excessively on computational advice, which may result in the loss of critical participation and domain expertise. To address this issue, the author proposed the idea that integrates AI agency and automation to enhance human ability instead of replacing human work. This approach aims to increase human productivity while preserving the human sense of control and responsibility. In order to investigate the most effective way of integrating the automated method into user-centric interactive systems, the paper examined the strategies. Designing shared representations of possible actions that enable people to perform computational reasoning around tasks so that people can view, select, modify, or cancel algorithm suggestions. Then, the author review three practical implementations that applied principles proposed as above: domain-specific language (DSL) for data transformation called Wrangle; an interactive system for data visualization and exploration, predictive translation memory (PTM) project. Finally, the author discussed the design property and user studies of these three projects.

Reflection:

I really like the idea that the primary goal of AI should be enhancing humans not replace humans. I think the author made a good point that currently, people tend to focus too much on the fully automated implementation of AI. The claim that AI can replace humans is even more exaggerated. Humans indeed make mistakes in judgment due to insufficient information or cognitive biases. However, human has the irreplaceable creativity and in-depth understanding of professional domain knowledge. Even with the help of dominant computing power and exhaustive data form computers, AI institutions cannot achieve it. Thus, I cannot more agree with that “need well-thought-out interactions of humans and computers to solve our most pressing problems.”

The reason why I am interested and passionate about HCI is that the elegant of HCI is that the subtle and simple design idea could have a huge impact on the overall performance. We are not seeking to develop a new interactive system or fancy interface, but design from a humble direction to make subtle and rational adjustments to improve user experience without causing any interruption. Just as the example demonstrated in the paper: spelling and grammar checking routines included within the word processor.

On the other hand, user feedback in the article caught my attention: “These related views are so good, but it’s also spoiling that I start thinking less. I am not sure if that’s really a good thing”. This is really impressive. For a long time, we have been pursuing how to facilitate human work efficiently and easily through using AI or interactive design to replace some tedious and later even developed to let machines to learn adaptively instead of human thinking. However, this comment just pointed out the question, which also stated in the paper that should we accept having the computer ‘think for us’? This is indeed a problem that we need to consider when designing.

Questions:

What do you think about the opinion that AI should “enhancing us, not replacing us”? Are you agree with the idea that we should integrate AI with IA perspectives?
Which do you support more? Direct manipulation, or interface agents? Why?
Take the data visualization system as an example. Would you like the system to provide adaptive recommendations? Do you think it’s helpful or annoying? Do you think it is spoiling and stop you from thinking initiative?