03/04/2020 – Dylan Finch – Real-Time Captioning by Groups of Non-Experts

March 4, 2020 Dylan Finch 3 Comments

Word count: 564

Summary of the Reading

This paper aims to help with accessibility of audio streams by making it easier to create captions for deaf listeners. The typical solution to this problem is to hire expensive, highly trained professionals who require specialized keyboards, stenographers. Or, in other cases, people with less training to create captions, but these captions may take longer to write, creating a latency between what is said in the audio and the captions. This is not desirable, because it makes it harder for the deaf person to connect the audio with any accompanying video. This paper aims to marry cheap, easy to produce captions with the ability to have the cpations created in real time and with little latency. The solution is to use many people who do not require specialized training. When working together, a group of crowd workers can achieve high caption coverage of audio with a latency of only 2.9 seconds.

Reflections and Connections

I think that this paper highlights one of the coolest things that crowdsourcing can do. It can take big, complicated tasks that used to require highly trained individuals and make them accomplishable by ordinary people. This is extremely powerful. It makes all kinds of technologies and techniques much more accessible. It is hard to hire one highly trained stenographer, but it is easy to hire a few normal people. This is the same idea that powers Wikipedia. Many people make small edits, using specialized knowledge that they know, and, together, they create a highly accessible and complete collection of knowledge. This same principle can and should be applied to many more fields. I would love to see what other professions could be democratized through the use of many normal people to replace one highly trained person.

This research also shows how it is possible to break up tasks that may have traditionally been thought of as atomic. Transcribing audio is a very hard task to solve using crowd workers because there are not real discrete tasks that could b e sent to crowd workers. The stream of audio is continuous and always changing. However, this paper shows that it is possible to break up this activity into manageable chunks that can be accomplished by crowd workers, the researchers just needed to think outside of the box. I think that this kind of thinking will become increasingly important as more and more work is crowdsourced. I think that as we learn how to solve more and more problems using crowdsourcing, the issue becomes less and less ot can we solve this using crowdsource and becomes much more about how can we break up this problem into manageable pieces that can be done by the crowd. This kind of research has applications elsewhere, too. I think that in the future this kind of research will be much more important.

Questions

What are some similar tasks that could be crowdsourced using a method similar to the one described in the paper?
How do you think that crowdsourcing will impact the accessibility of our world? Are there other ways that crowdsourcing could make our world more accessible?
Do you think there will come a time when most professions can be accomplished by crowd workers? What do you think the extent of crowd expertise will be?

03/04/2020 – Dylan Finch – Pull the Plug?

March 4, 2020 Dylan Finch 1 Comment

Word count: 596

Summary of the Reading

The main goal of this paper is to make image segmentation more efficient. Image segmentation as it is now, requires humans to help with the process. there are just some images that machines cannot segment on their own. However, there are many cases where an image segmentation algorithm can do all of the work on its own. This presents a problem: we do not know when we can use an algorithm and when we have to use a human, so we have to have humans review all of the segmentations. This is highly inefficient. This paper tries to solve this problem by introducing an algorithm that can decide when a human is required to segment an image. The process described in the paper involves scoring each segmented image done by machines, then giving humans the task of reviewing the lowest scoring images. Overall, the process was very effective and saved a lot of human effort.

Reflections and Connections

I think that this paper gives a great example of how humans and machines should interact, especially when it comes to humans and AIs interacting. Often times, we set out in research with the goal of creating a completely automated process that throws the human away and tries to create an AI or some other kind of machine that will do all of the work. This is often a very bad solution. AIs as they currently are, are not good enough to do most complex tasks all by themselves. In the cases of tasks like image segmentation, this is an especially big issue. These tasks are very easy for humans to do and very hard for AIs to do. So, it is good to see a researcher who is willing to use human strengths to make up for the weaknesses of machines. I think it is a good thing to have both things working together.

This paper also gives us some very important research, trying to answer the question of when we should machines and when we should use humans. This is a very tough question and it comes up in a lot of different fields. Humans are expensive, but machines are often imperfect. It can be very hard to decide when you should use one or the other. This paper does a great job of answering this question for image segmentation and I would love to see more similar research in other fields explain when it is best to use humans and machines in those fields.

While I like this paper, I do also worry that it is simply moving the problem, rather than actually solving it. Now, instead of needing to improve a segmentation algorithm, we need to improve the scoring algorithm for the segmentations. Have we really improved the solution or have we just moved the area that now needs further improvement?

Questions

How could this kind of technology be used in other fields? How can we more efficiently use human and machine strengths together?
In general, when do you think it is appropriate to create a system like this? When should we not fully rely on AI or machines?
Did this paper just move the problem, or do you think that this method is better than just creating a better image segmentation algorithm?
Does creating systems like this stifle innovation on the main problem?
Do you think machines will one day be good enough to segment images with no human input? How far off do you think that is?

02/26/2020 – Dylan Finch – Will You Accept an Imperfect AI? Exploring Designs for Adjusting End-user Expectations of AI Systems

February 25, 2020 Dylan Finch 1 Comment

Word count: 556

Summary of the Reading

This paper examines the role of expectations and the role of focusing on certain types of errors to see how this impacts perceptions of AI. The aim of the paper is to figure out how the setting of expectations can help users better see the benefits of an AI system. Users will feel worse about a system that they think can do a lot and then fails to live up to those expectations rather than a system that they think can do less and then succeeds at accomplishing those smaller goals.

Specifically, this paper lays some ways to better set user expectations: an Accuracy Indicator which allows users to better expect what the accuracy of a system should be, an explanation method based on examples to help increase user understanding, and the ability for users to adjust the performance of the system. They also show the usefulness of these 3 techniques and that systems tuned to avoid false positives are generally worse than those tuned to avoid false negatives.

Reflections and Connections

This paper highlights a key problem with AI systems: people expect them to be almost perfect and companies market them as such. Many companies that have deployed AI systems have not done a good job managing expectations for their own AI systems. For example, Apple markets Siri as an assistant that can do almost anything on your iPhone. Then, once you buy one, you find out that it can really only do a few very specialized tasks that you will rarely use. You are unhappy because the company sold you a much more capable product. With so many companies doing this, it is understandable that many people have very high expectations for AI. Many companies seem to market AI as the magic bullet that can solve any problem. But, the reality is often much more underwhelming. I think that companies that develop AI systems need to play a bigger role in managing expectations. They should not sell their products as a system that can do anything. They should be honest and say that their product can do some things but not others and that it will make a lot of mistakes, that is just how these things work.

I think that the most useful tool this team developed was the slider that allows users to choose between more false positives and more false negatives. I think that this system does a great job of incorporating many of the things they were trying to accomplish into one slick feature. The slider shows people that the AI will make mistakes, so it better sets user expectations. But, it also gives users more control over the system which makes them feel better about it and allows them to tailor the system to their needs. I would love to see more AI systems give users this option. It would make them more functional and understandable.

Questions

Will AI ever become so accurate that these systems are no longer needed? How long will that take?
Which of the 3 developed features do you think is most influential/most helpful?
What are some other ways that AI developers could temper the expectations of users?

02/26/2020 – Dylan Finch – Explaining Models: An Empirical Study of How Explanations Impact Fairness Judgment

February 23, 2020February 25, 2020 Dylan Finch Leave a comment

Word count: 573

Summary of the Reading

This paper investigates explaining AI and ML systems. An easy way to explain AI and ML systems is to have another computer program to help generate an explanation of how the AI or ML system works. This paper works towards that goal, comparing 4 different programmatically generated explanations of AI And ML systems and seeing how they impact judgments of fairness. These different explanations had a large impact on perceptions of fairness and bias in the systems, with a large degree of variation between each of the explanation systems.

Not only did the kind of explanation used have a large impact on the perceived fairness of the algorithm, but the pre-existing feelings of the participants towards AI and ML and bias in these fields also had a profound impact on whether or not participants saw the explanations as fair or not. People who did not already trust AI fairness equally distrusted all of the explanations.

Reflections and Connections

To start, I think that this type of work is extremely useful to the future of the AI and ML fields. We need to be able to explain how these kinds of systems work and there needs to be more research into that. This issue of explainable AI becomes even more important when we put it in the context of making AI fair to the people who have to interact with it. We need to be able to tell if an AI system that is deciding whether or not to free people from jail is fair or not. The only way we can really know if these models are fair or not is to have some way to explain the decisions that the AI systems make.

I think that one of the most interesting parts of the paper is the variation in the number of people with different circumstances who thought that the models were fair or not. Pre-existing ideas about whether or not AI systems are fair had a huge impact on whether or not people thought these models were fair when given an explanation of how they work. This shows how human of a problem this is and how hard it can be to decide if a model is fair or not, even when you have access to an explanation. Views of the model will differ from person to person.

I also found it interesting how the type of explanation used had a big impact on the judgment of fairness. To me, this congers up ideas of a future where the people who build algorithms can just pick the right kind of explanation to prove that their algorithm is fair, in the same way companies now use language in a very questionable way. I think that this field still has a long way to go and that it will become increasingly important as AI penetrates more and more fasciates of our lives.

Questions

When each explanation produces such different results, is it possible to make a concrete judgment on the fairness of an algorithm?
Could we use computers or maybe even machine learning to decide if an algorithm is fair or would that just produce more problems?
With so many different opinions, even when the same explanation is used, who should be the judge if an algorithm is fair or not?

2/19 – Dylan Finch – In Search of the Dream Team:Temporally Constrained Multi-Armed Bandits forIdentifying Effective Team Structures

February 18, 2020 Dylan Finch 1 Comment

Word count: 517

Summary of the Reading

This paper seeks to help make it faster and easier for teams to find their ideal team structure. While many services allow teams to test out many different team structures to find the best one, many of those services can take a lot of time and can greatly affect the people who work on the team. Often times they have to switch structures so often that it makes it hard for the teams to concentrate on getting work done.

The method proposed in the paper seemed to be very successful. It resulted in teams that were 38-46% more effective. The system works by testing different team structures and taking automatically generated feedback information (like performance metrics) to figure out how effective each structure is. It will then base its future combinations on this feedback. Each time a new structure is tested, it varies on a five dimensions: hierarchy, interaction patterns, norms of engagement, decision-making norms, and feedback norms.

Reflections and Connections

I think that this paper has an excellent idea for a system that can help teams to work better together. One of the most important things about a team is how it is structured. The structure of a team can make or break its effectiveness, so getting the structure right is very important to making an effective team. A tool like this that can help a team figure out the best structure with minimal interruption will be very useful to everyone in the business world who needs to manage a team.

I also thought that it was a great idea to integrate the system into Slack. When I worked in industry last summer, all of the teams at my company used Slack. So, it makes a lot of sense to implement this new system in a system that people are already familiar with. The use of Slack also allows the creators to make the system more friendly. I think it is much better to get feedback from a human-like Slack bot than some other heartless computer program. It is also very cool how the team members can interact with the bot in Slack.

I also found the dimensions that they used in the team structures to be interesting. It is valuable to be able to classify teams in some concrete way based on certain dimensions of how they perform. This also has a lot of real world applications. I think that a lot of the time, one of the hardest things in any problem space is just to quantify the possible states of the system. They did this very nicely with the team dimensions and all of their values.

Questions

Would you recommend this system to your boss at your next job as a way to figure out how to organize the team?
Aside from the ones listed in the paper, what do you think could be some limitations of the current system?
Do you think that the possible structures had enough dimensions and values for each dimension?

2/19 – Dylan Finch – The Work of Sustaining Order in Wikipedia: The Banning of a Vandal

February 18, 2020 Dylan Finch 2 Comments

Word count: 565

Summary of the Reading

This paper analyzes the use of autonomous technologies that are used on Wikipedia. These technologies help to keep the peace on the large platform, helping to flag malicious users and revert inaccurate and spammy changes so that Wikipedia stays accurate and up to date. Many people may think that humans play the major role in policing the platform, but machines and algorithms also play a very large part, aiding the humans to deal with the large amount of edits.

Some tools are completely automated and can prevent vandalism with no human input. Other tools give human contributors tips to help them spot and fight vandalism. Humans work together with the automated systems and each other to edit the site and keep the pages vandal free. The way in which all of the editors edit together, even though they are not physically together or connected as a team, is an impressive feat of human and AI interaction.

Reflections and Connections

To start, I think that Wikipedia is such and interesting thing to examine for a paper like this. While many organizations have a similar structure, I think that WIkipedia is unique and interesting to study because it is so large, so distributed, and so widely used. It can be hard enough to get a small team of people to work together on documentation. At Wikipedia’s size the complexities of making it all work must be unimaginable. It is so interesting to find out how machines and humans work together at that scale to keep the site running smoothly. The ideas and analysis seen here can easily be applied to smaller systems that are trying to accomplish the same thing.

I also think that this article serves as a great reminder of the power of AI. The fact that AI is able to do some much to help editors keep the site running smoothly even with all of the complexities of the site is amazing and it shows just how much power AI can have when applied to the right situation. A lot of the work done on Wikipedia is not hard work. The article mentions some of the things that bots do, like importing data and fixing grammatical mistakes. These things are incredibly tedious for humans to do and yet they are perfect work for machines. They can do this work almost instantly while it may take a human an hour. This not only serves as a great reminder of the power of AI’s and humans complimenting each other’s abilities, but it also shows what the power of what the internet can do. Something like this never would have been possible before in the history of human civilization. The mere fact that we can do something like this now speaks to the amazing power of the current age.

Questions

Does this research have applications elsewhere? What would be the best place to apply this analysis?
Could this process ever be done with no human input whatsoever? Could Wikipedia one day be completely self sufficient?
This article talks a lot about how the bots of Wikipedia are becoming more and more important, compared to the policies and social interactions between editors. Is this happening elsewhere? Are there bots other places that we might not see and might not notice, even though they are doing a larger and larger share of the work?

02/05/20 – Dylan Finch – Power to the People: The Role of Humans in Interactive Machine Learning

February 5, 2020February 5, 2020 Dylan Finch 1 Comment

Summary of the Reading

Interactive machine learning is another form of machine learning that allows for much more precise and continuous changes to the model, rather than large updates that drastically change the model. In interactive machine learning models, domain experts are able to continuously update the model as it produces results, reacting to the predictions it makes in almost real time. Examples of this type of machine learning system include online recommender systems like those on Amazon and Netflix.

In order for this type of system to work, there needs to be an oracle who can correctly label data. Usually this is a person. However, people do not like being an oracle and in some cases, they can be quite bad at it.Humans would also like richer more rewarding interactions with the machine learning algorithms. The paper suggests some way that these interactions could be made richer for the person training the model.

Reflections and Connections

At the end of the paper, the authors say that these new types of interaction with interactive machine learning is a potentially powerful tool that needs to be applied to the right circumstances. I completely agree. I think that this technology, like all technologies, will be useful in some places and not in others. I think that in cases of a simple recommender system, most people are happy to just give a rating every now and then or answer a survey question every now and then. In cases like this, I think that richer interactions would take away from the simplicity and usefulness of the system. But in other cases, it would be nice to be able to kind of work with the machine learning model to generate better answers in the future.

I also think that in some fields, technologies like the ones presented in his paper will be extremely valuable. I think that in life, it is very easy to get stuck in a rut and to not be able to think outside of the ways that we have always done things. But, it is important to do that to push technology forward. We have always thought of machine learning as an algorithm asking an oracle about specific examples. When we create interactive machine learning, we replaced the oracle with a person and applied the same ideas. But, as this paper points out, people are not oracles and they don’t like to be treated like them. So the ideas in this paper could be very impromat to unlock new ways of using machine learning in conjunction with people. And, the more we play to the strengths of people, we will be able to create better machine learning algorithms that take advantage of those strengths.

Questions

What is one place you think could use interactive machine learning besides recommender systems?
Which of the presented models for new ways for people to interact with machine learning algorithms do you think has the most promise?
Can you think of any other new interfaces for interactive machine learning not mentioned in the paper?

02/05/20 – Dylan Finch – Principles of mixed-initiative user interfaces

February 5, 2020February 5, 2020 Dylan Finch 1 Comment

Summary of the Reading

This paper seeks to help solve some of the issues present with automation in software. Often times, when a user’s tries to automate an action using an agent or tool, they may not get the result they were expecting. The paper lists many of the key issues with the then current implementations of this system.

The paper points out many of the issues that can plague systems that try to take action on behalf of the user. These include things like not adding value for the user, not considering the agent’s uncertainty about the user’s goals, not considering the status of the user’s attention when trying to suggest an action, not inferring the ideal action in light of costs and benefits, not employing a dialog to resolve key conflicts, and many others. After listing these key problems, the authors go on to describe a system that tries to solve many of these issues.

Reflections and Connections

I think that this paper does a great job of finding listing the obstacles that exist for systems that try to automate tasks for a user. It can be very hard for a system to automatically do some tasks for the user. Many times, the intentions of the user are unknown. For example, an automatic calendar event creating agent may try to create a calendar hold for a birthday party for a person that the user does not care about, so they would not want to go to the birthday party. There are many times where a user’s actions depend on much more than simply what is in an email or what is on the screen. That is why it is so important to take into account that fact that the automated system could be wrong.

I think that the authors of this paper did a great job trying to plan for and correct items when the automated system is wrong about something. Many of the key issues they identify have to do with the agent trying to correctly guess when the user does actually need to use the system and what to do when that guess is wrong. I think that the most important issues that they list are the ones that have to do with error recovery. No system will be perfect, so you should at least have a plan for what will happen when the system is wrong. The system that they describe is excellent in this department. It will automatically go away if the user does not need it, and it will use dailogs to get missing information and correct mistakes. This is exactly what a system like this should do when it encounters an error or does something wrong. There should be a way out and a way for the user to correct the error.

Questions

Which of the critical factors listed in the paper do you think is the most important? The least?
Do you think that the system the developed does a good job meeting all of the issues they brought up?
Agents are not as popular as they used to be and this article is very old. Do you think these ideas still hold relevance today?

01/29/20 – Human Computation: A Survey and Taxonomy of a Growing Field

January 28, 2020January 29, 2020 Dylan Finch 1 Comment

Summary of the Reading

This paper is a survey of the research in the field of human computation. The paper aims to classify human computation systems so that the similarities between different projects and the holes in current research can be seen more clearly. The paper also explores related fields like crowdsourcing.

The paper starts by defining human computation as “a paradigm for utilizing human processing power to solve problems that computers cannot yet solve.” The paper then goes on to discuss the differences between human computation and crowdsourcing, social computing, data mining, and collective intelligence.

The paper then goes on to classify human computation systems based on 6 dimensions: motivation, quality control, aggregation, human skill, process orders, and task-request cardinality. Each of these dimensions has several discrete options. Putting all of these dimensions together allows for the classification of any arbitrary human computation system. The paper also provides examples of systems that have various values on each dimension.

Reflections and Connections

I think this paper provides a much needed tool for further human computation and crowdsourcing research. The first step to understanding something is being able to classify that thing. This tool will allow current and future researchers to classify human computation systems so that they can apply existing research to other, similar systems and also allow them to see where existing research falls short and where they need to focus future research.

This research also provides an interesting perspective on current human computation systems. It is interesting to see how current human computation systems compare to each other and what each system does differently and what they have in common.

I also like the malleability of the classification system. They say in the future work section that this system is very easy to add to. Future researchers who continue the work on this project could easily add values to each of the dimensions to better classify the human computation systems. They could also add values to the dimensions if new human computation systems are invented and need to be classified using this system. There are a lot of good opportunities for growth from this project.

One thing that I thought this paper was missing is a direct comparison of different human computation systems on more than one dimension. The paper uses human computation systems as examples in the various values for each of the dimensions of the classification system, but it doesn’t put these dimensions together and compare the human computation systems on more than one dimension. I think this would have added a lot to the paper, but it would also make for a great piece of future work for this project. This idea is actually very similar to the other paper from this week’s bunch, titled “Beyond Mechanical Turk: An Analysis of Paid Crowd Work Platforms” and I think it would be helpful and build on both of these papers.

Questions

Do you think the classification system presented in this paper has enough dimensions? Does it have too many?
What is one application you see for this classification system?
Do you think this classification system will help crowdsourcing platforms deal with some of their issues?

01/29/20 – Dylan Finch – The Future of Crowd Work

January 28, 2020January 28, 2020 Dylan Finch Leave a comment

Summary of the Reading

This paper looks to build a model of crowd work to better understand some of the major challenges that the future of crowd work will face. The paper also collects data and makes suggestions on how to make the future of crowd work better for all those involved.

One of the main contributions of this paper is building a model to represent crowd work. They do this by using not only theoretical data, but also empirical data that is gathered from both requesters and workers on crowd work websites. This model helps the authors to visualize some of the problems that come from crowd work and allow them to better understand possible solutions.

The paper then goes into depth on 12 research foci that are part of the model. They consider the future of the crowd work processes, crowd computation, and crowd workers. After this, they synthesize these foci into concrete recommendations for the future of the industry, including creating career ladders, improving task design and communication, and facilitating learning.

Reflections and Connections

I think that this paper looks at a very important part of crowd work. Crowd work is a very powerful tool, but if we don’t do it right, it could lead to a terrible future where work is cheepened and workers are not respected. We need to make sure that while this industry is still young that it is shaped in a way that will allow it to be sustainable well into the future. Articles like these are important so that we can know what we need to change now, when the industry is still young so that when it grows up, it can be a great part of the economy.

I also like that the article has a very optimistic view of crowd work. The technology is very powerful and can allow people to do things that never could have been done without it, or could only have been done at a tremendous cost. If we put in the work to make this industry work well for its users, then we will be able to use crowd work to accomplish many great things and many people will be able to use it to make ends meet or to get an extra income on their own time. Crowd work unlocks new, more efficient ways to accomplish many old goals.

I also think that the goals laid out in this article will be very good for the industry. The idea of creating career ladders would be extremely beneficial to everyone involved. Many people in traditional jobs only try their best because of the possibility of moving up in the company. So, a job where you can’t move up will make people less invested and less motivated. The idea of introducing better communication would also be great for the industry. There will always be times when a task is poorly worded and a quick email can save hours of wasted time and effort. The facilitation of learning also seems extremely beneficial to everyone involved and would also increase the quality of tasks and results.

Questions

Considering this article was published in 2013, do you think the industry is moving in the right direction?
How practical do you think these recommendations are? Are they too ambitious?
Which of the recommendations do you think is most important for the industry to implement?