Author: Dylan Finch

04/29/2020 – Dylan Finch – IdeaHound: Improving Large-scale Collaborative Ideation with Crowd-powered Real-time Semantic Modeling

Dylan Finch 2 Comments

Word count: 550

Summary of the Reading

This paper aims to create a system that will improve large-scale collaboration by making collaboration easier. These researchers aim to accomplish this by giving idea creators access to a new and better form of semantic modeling that would allow them to more easily see all of the ideas that have been created so far on a project. They want this new system to be nearly-real time and self-sustainable (they want to limit external labor). 

The system has 3 main components: (1) it allows users to request to see the ideas of others from a global idea map, with some of the returned ideas will be very different from each other; (2) the user can request to see ideas similar to their own idea; and (3) the user can request to see a map of ideas, that shows how they relate to each other and how similar they are. 

An evaluation was conducted and found that on average most people thought that the system was helpful. 

Reflections and Connections

I find this idea to be very interesting because it seems like if it was done correctly, it would have many applications. I took an HCI class in undergrad which covered design (and ideation) and I always found ideation to be a hard process. The system proposed here could help with many parts of the ideation process. One of the hardest parts of ideation is keeping track of all the ideas. Sometimes, it can be overwhelming to try and organize all of the ideas that a group has come up with. When you get to a certain number of ideas, you just can’t cope with the sheer volume of information. It becomes extremely hard to organize the ideas in a coherent way and ideas you see one second can become lost in another. A system like this would make dealing with all these ideas much easier. Plus, with the weight of managing ideas lifted, the contributors would be free to come up with even more ideas.

I think that when it comes to ideaton, there really is no magic bullet solution. The fact is, ideation simply involves too much information. The human brain simply cannot deal with all of the information that comes with doing ideation. So, there is no perfect solution that will make ideation easy. This system does not eliminate all of the struggles of ideation. It is still hard to keep all the ideas you need in your head and it is still hard to deal with all the relevant information. But, this system does vastly improve on older, traditional methods, like sticky notes. This system makes it much easier to find ideas you are looking for and to manage all the ideas you’ve had. That is about as good as it gets when it comes to ideation.

Questions

  1. Could this system do anything else to help make ideation easier?
  2. What part of this system would you use most when you’re doing ideation?
  3. What makes it so hard to automate all or part of the ideation process? Can we ever achieve full automation of the whole process or even parts of it beyond what this system shows?

Read More

04/29/2020 – Dylan Finch – Accelerating Innovation Through Analogy Mining

Dylan Finch 1 Comment

Word count: 579

Summary of the Reading

This paper works to make it easier to find analogies in large, unstructured datasets that cover a variety of domains. The work has many real world applications, targeting real world examples of large, unstructured datasets, like data from the US patent office. The system works by collecting data about each entry in the dataset. This data includes the purpose and the mechanism that achieves the purpose. By collecting both of these features, it makes it easier to find analogies. 

This system was evaluated to see how well it helped with ideation by analogy. The evaluation used crowd workers to see how effective the new method is. Many workers were asked to come up with new product ideas. To help, these workers were shown example products. Some were shown products found to be similar using the system described in this paper. Others were shown products found to be similar by other means and others were shown random products. The evaluation showed that this system helped the workers to come up with better ideas than the other 2 methods.

Reflections and Connections

I think this problem is really one that spans generations. The paper brings up the troubles of trying to use a real world dataset like the one from the US patent office. In a case like this, I think that the data presents many different challenges, depending on when you look. For data from the past, there are probably inconsistencies in the formats of the data, the data may be in multiple different sources, and some of it may have been lost or changed over time. For data from the present, there is just so much of it. With so many more people and intellectual property more valuable than ever, the US patent office probably has more data about inventions than they can deal with. These represent two very different challenges that a dataset like the one from the US patent office face and they are a great reason why research like this is so sorely needed.

The idea of this paper also reminds me of an idea from last week’s papers: SOLVENT. Both papers try to make it easier for researchers to find analogies in data sources. In fact the existence of both of these papers I think helps to illustrate the need for technology like this. In fact, neither of these papers cite each other even though they are working on very similar research. Perhaps if there had been a widely available version of SOLVENT, they would have been able to find each other’s papers and build off of each other. 

I think that as these datasets get larger and larger, the need for easier ways to access things from them will become more and more important. The number of patents and papers is growing quicker than ever before and that means that it is easier than ever for valuable knowledge to be lost. We need to start implementing more ideas like this so that we don’t lose important knowledge. I hope that the existence of both of these papers helps to show others the real need for technology like this.

Questions

  1. Do you think this system is better or worse than SOLVENT?
  2. What is another real world, unstructured data source that a system like this might work well on?
  3. What are some applications for this system outside of ideation?

Read More

04/22/2020 – Dylan Finch – Opportunities for Automating Email Processing: A Need-Finding Study

Dylan Finch 1 Comment

Word count: 586

Summary of the Reading

This paper investigates automation with regards to emails. A large portion of many people’s days is devoted to sifting through the hundreds of emails that they receive. Many of the tasks that go into this might be automatable. This paper not only looks at how different tasks related to dealing with emails can be automated, but it also investigates the opportunities for automation in popular email clients. 

The paper found that many people wanted to automate tasks that required more data from emails. Users wanted access to things like the status of the email (pending, done, etc.), the deadline, the topic, the priority, and many other data points. The paper also noted that people would like to be able to aggregate responses to emails to more easily see things like group responses to an event. Having access to these features would allow for users to better manage their inboxes. Some current solutions exist to these issues, but some automation is held back by limitations in email clients.

Reflections and Connections

I love the idea of this paper. I know that ever since I got my email account, I have loved playing around with the automation features. When I was a kid it was more because it was just fun to do, but now that I’m an adult and receive many more emails than back then (and many more than I would like), I need automation to be able to deal with all of the emails that I get on a daily basis. 

I use Gmail and I think that it offers many good features for automating my inbox. Most importantly, Gmail will automatically sort mail into a few major categories, like Primary, Social, and Promotions. This by itself is extremely helpful. Most of the important emails get sent to the Primary tab so I can see them and deal with them more easily. The Promotions tab is also great at aggregating a lot of the emails I get from companies about products or sales or whatever that I don’t care about most of the time. Gmail also allows users to make filters that will automatically do some action based on certain criteria about the email. I think both of these features are great. But, it could be so much more useful.

As the paper mentions, many people want to be able to see more data about emails. I agree. The filter feature in Gmail is great, but you can only filter based on very simple things like the subject of the email, the date it was sent, or the sender. You can’t create filters for more useful things like tasks that are listed in the email, whether or not the email is an update to a project that you got other emails about, or the due date of tasks in the email. Like the paper says, these would be useful features. I would love a system that allowed me to create filters based on deeper data about my emails. Hopefully Gmail can take some notes from this paper and implement new ways to filter emails.

Questions

  1. What piece of data would you like to be able to sort emails by?
  2. What is your biggest problem with your current email client? Does it lack automation features? 
  3. What parts of email management can we not automate? Why? Could we see automatic replies to long emails in the future?

Read More

04/22/2020 – Dylan Finch – SOLVENT: A Mixed Initiative System for Finding Analogies Between Research Papers

Dylan Finch 1 Comment

Word count: 566

Summary of the Reading

This paper describes a system called SOLVENT, which uses humans to annotate parts of academic papers like the high-level problems being addressed in the paper, the specific lower-level problems being addressed in the paper, how the paper achieved its goal, and what was learned/achieved in the paper. Machines are then used to help detect similarities between papers so that it is easier for future researchers to find articles related to their work.

The researchers conducted three studies where they showed that their system greatly improves results over similar systems. They found that the system was able to detect near analogies between papers and that it was able to detect analogies across domains. One interesting finding was that even crowd workers without extensive knowledge about the paper they are annotating can produce helpful annotations. They also found that annotations could be created relatively quickly.

Reflections and Connections

I think that this paper addresses a real and growing problem in the scientific community. With more people doing research than ever, it is increasingly hard to find papers that you are looking for. I know that when I was writing my thesis, it took me a long time to find other papers relevant to my work. I think this is mainly because we have poor ways of indexing papers as of now. Really the only current ways that we can index papers are by the title of the paper and by the keywords embedded in the paper, if they exist. These methods can help find results, but they are terrible when they are the only way to find relevant papers. A title may be about 20 words long, with keywords being equally short. 40 words does not allow us to store enough information to fully represent a paper. We lose even more space for information when half of the title is a clever pun or phrase. These primitive ways of indexing papers also lose much of the nuance of papers. It is hard to explain results or even the specific problem that a paper is addressing in 40 words. So, we lose that information and we cannot index on it. 

A system like the one described in this paper would be a great help to researchers because it would allow them to find similar papers much more easily. This doesn’t even mention the fact that it lets researchers find papers outside of their disciplines. That opens up a whole new world of potential collaboration. This might help to eliminate the duplication of research in separate domains. Right now, it is possible that mathematicians and computer scientists, for example, try to experiment on the same algorithm, not knowing about the team from the other discipline. This wastes time, because we have two groups researching the same thing. A system like this could help mitigate that.

Questions

  1. How would a system like this affect your life as a researcher?
  2. Do you currently have trouble trying to find papers or similar ideas from outside your domain of research?
  3. What are some limitations of this system? Is there any way that we could produce even better annotations of research papers?
  4. Is there some way we could get the authors of each paper to produce data like this by themselves?

Read More

04/15/2020 – Dylan Finch – Believe it or not: Designing a Human-AI Partnership for Mixed-Initiative Fact-Checking

Dylan Finch Leave a comment

Word count: 567

Summary of the Reading

This paper presents the design and evaluation of a system that is designed to help people check the validity of claims. The system starts with a user entering a claim into the system. The system then shows the user a list of articles related to the claim with a prediction that is based on the articles, about whether or not the claim is true. The system will give a percentage chance for whether the article is true. Each article that the system shows also shows a reputation score for the source of the article and a support score for the article. The user can then adjust these if they don’t think that the system has accurate information. 

The system seemed to help users come to the right conclusions when it had the right data but also seemed to make human judgements worse when the system had inaccurate information. This shows the usefulness of such a system and also gives a reason to be careful about implementing it.

Reflections and Connections

I think that this article tackles a very real and very important problem. Misinformation is more prevalent now than ever, and it is getting harder and harder to find the truth. This can have real effects on people’s lives. If people get the wrong information about a medication or a harmful activity, they may experience a preventable injury or even death. Misinformation can also have a huge impact on politics and may get people to vote in a way they might not otherwise.

The paper brings up the fact that people may over rely on a system like this, just blindly believing the results of the system without putting more thought into it and I think that is the paradox of this system. People want correct information and we want it to be easy to find out if something is correct or not, but the fact of the matter is that it’s just not easy to find out if something is true or not. A system like this would be great when it worked and told people the truth, but it would make the problem worse when it came to the wrong conclusion and then made more people more confident in their wrong answer. No matter how good a system is, it will still fail. Even the best journalists in the world, writing for the most prestigious newspapers in the world, will get things wrong. And, a system like this one will get things wrong even more often. The fact of the matter is that people should always be skeptical and they should always do research before believing that something is true or not, because no easy answer like this can ever be 100% right and if it can’t be 100% right, we shouldn’t trick ourselves into trusting it any more than we should. This is a powerful tool, but we should not rely on it or anything like it.

Questions

  1. Should we even try to make systems like this if they will be wrong some of the time?
  2. How can we make sure that people don’t over rely on systems like this? Can we still use them without only using them?
  3. What’s the best way to check facts? How do you check your facts?

Read More

04/15/2020 – Dylan Finch – What’s at Stake: Characterizing Risk Perceptions of Emerging Technologies

Dylan Finch 2 Comments

Word count: 553

Summary of the Reading

This paper presents a review of expert and non-expert feelings toward risks with emerging technologies. The paper used a risk survey that was previously used to assess perceptions of risk. This survey was sent out to experts, in the form of people with careers related to technology, and non-experts, in the form of workers on MTurk. While MTurk workers might be slightly more tech-savvy than average, they also tend to be less educated. 

The results showed that experts tended to think more things were more risky. The non-experts tended to downplay the risks of many activities much more than the experts. The results also showed that more voluntary risks were seen as less risky than other forms of risk. It seems like people perceive more risk when they have less control. It also showed that both experts and non-experts saw many emerging technologies as non voluntary, even though these technologies usually get consent from users for everything.

Reflections and Connections

I think that this paper is more important than ever, and it will only continue to get more important as time goes on. In our modern world, more and more of the things we interact with everyday are data driven technologies that weld extreme power, both to help us do things better and for bad actors to hurt innocent people. 

I also think that the paper’s conclusions match up with what I expected. Many new technologies are abstract and the inner workings of them are never seen. They are also much harder to understand for laypersons than the technology of decades past. In the past, you could see that your money was secure in a vault, you could see that you had a big lock on you bike and that it would be hard to steal it, you would know that the physical laws of nature make it hard for other people to steal your stuff, because you had a general idea of how hard it was to break your security measures and because you could see and feel the things you had to protect yourself. Now, things are much different. You have no way of knowing what is protecting your money at the bank. You have no way of knowing, much less understanding the security algorithms that companies use to keep your data safe. Maybe they’re good, maybe they’re not, but you probably won’t know until someone hacks in. The digital world also disregards many of the limits that we experienced in the past and in real life. In real life, it is impossible for someone in India to rob me, without going through a lot of hassle. But, an online hacker can break into bank accounts all across the world and be gone without a trace. This new world of risk is just so hard to understand because we aren’t used to it and because it looks so different to the risks we experience in real life.

Questions

  1. How can we better educate people on the risks of the online world?
  2. How can we better connect abstract online security vulnerabilities to real world, easy to understand vulnerabilities?
  3. Should companies need to be more transparent about security risks to their customers?

Read More

04/08/2020 – Dylan Finch – Agency plus automation: Designing artificial intelligence into interactive systems

Dylan Finch 1 Comment

Word count: 667

Summary of the Reading

This paper focuses on the problem of how humans interact with AI systems. The paper starts with a discussion of automation and how fully automated systems are a long way off and are currently not the best way to do things. We do not have the capabilities to make fully automated systems now, so we should not be trying to. Instead, we should make it easier for humans and machines to interact and work together.

The paper then describes three systems that try to use these principles to make systems that have humans and machines working together. All of these systems give the human the most power. They always have the final say on what to do. The machine will give suggestions, but the humans decide whether or not to accept these ideas. The systems include a tool to help with data analytics, a tool to help with data visualization, and a tool to help with natural language processing.

Reflections and Connections

This paper starts with a heavy dose of skepticism about automation. I think that many people are too optimistic about automation. This class has shown me how people are needed to make these “automated” systems work.Crowd workers often fill in the gaps for systems which pretend to be fully automated, but are not actually fully automated. Rather than pretend that people aren’t needed, we should embrace it and build tools to help the people who make these systems possible. We should be working to help people, rather than replace them. It will take a long time to fully replace many human jobs. We should build tools for the present, not the future.

The paper also argues that a good AI should be easily accessed and easy to dismiss. I completely agree. If AI tools are going to be commonplace, we need easy ways to get information from them and dismiss them when we don’t need them. In this way, they are much like any other tool. For example, suggesting software should give suggestions, but should get out of the way when you don’t like any of them. 

This paper brings up the idea that users prefer to be in control when using many systems. I think this is something many researchers miss. People like to have control. I often do a little more work so that I don’t have to use suggested actions, so that I know exactly what is being done. Or, I will go back through and try to check the automated work to make sure I did it correct. For example, I would much rather make my own graph in Excel than use the suggested ones. 

Questions

  1. Is the public too optimistic about the state of automation? What about different fields of research? Should we focus less on fully automating systems and instead on improving the systems we have with small doses of automation?
  2. Do companies like Tesla need to be held responsible for misleading consumers about the abilities of their AI technologies? How can we, as computer scientists, help people to better understand the limitations of AI technologies?
  3. When you use software that has options for automation, are you ever skeptical? Do you ever do things yourself because you think the system might not do it right? When we are eventually trying to transition to fully automated systems, how can we get people to trust the systems?
  4. The natural language translation experiment showed that the automated system made the translators produce more homogenous translations. Is this a good thing? When would having more similar results be good? When would it be bad?
  5. What are some other possible applications for this type of system, where an AI suggests actions and a user decides whether to accept those actions or not? What are some applications where this kind of system might not work? What limitations cause it not to work there?

Read More

04/08/2020 – Dylan Finch – CrowdScape: interactively visualizing user behavior and output

Dylan Finch 1 Comment

Word count: 561

Summary of the Reading

This paper describes a system for dealing with crowdsourced work that needs to be evaluated by humans. For complex or especially creative tasks, it can be hard to evaluate the work of crowd workers, because there is some much of it and most of it needs to be evaluated by another human. If the evaluation takes too long, you lose the benefits of using crowd workers in the first place. 

To help with these issues, the researchers have developed a system that helps an evaluator deal with all of the data from the tasks. The system leans heavily on the data visualization. The interface for the system shows the user a slathering of different metrics about the crowd work and the workers to help the user determine quality. Specifically, the system helps the user to see information about worker output and behavior at the same time, giving a better indication of performance.

Reflections and Connections

I think that this paper tries to tackle a very important issue of crowd work: evaluation. Evaluation of tasks is not an easy process and for complicated tasks, it can be extremely difficult and, worst of all, hard to automate. If you need humans to review and evaluate work done by crowd workers, and it takes the reviewer a non-insignificant amount of time, then you are not really saving any effort by using the crowd in the first place. 

This paper is so important because it provides a way to make it easier for people to evaluate work done by crowd workers, making the use of crowd workers much more efficient, on the whole. If evaluation can be done more quickly, the data from the tasks can be used more quickly, and the whole process of using crowd workers has been made much faster than it was before. 

I also think this paper is important because it gives reviewers a new way to look at the work done by crowds: it shows the reviewer both worker output and worker behavior. This would make it much easier for reviewers to decide if a task was completed satisfactorily or not. If we can see that a worker did not spend a lot of time on a task and that their work was significantly different from other workers assigned to the same task, we may be able to tell that that worker did a bad job, and their data should be thrown out.

From the pictures of the system, it does look a little complicated and I would be concerned that the system is hard to use or overly complicated. Having a system that saves time, but that takes a long time to fully understand can be just as bad as not having the time saving system. So, I do think that some effort should be used to make the system look less intimidating and easier to use. 

Questions

  1. What are some other possible applications for this type of software, besides the extra  one mentioned in the paper?
  2. Do you think there is any way we could fully automate the evaluation of the creative and complex tasks focused on in this research?
  3. Do you think that the large amount of information given to users of the system might overwhelm them?

Read More

03/25/2020 – Dylan Finch – Evaluating Visual Conversational Agents via Cooperative Human-AI Games

Dylan Finch 1 Comment

Word count: 568

Summary of the Reading

This paper makes many contributions to the field of human and AI interaction. It focuses on presenting a new way to evaluate AI agents. Most evaluations of AI systems are done in isolation, with no human input. One AI system interacts with another AI system and their combined interaction forms the basis of the evaluation for the AI systems. This research presents a new way to evaluate AI systems: bringing humans into the loop and getting them to replace one of the AI systems to better evaluate how AIs work within a more real world scenario: one where humans are present. This paper finds that these two evaluation methods can produce different results. Specifically, when comparing the AI systems, the one that performed worse when evaluated with another AI system actually performs better when evaluated by a human. This raises important questions about the way we test AI systems and suggests that testing should be more human focused.

Reflections and Connections

I think that this paper highlights an important issue that I had never really thought about. Whenever we build any kind of new tool or new system, it must be tested. And, this testing process is extremely important in deciding whether or not the system works. The way that we design tests is just as important as the way that we design the system in the first place. If we design a great system, but design a bad test and then the test says that the system doesn’t work, we have lost a good idea because of a bad test. I think this paper will make me think more critically about how I design my tests in the future. I will put more care into them and make sure that they are well designed and will give me the results that I am looking for. 

When these ideas are applied to AI, I think that they get even more interesting. AI systems can be extremely hard to test and oftentimes, it is much easier to design another automated system, whether that be another AI system or just an automated script, to test an AI system, rather than getting real people to test it. It is just much easier to use machines than it is to use humans. Machines don’t require IRB approval, machines are available 24/7, and machines provide consistent results. However, when we are designing AI systems and especially when we are designing AI systems that are made to be used by humans, it is important that we test them with humans. We cannot truly know if a system designed to work with humans actually works until we test it with humans. 

I hope that this paper will push more research teams to use humans in their testing. Especially with new tools like MTurk, it is easier and cheaper than ever to get humans to test your systems. 

Questions

  1. What other kinds of systems should use humans in testing, rather than bots or automated systems?
  2. Should all AI systems be tested with humans? When is it ok to test with machines?
  3. Should we be more skeptical of past results, considering that this paper showed that an evaluation conducted with machines actually produced a wrong result (the wrong ALICE bot was chosen as better by machines)?

Read More

03/25/2020 – Dylan Finch – “Like Having a Really bad PA”: The Gulf between User Expectation and Experience of Conversational Agents

Dylan Finch 1 Comment

Word count: 575

Summary of the Reading

This paper focuses on cataloguing issues that users have with conversational agents and how user expectations of what conversational agents can do differ dramatically from what these systems can actually do. This is where the main issue arises: the difference in perceived and actual usefulness. Conversational agents are the virtual assistants that most of us have on our smartphones. They can do many things, but they will often have trouble with more complicated tasks and they may not be extremely accurate. Many participants in the study said that they would not use their conversation agents to do complicated tasks that required precision, like writing long emails. Other uses assumed that the conversation agents could do something, like book movie tickets, but the system could not accomplish the task for the first few times it was tried. This made the user less likely to try to use those features in the future. This paper lists more of these types of issues and tries to present some solutions to them.

Reflections and Connections

I think that this paper highlights a big problem with conversation agents. It can sometimes be very hard to know what conversational agents can and cannot do. Oftentimes, there is no explicit manual that lists all of the kinds of questions that you can ask or that tells you the limits of what the agent can do. This is unfortunate because being upfront with users is the best way to set expectations to a reasonable level. Conversational agents should do a better job of working expectations into their setups or their instructions.

Companies should also do a better job of telling consumers what these agents can and cannot do. Companies like Apple and Google, some of the companies highlighted in the paper, often build their agents up to be capable of anything. Apple tries to sell you on Siri by promising that it can basically do anything. Apple encourages you to use Siri for as many tasks as you can and advertaties this. But, oftentimes, Siri can’t do everything they imply it can. Or, if Siri can do it, she does it poorly. This compounds the problem even more because it sets user expectations extremely high. Then, users will try to actually use the agents, find out that they can’t do as many things as was advertised, and give up on the system altogether. Companies could do a lot to help solve this problem by just being honest with consumers and saying that there are certain things their agents can do and certain things their agents cannot do. 

This is a real problem for people who use these kinds of technologies. When people do not know what kinds of questions the agents can actually answer, they may be more scared to ask any questions, severely limiting the usefulness of the agent. It would vastly improve the user experience if we could solve this issue and make people have more accurate expectations for what conversational agents can do.

Questions

  1. How can companies better set expectations for their conversational agents?
  2. Does anyone else have a role to play in educating people on the capabilities of conversation agents besides companies?
  3. Do we, as computer scientists, have a role to play in educating people about the capabilities of conversational agents?

Read More