4/29/2020 – Nurendra Choudhary – Accelerating Innovation Through Analogy Mining

April 28, 2020April 28, 2020 Nurendra Choudhary Leave a comment

Summary

In this paper, the authors argue the need for an automated tool for finding analogies from large research repositories such as US patent databases. Previous approaches in the area include manually constructing large structured corpora and automated approaches that are able to find semantically relevant research but cannot identify proper structure in the documents. The manual corpora are expensive to construct and maintain, whereas, automated detection is inefficient due to lack of structure identification.

The authors propose an architecture that defines a structured purpose-mechanism schema for the analogy identification between two research papers. The purpose and mechanism are identified by crowd-workers and a word vectorization is utilized to represent the different sections as vectors. The similarity metric is cosine similarity between the query vectors. The query vectors for the experiments are purpose, mechanism and concatenation of purpose and mechanism. The authors utilize the Precision@K metric to evaluate and compare to conclude the efficiency of mechanism only and concat purpose-mechanism queries over other types.

Reflection

The paper is very similar to the SOLVENT discussed in the previous class. I believe they were both developed by the same research group and also share the same authors. I believe SOLVENT solves a range of problems in this paper. For example, the problem that purpose-mechanism cannot be generalized to all research fields and there is a need to add additional fields to make it work better for a wider range of fields.

The baselines do not utilize the entire paper. I do not think it is fair to compare abstracts of different domains to find analogies. The abstract do not always speak about the problem or solution in necessary depth. According to me, we should add more sections such as Introduction, Methodology and Conclusion. I am not sure if they would perform better but I would like to see these metrics reported. Also the diversity of fields used in the experiments is limited to engineering backgrounds. I think this should be expanded to include other fields such as medicine, business and humanities (a lot of early scientists were philosophers :P).

Questions

What problems in this paper does SOLVENT solve? Do you think the improvement in performance is worth the additional memory utilized?
How do you think this framework will help in inspiring new work? Can we put our ideas into a purpose schema mechanism to get a set of relevant analogies that may inspire further research?
The authors only utilize abstract to find analogies. Do you think this is a good enough baseline? Should we utilize the entire paper as a baseline? What are the advantages and disadvantages of such an approach? Is it not more fair?
Currently, can we learn purpose mechanism schemas for different fields independently and map between them? Is there a limit to the amount of variation that limits this framework? For example, is it fair to use medical documents’ abstracts and compare them to abstracts to CS papers given the stark amount of difference between them?

Word Count: 509

4/29/2020 – Nurendra Choudhary – DiscoverySpace: Suggesting Actions in Complex Software

April 28, 2020April 28, 2020 Nurendra Choudhary Leave a comment

Summary

In this paper, the authors introduce Discovery Space, an extension to Adobe Photoshop that suggests high-level macro actions based on visual features. Complex platforms such as Photoshop are great tools to aid creativity. However, their features are rather complex for beginners making for a steep learning curve. DiscoverySpace utilizes one-click actions suggested in the online community and makes these macro actions available to new users thus softening their introduction to the software.

For the experiment, the authors maintain two independent control groups. One group has access to the DiscoverySpace panel in Photoshop and the other only has the basic tool. Experiments show that DiscoverySpace helps beginners by suggesting initial macro actions. Subjects in the no tool group were frustrated with the tool’s complex features producing worse results than the subjects in DiscoverySpace group. Also, the authors suggest that some steps in the process can be replaced by advances in AI algorithms in the future which will lead to faster processes.

Reflection

The paper is really interesting in its approach to reduce the system’s complexity by integrating macro action suggestions. The framework is very generalizable and can work in a multiple number of complex softwares such as Excel sheets (to help with common macro functions), Powerpoint presentations (to apply popular transitions or slide formats) and AI frameworks (pre-building popular networks). Another important aspect is that such technologies are already being applied in several places. Voice assistants have specific suggestions to introduce users to common tasks such as setting up alarms, checking weather, etc.

However, the study group is relatively very small. I do not understand the reason for this. The tasks could be put into an MTurk type format and given to several users. Given the length of the task (~30 min), the authors could potentially use train and work platforms such as Upwork too. Hence, I believe the conclusions of the paper are very specific to the subjects. Also, the authors suggest the potential of integrating AI systems to their framework. I think it would help if more examples were given for such integrations.

Also, utilizing DiscoverySpace-like mechanisms draws in more users. This provides a monetary-incentive to businesses to invest in more such ideas. One example can be the paper-clip assistant in initial versions of Windows that introduced users to the operating system.

Questions

I believe machine learning frameworks like tensorflow and pytorch have examples to introduce themselves to beginners. They could benefit from a DiscoverySpace-like action suggestion mechanism. Can you give some examples of softwares in your research area that could benefit from such frameworks?
I believe the limited number of subjects is a huge drawback to trust in the conclusions of the paper. Can you provide some suggestions on how the experiments could be scaled to utilize more workers at a limited cost?
The authors provide the example of using advances in image analysis to replace a part of DiscoverySpace. Can you think of some other frameworks that have replaceable parts? Should we develop more architectures based on this idea that they can be replaced by advances in AI?
Give some examples of systems that already utilize DiscoverySpace-like framework to draw in more users?

Word Count: 538

04/29/2020 – Bipasha Banerjee – Accelerating Innovation Through Analogy Mining

April 28, 2020April 28, 2020 bipashab Leave a comment

Summary

The paper by Hope et al. talks about analogy mining from texts, primarily data that is unstructured. They have used a product description dataset from Quirky.com, which is described to be a product innovation website, to find products that are similar. To be specific, they have used “purpose” and the “mechanism” of products to find analogies between items. They have also considered the traditional similarity metrics and techniques, namely the TF-IDF, LSA, GloVe, and LDA, to compare the proposed approach. Amazon Mechanical Turk crowd workers were used to create the training dataset. A recurrent neural network was then used to learn the representations of purpose and mechanism from the human-annotated training data. It is mentioned in the paper that they wanted to see if their approach enhanced creativity in idea generation. They tried to measure this by using graduate students to judge the idea generated on three main areas, novelty, quality, and feasibility. It was concluded that there was an increase of creativity among the participants of the study.

Reflection

The paper is an interesting read on the topic of finding analogies in texts. I really found it interesting how they defined similarities based on the purpose and the mechanisms in which the products worked. I know that the authors mentioned that since the dataset was about product description, the purpose-mechanism structure worked in finding analogies. However, they suggested some complex or hierarchial levels for more complex datasets like scientific papers. The only concern I had with this comment was, wouldn’t increasing the complexity of the training data further complicate the process of finding analogies? Instead of hierarchical level, I think it is best to add other labels to the text to find similarities. I think what I am suggesting is similar to what was done in the paper by Chang et al. [1], where background and findings were also included along with the labels included here.

The paper is a good groundwork on the work of finding similarities while using crowd workers to create the training data. This methodology, in my opinion, truly forms a mixed-initiative structure. Here, the authors did extensive evaluation and experimentation on the AI side of things. I really liked the way they compared against the traditional information retrieval mechanisms to find analogies.

I liked that the paper also aimed to find if the creativity was increased. My only concern was “creativity” although defined is subjective. They said that they used graduate students but did not mention their background. Hence, a graduate student with a relatively creative background, say a minor in a creative field may view things differently.

In conclusion, I found this research to be strong as it included verification and validation of the results from all angles and not only the AI or the human side.

Questions

Are you using a similarity metric in your course project? If yes, what are the algorithms you are using? ( In our project, we are not using any similarity metric, but I have used all the traditional metrics mentioned in the paper in my research work before).
Other than scientific data, what other kinds of complex datasets would need additional labels or hierarchical labeling?
Do you agree with the authors’ way of finding if the study had enhanced creativity?

References

Chan, Joel, et al. “Solvent: A mixed initiative system for finding analogies between research papers.” Proceedings of the ACM on Human-Computer Interaction 2.CSCW (2018): 1-21.

04/29/2020 – Akshita Jha – DiscoverySpace: Suggesting Actions in Complex Software

April 28, 2020April 28, 2020 Akshita Jha 2 Comments

Summary:
“DiscoverySpace: Suggesting Actions in Complex Software” by Fraser et. al. talks about complex software and ways novice users can navigate these complex systems. “DiscoverySpace is a prototype extension panel for Adobe Photoshop that suggests task-level action macros to apply to photographs based on visual features.” The authors find out that the actions suggested by DiscoverSpace help novices maintain confidence, accomplish tasks, and discover new features. The work highlights how user generated content can be leveraged by interface designers to help new and inexperienced navigate a complex system. There are several problems that a beginner might face when trying to access new complex systems: (i) the novice user might not be familiar with the technical jargon used by the software, (ii) the online tutorials might be difficult to follow and assume a certain amount of background knowledge, (iii) there can be several ways to accomplish the same task and the user might get overwhelmed and confused when getting to know about them. The paper presents DiscoverySpace which is a prototype action suggestion software to help beginners get started without feeling overwhelmed.

Reflections:
This is an interesting paper as it talks about the methodology that can be adopted by software designers to build a system that aids novice users, instead of overwhelming them. This reminds me of the popular ‘cold start’ problem in computational modeling. The term essentially refers to the problem when computers do not have enough information to model the desired human behavior. This is due to the lack of initial user interactions. The authors try to mitigate this problem in DiscoverySpace by conducting a survey to help identify and narrow down the kind of help novices need. It was an interesting find that participants who used the web to look for help achieved their results less often. I would have expected it to be the other way round. The authors suggest that the participants failed to find the best way to accomplish the task and Google does not always help find the best results. One of the limitations of the study was that the task was open-ended. If the task were more directed, the results would have led to better findings. Also, self-reporting expertise on a task might not be the most reliable way to assess the user as a novice or an expert. Another thing to note here is that all the participants had some kind of domain knowledge, either through the basic principles of photography or through simpler photo-editing software. It would be interesting to see how the results pan out for users from a different field. I was also wondering if the design goals presented by the authors are too generic. This can be a good thing as it allows other systems to take these goals into consideration but it might also prove harmful as it might limit the capability of DiscoverySpace by not taking into account the specific design goals that this particular system might benefit from.

Questions:
1. Did you agree with the methodology of the paper?
2. Which design goal do you think would apply to you?
3. Can you think of any other software that is complex enough to require design interventions?
5. How are you incorporating creativity into your project?

04/29/20 – Jooyoung Whang – Accelerating Innovation Through Analogy Mining

April 28, 2020April 28, 2020 Jooyoung Whang 1 Comment

This paper sought to find analogies in big messy real-world natural language data by applying the structure of purpose and mechanism. The authors created binarized data for each purpose and mechanism of a document by setting some words to 1 if one of purpose or mechanism could be represented by the word and 0 if not. Then the authors could evaluate distances between each of these vectors to let users generate creative ideas that have a similar purpose but the different mechanisms. The authors utilized Mturk to generate training sets as well as for evaluation. They measured creativity in terms of novelty, quality, and feasibility. The authors report significantly improved performance than the baseline of plain TF-IDF or random.

This paper appeared to be similar to the SOLVENT paper from last week, except that this one worked with product descriptions, only used the purpose-mechanism structure, and evaluated based on creativity. This was actually a more interesting read for me because it was more relevant to my project. I was especially inspired by the authors’ method of evaluating creativity. I think I may be able to do something similar for my project.

I took special note to the amount of compensation the authors paid to Mturk workers and tried to reverse-calculate the time they allotted for each worker. The authors paid $1.5 for a task that required redesigning an existing product, using 12 near-purpose far-mechanism solutions found by the authors’ approach. This must be a lot of reading (assuming 150 words per solution, that’s 1800 words of reading leaving out the instructions!) and creative thinking. Based on the amount paid, the authors expected about 10 minutes for the participants to finish their work. I am unsure if this amount was appropriate, but based on the authors’ results, it seems successful. It was difficult for me to gauge how much I should pay for my project’s tasks, but I think this study gave me a good anchor point. My biggest dilemma was balancing out the number of creative references provided by my workers versus the quality (more time needed to generate, thus more expensive) for each of the references.

These are the questions that I had while reading the paper:

1. One of the reasons why the SOLVENT paper expanded their analogy structure to purpose-background-mechanism-findings was because not all papers had a “mechanism” or a “solution.” (i.e. some papers were about simple findings of a problem or domain.) Do you think the same applies to this study?

2. Do you think the amount of compensation the authors paid was appropriate? If not, how much do you think would have been appropriate? I would personally really like to read some answers about this question to apply to my project.

3. What other ways could be used to measure “creativity”? The authors did a great job by breaking down creativity into smaller measurable components (although still being qualitative ones) like novelty, quality, and feasibility. Would there be a different method? Would there be more measurable components? Do you think the authors’ method captures the entirety of creativity?

04/29/2020 – Vikram Mohanty – VisiBlends: A Flexible Workflow for Visual Blends

April 28, 2020 Vikram Mohanty Leave a comment

Authors: Lydia B. Chilton, Savvas Petridis, and Maneesh Agrawala.

Summary

This paper discusses the concept of visual blends, which are an advanced design technique to draw attention to a message. This paper presents VisiBlends, a workflow for creating visual blends, by decomposing the process into computational techniques and human micro-tasks. Users can collaboratively create visual blends with steps involving brainstorming, synthesis and iteration. An evaluation of the workflow showed that it improved novices’ ability to create visual blends, and it works well for both decentralized and co-located groups.

Reflection

Stemming from a poor design sense personally, I appreciated how this paper read and what it has to contribute a lot. Creativity is a complex topic, and designing computational tools to support it can get really tricky. This paper was easy to read, and very well supported by figures for helping readers understand not only the concept of Visual Blends, but also the findings. VisiBlends opens up new possibilities of how tools can extend support to other design applications such as web design. (As per my knowledge, there are AI engines for generating web design templates, color schemes, etc. but I am not aware of user studies for these AI solutions)

This paper echoes something that we have read in a lot of papers — decomposing big tasks into smaller, meaningful chunks. However, decomposing creative tasks can become tricky. The steps were simple enough for onboarding novice users and the algorithm was intuitive. This human-AI collaboration seemed a bit unique to me particularly because the success of the whole endeavor also depended on how well the user understood how the algorithm works. This is a stark contrast to the black-box vision of algorithms. Will it become difficult as the algorithm gets more complex to support more complex blends?

All of the findings supported the usefulness of VisiBlend approach. However, I wonder if there’s a possibility of the task or the concept of visual blends (ticking off the checklist of requirements) being complex enough to understand in the first attempt. I am sure the training process, which they stress to be important, was thorough and comprehensive. But, at the end of the day, it boils down to learning from experience. I feel one would understand the requirements of visual blends better through the tool and may face difficulty in the control condition.

I really liked how the participants iterated in different ways to improve upon the initial visual blend. This is a great demonstration of human-machine collaboration, where people use machine suggestions to refine the original parameters and improve the whole process. I am also glad they addressed the issue of gender stereotypes for the “Women in CS” design task as that was something on my mind as well.

Questions

What do you feel about decomposing creative tasks? What are the challenges? Do you think it’s always possible?
Do you think users should always have a good sense of how the algorithm works? When do you think it’s necessary?
What changes are necessary for this to scale up to more complex designs? How would a complex algorithm affect the whole process?

04/29/2020 – Vikram Mohanty – DiscoverySpace: Suggesting Actions in Complex Software

April 28, 2020 Vikram Mohanty 1 Comment

Authors: C. Ailie Fraser, Mira Dontcheva, Holger Winnemöller, Sheryl Ehrlich, and Scott Klemmer.

Summary

This paper proposes DiscoverySpace, an extension panel for Adobe Photoshop, to help onboard new users execute tasks and explore the features of the software. DiscoverySpace suggests task-level action macros to apply to photographs based on visual features. These actions are retrieved from an online user community. The findings from user evaluation showed that it helped novices maintain confidence, accomplish tasks and discover features.

Reflection

This paper addresses an important problem that I often encounter while building tools/interfaces for novice users — how do we efficiently handle the onboarding process? While DiscoverySpace, in its current form, is far from being the ideal tool, it still opens the doors for how we can leverage the strengths of recommender engines, NLP and UI designs to build future onboarding tools for complex softwares.

Something we discussed in the last class – DiscoverySpace also demonstrates how need-finding exercises can be translated into design goals, and in doing so, it increases the likelihood of being useful. I wonder, how this process can be scaled up for a general purpose workflow of designing onboarding tools for any software (or maybe it is not necessary).

In one of my previous reflections, I mentioned about how in-app recommendations for using different features helped users explore more, which was a stark contrast to the notion of filter bubbles. This paper also demonstrated a similar finding, which leads me to believe that maybe in-app feature recommendations are useful for exploring the space when the users, by themselves, cannot explore or unaware of the unknown space. I am hoping to see a future study by the RecSys community, if there isn’t one already, to understand the correlation between tool feature recommendations and the user’s expertise level.

This paper certainly made me think a lot more about general purpose applicability i.e. how can we build a toolkit that can work for any software. I really liked the discussion section of the paper as it discussed the topics that would essentially form the pathway to such a toolkit. Building a corpus of actions is certainly not impossible, considering the number of users for a software. Most plugins and themes are user-generated, and that’s possible because of the low barrier to contribution. Similar pathways and incentives can be created for users to build a repository of actions, which can be easily imported into DiscoverySpace, or whatever the future version is called. The AI engine can also learn when to recommend what kind of actions with increasing data. Considering how big the Open Source Community is, that would be a good starting place to deploy such a toolkit.

Questions

Have you ever had trouble onboarding novice users for your software/tool/interface? What did you wish for, then?
Do you think a general purpose onboarding toolkit can be built basing off the concepts in DiscoverySpace?
The paper mentions about the issue of end-user control in DiscoverySpace. What are the challenges of designing DiscoverySpace if we think about extending the user base to expert users (obviously, the purpose will change)?

04/29/20 – Lulwah AlKulaib-Siangliulue et al., “IdeaHound”

April 28, 2020 Lulwah AlKulaib 1 Comment

Summary

Collaborative-creative online communities where users propose ideas and solutions to existing problems have shown that the suggestions can be simple, repetitive, and mundane. Thus, organizations view these platforms as marketing venues instead of innovation resources. Small groups and individual targeted creativity interventions demonstrated that they can improve the creativity outcomes. The authors propose a system that improves the quality of ideas proposed in large scale collaborative platforms. IdeaHound is an ideation system that integrates the task of deﬁning semantic relationships among ideas into the primary task of idea generation. The proposed system creates a semantic model of the solution space by combining implicit human actions with machine learning. The system uses the semantic model to enable three main idea-enhancing interventions: sampling diverse examples, exploring similar ideas, and providing a visual overview of the emerging solution space. Users could use the system features to produce higher quality ideas. The authors conducted studies and experiments to show the effectiveness of the system and the need that this system fills.

Reflection

This was an interesting paper to read. I have never used an ideation platform and have no background in this area but I learned a lot from this reading. To my understanding, the proposed system would be helpful in crowd lead communities that propose ideas for an issue or a problem. This system seems to be useful in keeping track of similar ideas semantically and grouping them for users when they type their own. This solution would drastically reduce repetitiveness and help users to build on top of others ideas. Also, allowing users to have a visual overview of the solution space is another tool that would help in brainstorming. I think that the visualization is helpful to identify different topics, least explored ideas, and most focused on ideas. Which could help redirect users in different directions based on the cluster visualization.

I think that the interface when looking at the big picture is reasonable and very helpful. Yet when looking at the whiteboard and clusters of ideas up close, I feel like this could have been done better. Even though it is simple, using sticky notes as a resemblance of scrap notes, the page looks crowded and can be overwhelming for some users (at least me). I believe that there should be best practices for organizing ideas and maybe having a different interface would help making that process less messy and easier to understand.

I am conflicted about the paper. I like the idea behind the system but the user interface is just too overwhelming for me. I do not know what would be the best visual item to replace the sticky notes or how that section of the system would be altered but maybe in a future version they would offer choices for different users (sticky notes and maybe a menu with toggle/sorting ability items?).

Discussion

Do you agree with the user interface design for the user workspace?
How would you design this system’s user interface?
Have you ever been part of a crowd based ideation process? What were your thoughts about that experience? Would you use a system similar to the proposed system for your task?
What would be a better way to deal with a massive amount of ideas than sticky notes?

04/29/20 – Lulwah AlKulaib-Fraser et al., “DiscoverySpace”

April 28, 2020April 28, 2020 Lulwah AlKulaib Leave a comment

Summary

As software develops over time the complexity increases with all the newly added features. The acquired complexity over time could be beneficial for experts yet it presents issues for beginner end users. When thinking of developing off the shelf software for end users, developers must consider the different technical backgrounds their end users have and try to have the interface accessible for all potential users. To resolve this difficulty in Adobe Photoshop, the authors present DiscoverySpace, a prototype extension panel that suggests task-level action macros to apply to photographs based on visual features. The extension is meant to help new Adobe Photoshop users by making suggestions once the user starts a task (opens a picture), uses simple human language in search, shows previews of what a suggestion does (before and after), offer faceted browsing to make searching a better experience, and show suggestions that are relevant to the users’ current task which also alerts him to new or unknown possibilities. The authors investigate the effectiveness of the extension by running a study and comparing two groups, one was using the extension, and the other did not. They find that action suggestions might help new users from losing confidence in their abilities, help them accomplish their tasks, and discover new features.

Reflection

As an on-and-off Adobe Photoshop user, I was interested in this paper and this extension. I thought it would be nice to have those suggestions as a reminder when I use the software after months of not using it. Since I am more focused on Adobe Lightroom when it comes to editing photos, it is easy for me to confuse the panels and actions available in both softwares. I was somewhat surprised that the users who had the extension were still answering that they couldn’t figure something out 50% of the time. Even though there was a drop of 30% from users who were not using the extension, it still raises the question: where is the problem? Was it the software? Extension lacks some details? Or was it just the fact that users need time to become familiar with the interface?

I also was puzzled when I saw that the authors used random sampling when it comes to suggesting actions to the user. I feel like editing photos is a process and depending on the photo there are actions that should be taken before others. Maybe using that functionality would be specifically for learning about the interface or the result of each action. Else, I don’t think it was the best functionality to propose.

I don’t know if I agree with the authors with how they measure the performance confidence in their survey. Using technology has always made us feel more confident. I trust a calculator more than doing simple math in my head real quick. I felt that this wasn’t a fair comparison measure.

Discussion

Would you use this extension if you had Adobe Photoshop? Why? Or Why not?
What would you change about this extension? Why?
Can you think of other extensions that you use on a regular basis that are useful in terms of learning about some software or platform?

04/29/2020 – Subil Abraham – Chilton et al., “VisiBlends”

April 27, 2020 Subil Abraham Leave a comment

Visual blending is the idea of taking two objects or concepts that you want to represent and combining them in a way that both concepts are identifiable, while also implying that one concept is applicable to the other. The creation of visual blends is a very creative process. But with Visiblend, the authors have created a system that has greatly streamlined the process of creating visual blends, and even split the process up as microtasks. The process of creating visual blends is split up into tasks of ideation of related concepts, searching of singular representative images with simple iconic shapes for the ideated concepts, and annotate the shapes on the images. The system then takes this information to create different combinations (blends) of the images and returns them to the user for evaluation and further iteration. They conduct three case studies looking at how Visiblend can be used under different situations. They also make note of the limitations that Visiblend can only deal with simple iconic shapes and cannot do more complex stuff like animations.

The visual blended images are some of the most powerful imagery I’ve ever come across. They convey ideas so well. I think that this is a really good project that is streamlining the process of creating these powerful images. I am actually shocked at how simple the steps are (granted, it takes a lot more work to actually make it look good). But still, very surprising. Initially, I felt that the system was very limited because all it was doing was cropping and overlaying one picture on top of the other. How could that be of possible use? But then I realized that the real value was coming from the fact that it is able to perform so many blends automatically with no human effort and demo them all. It’s utility comes from the speed and iteration of the visual blends that we can do through it. It’s also really interesting how the tool allows to visualize really unintuitive combinations (like the McDonald’s + energy example). Where a human doing it would be really limited by their preconceived notions of both those concepts, a machine doesn’t have those blocks and can therefore present any combination of zany ideas that a human can look at go “Oh! That does work!”. So it serves as the perfect tool to come up with ideas because it does not have any inhib
itions.

What kind of workflow would be necessary to do something like this, but for animation and gifs instead of static images?
Do you think this streamlined workflow would impede some creative ideas from being conceptualized because people’s thought processes are trained to think this way?
In your opinion, does Visiblend function better as a centralized collaborative tool (where everyone is in the same room) or as a distributed collaborative tools (i.e. using crowd workers on crowd work platforms)?