Summary

Hope’s paper “Accelerating Innovation Through Analogy Mining” studies how to boost knowledge discovery through searching for analogies in massive and unstructured real-world datasets. This research is motivated by the availability of large idea repositories which can be used as databases to search for analogous problems. However, it is very challenging to find useful analogies among the massive and noisy real-world repositories. Manual and automated methods have their own advantages and disadvantages: hand-created databases have a high relational structure which is central to analogy search but expensive to obtain; naive machine learning or information retrieval can be easily scaled to large datasets with similarity metrics but fail to incorporate structural similarity. To address the challenges, the researchers explore the potential of learning “problem schemas,” which are simpler structural representations that specify the purpose and mechanism of a product. Their proposed approach leverages the power of crowdsourcing and recurrent neural networks to extract purpose and mechanism vector representations from product descriptions. The experimental results indicate that the learned vectors can facilitate the search of analogies with higher precision and recall than traditional information retrieval methods. 

Reflection

This paper introduces an innovative approach for analogy search, the task of which is very similar to the “SOLVENT” paper we discussed last week. The “weaker structural representations” idea is very interesting and it allows more flexibility in automatic algorithm design compared with relying on fully structured analogical reasoning. My impression is that the human component in the model design is comparatively weaker than that of the approaches presented in other readings we discussed before. Crowd work in this paper is leveraged as an approach to generate training data and evaluate the experimental results. I have been thinking about if there is any place that has the potential to incorporate human interaction in the model design. As we know, recurrent neural networks have certain limitations. The first weakness is variable length, which means the RNN models cannot handle long sequence data, and this largely constrains the usage scenarios. The second weakness is the sliding window, which ignores the continuity between the sliced subsequences, which is more close to the surface but not as deep as the paper claims. I am wondering if there is a possibility to leverage human interaction to overcome the shortcoming of the model itself.

The performance of machine learning models is highly driven by the quality of the training data. With the crowdsourced product purpose and mechanism annotations, I feel there is a need to incorporate some quality control components in the proposed framework. A few papers we discussed before touching upon this point. Also, though very powerful in numerous complex tasks, deep learning models are usually criticized due to its lack of interpretability. Although the RNN performance reported in the paper in regards to recall and precision is better than that of traditional information retrieval methods. However, those similarity-based methods have their own merits as their mechanism and decision boundaries are more transparent so it would be possible to detect where the problem is and why the results are not desirable. In this case, there is a trade-off between the model performance (accuracy, precision, and recall) and interpretability, it is worthy of thinking about which one to choose over the other.

Discussion

I think the following questions are worthy of further discussion.

• What do you think can incorporate more human components in the model design?
• Compared with last week’s approach SOLVENT, which one you think works better? Why?
• What are some other potential applications for this system outside of knowledge discovery?
• Do you think the recurrent neural network method is better than traditional similarity-based methods such as TF-IDF in the analogy search task and other NLP tasks? Why or why not?

Read More

SUMMARY

In this paper, the authors attempt to facilitate the process of finding analogies with a view to boost creative innovations by exploring the value that can be added by incorporating weak structural representations. They leverage the vast body of online information available (for the purpose of this study, product descriptions from Quirky.com). They generate microtasks for crowdworkers to perform that were designed to label the ‘purpose’ and ‘mechanism’ parts of a product description. The authors use GloVe word vectors to represent their purpose and mechanism words and use a BiRNN to learn the purpose and mechanism. In order to collect analogies, they use AMT crowd workers to find analogies for 8000 product descriptions. In the evaluation stage, the authors attempt to weigh the usefulness of their algorithm by having participants redesign a product. 38 AMT workers were recruited for the same and the task was to design a cell phone charger case. 5 graduate students were recruited to evaluate the ideas generated by the workers. Based on a predefined criterion of ‘good’ ideas, 208 were produced out of 749 total ideas (with 2 judges rating it as good) and 154 were produced out of 749 total ideas (with 3 judges rating it as good). In both cases, the analogy approach proposed by the authors out-performed the TF-IDF baseline model and random model.

REFLECTION

I found the motivation of this study to be very good – especially based on ‘bacteria-slot machine’ analogy example highlighted in the introduction of the paper. I agree that given the vast amount of data available, having such a system that would accelerate the process of finding analogies could very well aid in quicker innovation and discovery.

I like that the authors chose to present their approach by using product descriptions. I also like the use of ‘purpose’ and ‘mechanism’ annotations and feel that given the more general domain of this study, the quality of annotations by the crowdworkers would be better than in the case of the paper on ‘SOLVENT: A Mixed Initiative System for Finding Analogies between Research Papers’.

I also liked that the authors presented the results given by using the TF-IDF baseline as it indicated the findings that would have been generated by near-domain results. I felt that it was good that the authors added a criterion to judge the feasibility of an idea, one that could be implemented using existing technologies.

Additionally, while I found that the study and methods proposed by this paper were good, I did not like the organization of the paper.

QUESTIONS

1. How would you rate the design of the interface used to collect ‘purpose’ and ‘mechanism’ annotations? What changes might you propose to make this better?
2. The authors do not mention the details or experiences of the AMT workers. How much would workers prior experience influence their ability to find ideas using this approach? Can this approach aid more experienced people working with product innovations?
3. The authors of SOLVENT leverage the mixed-initiative system proposed by this paper to find analogies between research papers. Which are the domains where this approach would fail to a great extent (even if modifications were to be made)?

Read More

Paper: Tom Hope, Joel Chan, Aniket Kittur, and Dafna Shahaf. 2017. Accelerating Innovation Through Analogy Mining. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD ’17), 235–243. https://doi.org/10.1145/3097983.3098038

Summary: This paper talks about the challenge of mining analogies from large, real-world repositories, such as patent databases. Such databases pose challenges because they are highly relational but sparse in nature. This s a reason why machine learning approaches do not fare well when applied to these types of databases, especially since they cannot formulate a patter of the underlying structure, which is important for analogy mining. The corpora are also expensive to build, store, and update, while automation cannot be easily applied. The authors overcome these limitations by leveraging the creativity of the crowd and affordable computational capabilities of RNNs. The approach is a structured purpose-mechanism schema for identifying analogies between two research papers. Finally, the authors evaluate crowd worker performance by asking graduate students to annotate the ideas generated around three main ideas: quality, novelty, and feasibility. They find that their approach increased feasibility among the participants in the study.

Reflection:
Overall, I really liked the paper in how it attempts to solve a hard problem by using a scalable approach: crowds and RNNs, and tests it on a real-world dataset. I also liked how the paper defines similarity between different ideas (i.e. analogies) based on purpose and the mechanisms through which products work. Further, the paper suggests more complex metrics for research papers. This raises the question: how much more difficult is it to mine analogies for complex/more abstract ideas, compared to simple ideas? Perhaps structured labels could help in that regards.

The approach itself is commendable since it is a great example of a mixed-initiative user interface that combines the creativity of the crowd and the affordable computation of RNNs. Further, this approach does not needlessly waste human computation. The authors also completed a thorough evaluation of the machine intelligence portion.

Second, I appreciate the approach taken towards making something subjective—in this case, creativity—into something more objective, by breaking it down into different rate-able metrics.

Finally, the idea of using randomly generate analogies to “spark creativity” and the results of that show that creativity really does need diverse ideas. I wonder why this may be, and how to introduce such randomness into real-world work practice.

Questions:
1. How scalable do you think the system is? What other limitations does it have?
2. Can this approach be used to generate analogies in other fields? What would be different?
3. Do you think creativity is subject? Can it be made into something objective?

Read More

Summary:
“Accelerating Innovation Through Analogy Mining” by Hope et al. talks about the problem of analogy mining from messy and chaotic real-world datasets. Hand created databases have high relational structures but are sparse in nature. On the other hand, machine learning and information retrieval techniques can be used but they lack the understanding of the underlying structure which is crucial to analogy related tasks. The authors leverage the strengths of both crowdsourcing and machine learning techniques to learn analogies from these real-world datasets. They make use of the creativity of the crowds with the cheap computing power of recurrent neural networks. The authors extract meaningful vector representations from product descriptions. They observe that this methodology achieves greater precision and recall than the traditional information-retrieval methods. The authors also demonstrate that the models significantly helped in generating more creative ideas compared to analogies retrieved by traditional methods.

Reflections:
This is a really interesting paper that talks about a scalable approach to finding analogies in large, real-world, messy datasets. The authors use a bi-directional Recurrent Neural Network (RNN) with Gated Recurrent Units (GRU) to learn the purpose and mechanism vectors for product descriptions. However, since the paper came out there have been great advances in the field of natural language processing tasks because of BERT: Bidirectional Encoder Representations from Transformers. BERT has achieved a state of the art results for many natural language tasks like question answering, natural language understanding, search, and retrieval, etc.. I’m curious to know how BERT would affect the results of this current system. Would we still need crowd workers for analogy detection or would using BERT alone for analogy computation suffice? One of the limitations of RNN is that it is directional, i.e., it can either read from right to left or left to right, or both. BERT is essentially non-directional, i.e, it takes all the words as inputs at once and hence, can compute the complex non-linear relationships between them. This would definitely prove helpful for detecting analogies. the approach taken by the authors by using TF-IDF did result in diversity but did not take into account the relevance. Also, the purpose and mechanism vector by the authors did not distinguish between high and low-level features. These learned vectors also did not take into account the intra-dependencies between different purposes and mechanisms or the inter-dependency between various purposes and mechanisms. It would be interesting to observe how these dependencies could be encoded and whether they would benefit the final tasks of analogy computation. Another aspect that can be looked into is the trade-off between generating useful vector and its relation with the creativity of the crowd-workers. Does creativity increase, decrease, or remain the same?

Questions:
1. Which another creative task can benefit from human-AI interaction?
2. Why is the task of analogy computation important?
3. How are you incorporating and leveraging the strength of crowd-workers and machine learning in your project?

Read More