Summary
Hope’s paper “Accelerating Innovation Through Analogy Mining” studies how to boost knowledge discovery through searching for analogies in massive and unstructured real-world datasets. This research is motivated by the availability of large idea repositories which can be used as databases to search for analogous problems. However, it is very challenging to find useful analogies among the massive and noisy real-world repositories. Manual and automated methods have their own advantages and disadvantages: hand-created databases have a high relational structure which is central to analogy search but expensive to obtain; naive machine learning or information retrieval can be easily scaled to large datasets with similarity metrics but fail to incorporate structural similarity. To address the challenges, the researchers explore the potential of learning “problem schemas,” which are simpler structural representations that specify the purpose and mechanism of a product. Their proposed approach leverages the power of crowdsourcing and recurrent neural networks to extract purpose and mechanism vector representations from product descriptions. The experimental results indicate that the learned vectors can facilitate the search of analogies with higher precision and recall than traditional information retrieval methods.
Reflection
This paper introduces an innovative approach for analogy search, the task of which is very similar to the “SOLVENT” paper we discussed last week. The “weaker structural representations” idea is very interesting and it allows more flexibility in automatic algorithm design compared with relying on fully structured analogical reasoning. My impression is that the human component in the model design is comparatively weaker than that of the approaches presented in other readings we discussed before. Crowd work in this paper is leveraged as an approach to generate training data and evaluate the experimental results. I have been thinking about if there is any place that has the potential to incorporate human interaction in the model design. As we know, recurrent neural networks have certain limitations. The first weakness is variable length, which means the RNN models cannot handle long sequence data, and this largely constrains the usage scenarios. The second weakness is the sliding window, which ignores the continuity between the sliced subsequences, which is more close to the surface but not as deep as the paper claims. I am wondering if there is a possibility to leverage human interaction to overcome the shortcoming of the model itself.
The performance of machine learning models is highly driven by the quality of the training data. With the crowdsourced product purpose and mechanism annotations, I feel there is a need to incorporate some quality control components in the proposed framework. A few papers we discussed before touching upon this point. Also, though very powerful in numerous complex tasks, deep learning models are usually criticized due to its lack of interpretability. Although the RNN performance reported in the paper in regards to recall and precision is better than that of traditional information retrieval methods. However, those similarity-based methods have their own merits as their mechanism and decision boundaries are more transparent so it would be possible to detect where the problem is and why the results are not desirable. In this case, there is a trade-off between the model performance (accuracy, precision, and recall) and interpretability, it is worthy of thinking about which one to choose over the other.
Discussion
I think the following questions are worthy of further discussion.
- What do you think can incorporate more human components in the model design?
- Compared with last week’s approach SOLVENT, which one you think works better? Why?
- What are some other potential applications for this system outside of knowledge discovery?
- Do you think the recurrent neural network method is better than traditional similarity-based methods such as TF-IDF in the analogy search task and other NLP tasks? Why or why not?
I agree with your points that using an RNN, though it provides better results, really suffers from not being understandable. But if it’s results that you want, you can’t really argue with the RNN approach in this case. Hopefully we can get to a point where RNNs are as easy to explain as TF-IDF so that we can have the best of both worlds.