Paper: Joel Chan, Joseph Chee Chang, Tom Hope, Dafna Shahaf, and Aniket Kittur. 2018. SOLVENT: A Mixed Initiative System for Finding Analogies Between Research Papers. Proc. ACM Hum.-Comput. Interact. 2, CSCW: 31:1–31:21

Summary: In this paper, the authors attempt to help researchers by generating (mining)  analogies in other domains to help support interdisciplinary research. The paper proposes a unique annotation schema to extend prior work by Hope et al. and has four key facets: background, purpose, mechanism, and findings. The paper also has 3 interesting studies. First, it was collecting annotations from domain experts in research fields, and second, using the Solvent system to generate analogies with real-world usefulness. Finally, the authors scaled up Solvent through the use of crowdsourcing workflows. In each of the three studies, they used semantic vector representations for the annotations. The first study had a dataset focused on papers from CSCW and annotated by a member of the research team, while the second study involved working with an interdisciplinary team in bioengineering and mechanical engineering to determine whether Solvent could help identify analogies not easily found with citation tree search. Finally, in the third study, the authors leveraged crowd workers from UpWork and Amazon Mechanical Turk to generate annotations, and the authors found that workers had difficulties with the purpose and mechanism type annotations. On the whole, the Solvent system was found to help researchers and generate analogies effectively. 

Reflection: Overall, I think this paper is well-motivated, and the 3 studies that form the basis for the results are impressive. It was also interesting that there was significant agreement between crowd workers and researchers in terms of annotation percentage. This proves a useful finding more broadly in that novices may be able to contribute to science not necessarily by doing science (especially as science gets harder to do by “normal” people, and is done in larger and larger teams), but by finding analogies between different disciplines’ literatures.

For their second study, the authors trained a word2vec model on a curated dataset of over 3000 papers from 3 domains. This was also good because they did not limit their work to just one domain and strived to generalize their work/findings. However, they are still largely engineering disciplines, albeit CSCW has a somewhat social science component to it. I wonder how it would work in other disciplines such as between the pure sciences? That might be an interesting follow up study. 

I wonder how such a system might be deployed more broadly, as compared to a limited way that was done in this paper. I also wonder how long it would have taken crowd workers to go through the tasks and generate the findings in total.

Questions:

1. What other domains do you think Solvent would be useful in? Would it easily generalize?
2. Is majority vote an appropriate mechanism? What else could be used?
3. What are the challenges to creating analogies?

Read More

Summary:

This paper mainly talks about a new paper search system, and I think that it can be thought as a thought initiation system rather than a paper search system. The article first proposes that scientific discovery is usually promoted by finding analogies in distant fields, but as more and more papers are published now, it is difficult to find papers with relevant ideas in a field, even though in those cross-field. Therefore, in order to achieve this aim, the authors introduced a hybrid system. In this system, crowdsourcing workers are mainly responsible for reading and understanding an article. People need to analyze an article from four aspects, Background, Purpose, Mechanism, Findings. The computer then analyzes the article based on these semantic frameworks, such as through TFIDF, or a combination of different architectures, and then finds similar usages from research papers. And through verification, found that these annotations are more effective, and can help experts.

Reflection:

First of all, I agree with the goal of the thesis, helping more researchers to obtain new ideas by analogy from outside the field, and then use these ideas and innovative ideas to promote the development of science and technology. I also think that analogies can help technology anyway. The article also cited quite a few examples to show the effectiveness of analogy and the breakthrough of research after the analogy. And in my reading in the past few weeks, I often feel that the article uses analogy. Since then, I have been thinking about how to help people more effectively through analogy and learn from other subject areas. Secondly, I think it is a very effective method to introduce people to assist in the completion. This is also the biggest gain in the course of my word. When a problem is encountered, the consideration is to use human power to solve it. And in the article, let the crowd source workers to annotate the article from four directions, I think it can greatly decompose an article, so that the computer can better understand this article. It is still difficult for a computer to directly understand an article and find out from it, but based on these architectures, finding the connection between articles is indeed a relatively simple task. But at the same time, I have some doubts about the effectiveness of the system proposed in the article. The article also spent a considerable amount of space to describe the limitations of this system. And my doubts are mainly focused on the third point, the usefulness in the real world. I think there are many aspects that will affect this practicality. For example, when the data increases, there will be more similar analogies, and the quality of these analogies is difficult to control. As we know, not all the lessons are useful, some ideas may bring Other problems, such as reduced efficiency, wasted resources, etc. The final point is that I think it takes a lot to get a good idea. We also need to control the quality of the work done by the workers and whether the algorithm can be so enlightening. In my opinion, a good innovation is usually electro-optical flint. Although this may be relatively easy to achieve on the basis of analogy, it still needs a good collaboration between human and machine to complete.

Question:

1. Do you think finding analogy by analyze papers based on their framework is a useful way?
2. Is there any other factors might influence this system, such as the increase of similar articles or different understanding between workers?
3. Except the methods mentioned in the article, as computer scientists, what can we do to inspire people?

Read More

Summary

Chan et al.’s paper “SOLVENT: A Mixed Initiative System for Finding Analogies Between Research Papers” explores the feasibility to leverage a mixed-initiative system to categorize research papers into their relational schemas by a collaborative Human-AI team, which can be utilized to identify analogical research papers potentially leading to innovative knowledge discoveries. The motivation of the researchers is the boom of research papers during recent decades, which makes searching for relevant papers in one domain or cross domains become more and more difficult. To facilitate the paper retrieval and interdisciplinary analogies search, the researchers develop a mixed-initiative system called SOLVENT in which humans mark the key aspects of research papers (their background, purpose, mechanism, and findings) with a computational machine learning model extracting semantic representations from these key aspects, which can facilitate the identifying analogies across different research domains.

Reflection

I think this paper conducted an innovative study on how the proposed system can actually support knowledge sharing and discovery in one domain and across different research communities. In the research explosion era, researchers would greatly benefit from using such a system for their own research and explore more interdisciplinary possibilities. That makes me think about why the system can achieve good performance via annotating the content of abstracts in the domains they conducted experiments. As we know, abstracts of the papers usually summarize the most important point of the research papers at a high-level. So it is intuitive and wise to utilize that part for annotating and further tasks. The researchers adopt the pre-trained word embedding models to generate semantic vector representations for each component, which performs pretty well in the tasks presented in the paper. I would imagine that the framework would probably work especially well for experimentation-driven domains, computer science, civil engineering, biology, etc., in which the research papers follow a specific writing structure. Can the proposed framework scale up to other less structured text materials, such as essays, novels, by extending it to full content instead of focusing on the abstracts? I think that would be an interesting future direction to explore.

In addition, one potential future work discussed in the paper is to extend the content-based approach with graph-based approaches like citation networks. I feel this is a novel idea and there is a lot of potential in this direction. Since the proposed system has the ability to find analogies across various research areas, I would be curious to see if it is possible to generate a knowledge graph based on the analogy pairs that can create something similar to a research road map, which indicates how the ideas from different papers in various research areas relate in a larger scope. I would imagine researchers would benefit from a systematized collection of research ideas. 

Discussion

I think the following questions are worthy of further discussion.

• Would you use this system to support your own research? Why or why not?
• Do you think that the annotation categories capture the majority of the research papers? Can you think about other categories the paper did not mention? 
• What do you think of the researchers’ approach to annotating the abstracts? Would it be helpful to expand on this work to annotate the full content of the papers?
• Do you think the domains involved in cross-domain research share the same purpose and mechanism? Can you think about some possible examples?

Read More