Automated methods to collect papers for literature reviews have the potential to save time and provide new insights. However, a lack of labeled ground-truth data has made it di cult to develop and evaluate these methods. We propose a framework to use the reference lists from existing review papers as labeled data to train supervised classi ers, allowing for experimentation and testing of models and features at a large scale. We demonstrate our method by training classi ers using both citation- and text-based features on 654 review papers. We also demonstrate how this method may be extended to generate a novel review collection for a newly emerging research eld.
Conducting a literature review, or survey, is an important part of research. The
vast and exponentially growing body of literature makes it increasingly di cult
to identify even a slice of the relevant papers for a given topic [
Many methods have been developed to recommend relevant papers, using
features related to textual similarity, keywords, and structural information such as
relatedness in a citation network [
We begin by developing methods using the citation list from a single review article as a benchmark (section 3.1). We then show how this method can be applied to a large number of review articles (section 3.2). Finally, we apply these methods as a case study to the emerging eld of misinformation studies (section 3.3). We make code and sample data for this project available at https: //github.com/h1-the-swan/autoreview.
There have been several previous attempts at automated or semi-automated
literature surveys. These approaches have tended to be smaller scale and rely
on more qualitative means of evaluations, which are di cult to replicate and
compare across studies. For example, Chen [
We are aware of two previous attempts that use review articles to test an
automated literature review system. Belter used a semi-automated technique to
retrieve documents for systematic reviews using citations [
A related problem to the one of identifying papers for surveys is the
recommendation of scholarly papers. This topic has been extensively studied; a recent
survey paper on research paper recommender systems [
The network data used in our analysis came from an October, 2017 snapshot
of the Microsoft Academic Graph, an indexing service for scholarly publications
consisting of 1.2 billion directed citation links between 77 million papers [
We used Infomap to cluster the citation network [
Our procedure is presented in Fig. 1. The rst step is to randomly split the papers into a set of \seed" papers and a set of \target" papers. We are imagining a researcher who is starting with a set of papers relating to a topic.
This researcher wants to expand this set to nd the other relevant and important
papers in the topic. Ideally, we would like to search for these target papers within
the total set of papers in our data set. However, it is infeasible to generate
features and train models using the total set of 77 million papers. To narrow the
total set to a more reasonable number of candidate papers, we collect all of the
papers that have either cited or been cited by the seed papers. We then go one
more degree out, taking all of the papers that have cited or been cited by all of
those. This process of following in- and out-citations imitates the recommended
practice for a researcher looking for papers to include in a survey, but at a larger
scale [
Our next step is to generate features to use in a classi cation model. To
incorporate the clustering information we have, one feature we use is the average
cluster distance between a paper and the 50 seed papers. Distance for two papers
i and j is de ned as (Di + Dj 2DLCA)=(Di + Dj ) where Di and Dj represent
the depth in the clustering tree hierarchy of i and j, and DLCA represents the
depth of the lowest common ancestor of the two papers' clusters. The feature for
paper i is the average distance to each of the seed papers. We also use pagerank
as a measure of citation-based importance [
Table 1 shows the results from ve splits, each using a di erent random seed. The \random seed" is an integer that the sampler uses as a starting point; each di erent random seed leads to a di erent split of the initial set of papers into seed and target sets. For each run, we split the 447 papers into a set of 50 seed papers and 397 target papers. After collecting candidate papers, we cleaned the data by removing the seed papers, papers for which we did not have titles, and papers published after the year the review paper was published (2010). Each seed (i.e., each row of Table 1) represents one instance of the process in Fig. 1. We report the number of candidate papers in the nal set for each run. These sets of candidate papers range in size from 600K to 1.6M papers. In each case, only 397 of these papers are in the positive class. This parallels the experience of a researcher trying to do an e ective survey of a topic|the goal is to nd the right papers in a vast sea of literature.
We report the performance of the models as the R-Precision, the fraction of
target papers found in the top N papers, where N is the total number of target
papers|397 in this case [
Our next step was to apply these same methods to more review papers. In order to identify a set of review papers from which we could pull bibliographies, we turned to the Web Of Science (WoS), which identi es review articles in its citation index data. In order to smoothly apply the same method as above, we limited our sample of review papers to those that could easily be linked to the Microsoft Academic Graph using a Document Object Identi er (DOI). We tested a sample of 648 review articles, choosing papers with the largest bibliographies in order to limit artifacts from insu cient input data.
For each review article, we gathered the cited papers from MAG, and trained models for 5 di erent random seeds, representing 5 di erent splits of the data into seed and target papers. We chose the best-performing model for each split| invariably a random forest classi er using the network and title-text features described above. Fig. 2 shows the distribution of R-Precision scores (number of correctly predicted target papers divided by total number of target papers) for 3,259 classi ers, each trained and tested on one of the 648 review articles. The average score was 0.30 (standard deviation 0.11); the highest score was 0.76. 3.3
The method we introduce can be adapted as a tool for exploring key papers
in an emerging eld. In this use case, it is the papers the classi er \misses"
that we are interested in. The classi er, attempting to predict the target papers,
assigns a con dence score to each of the candidate papers. We are interested in
those candidate papers which received a high score, yet were not actually target
papers. In the classic classi cation task, these would be considered misidenti ed,
2 Machine learning experiments were conducted using scikit-learn version 0.19.1
running on Python 3.5.2. Trying a variety of classi ers, we saw the best performance
with logistic regression and random forest models.
but in this task we consider the possibility that their similarity to the seed
papers may make them relevant papers for this eld. This is consistent with
Belter's suggestion of \supplement[ing] the traditional method by identifying
relevant publications not retrieved through traditional search techniques" [
As part of this case study and in collaboration with the National Academy of Sciences, we curated a collection of important papers in this eld3 and used this collection as a seed set to identify other related papers that might have been missed by our more manual methods. Evaluating these results brings us back to shaky territory where we do not have ground truth. However, conversations with domain experts interested in formally characterizing these elds have been encouraging, suggesting the utility of these methods in identifying relevant papers. 4
Our preliminary results suggest that it is possible using these automated methods to identify many of the most relevant papers for a literature review from a large set of candidate papers. We believe that, by trying new features and tuning model parameters, we can increase performance and learn more about what distinguishes these papers. We have also seen promise in using these methods to build novel surveys of topics from a set of seed papers.
Furthermore, we see potential in using this framework to develop and evaluate methods for literature survey generation and related problems such as scholarly recommendation and eld identi cation. The objective we propose for our modeling task|accurately nding all of the remaining references from a review paper given a held out sample of seed papers from those references|is not a perfect one. We assume that the references in a review paper represent domain experts' best attempt to collect the relevant literature in a single research topic; however, there exist several di erent types of review article (systematic review, meta-analysis, broad literature survey, etc.), and our current method ignores potential nuance between them. Additionally, we assume that every article in a review paper's bibliography is a relevant article to be included in a eld's survey; in practice, an article can be cited for many di erent reasons, even within a review article. Despite these limitations, the large amount of available data allows our framework to provide a means of experimenting with and developing methods for automated literature surveys. There are many review articles similar to the ones we used that have their bibliographies available and so it will be possible to do this development and analysis on a large scale across many domains. Using this framework, it will be possible to empirically evaluate novel features for their use in identifying papers relevant to a survey in a given topic. 3 See Data and Methods at http://www.misinformationresearch.org for details
Below is a sample of results (random samples of true positives, false positives,
true negatives, and false negatives) from the autoreview classi er using the
references from Fortunato et al. [
True Positives Rank
Title Year
Modularity and community structure in networks. 2006 Optimization by simulated annealing. 1983 An iteration method for the solution of the eigenvalue problem of linear 1950 di erential and integral operators Maps of random walks on complex networks reveal community structure 2008 An e cient heuristic procedure for partitioning graphs 1970 Near linear time algorithm to detect community structures in large-scale 2007 networks Graphs over time: densi cation laws, shrinking diameters and possible 2005 explanations Evolutionary spectral clustering by incorporating temporal smoothness 2007 The Elements of Statistical Learning 2001 Community detection by signaling on complex networks 2008 4 These results use a slightly di erent version of the input data than our original pilot study in section 3.1, which is why there are more target papers (411) than in the pilot study.
False Positives Rank
Elements of information theory The complexity of theorem-proving procedures Clustering Algorithms The Concept and Use of Social Networks Fundamental statistics in psychology and education Line graphs of weighted networks for overlapping communities Some simpli ed NP-complete graph problems Quantizing for minimum distortion The advanced theory of statistics Handbook of Mathematical Functions
True Negatives Rank 50738 61089 121773 151627 192168 624287 1011214 1057264 1099562 1122428 Community Structure in Congressional Cosponsorship Networks Local method for detecting communities.
On Modularity - NP-Completeness and Beyond A method for nding communities of related genes Self-similar community structure in a network of human interactions. Spectral coarse graining and synchronization in oscillator networks Modular organization of cellular networks Sequential algorithm for fast clique percolation Categorical Data Analysis of Single Sociometric Relations Cliques, clubs and clans Linguistic Bayesian Networks for reasoning with subjective probabilities 2003 in forensic statistics Comparison of Sensor Management Strategies for Detection and Classi - 1996 cation.
Testing goodness of t for the distribution of errors in multivariate linear 2005 models Low-cost, bounded-delay multicast routing for QoS-based networks 1998 4 Cross-language facilitation, repetition blindness, and the relation be- 2002 tween language and memory: Replications of Altarriba and Soltano (1996) and support for a new theory Developing visual sensing strategies through next best view planning 2009 Mis-generalization: An Explanation of Observed Mal-rules. 1984 Global xed-priority scheduling of arbitrary-deadline sporadic task sys- 2008 tems Facilitation Catalyst for Group Problem Solving 1989 Discriminative analysis of brain function at resting-state for attention- 2005 de cit/hyperactivity disorder
False Negatives