Scienti c dissemination is of central importance for the scienti c process. This paper presents CiteTracked, a dataset of peer reviews and citation statistics covering scienti c papers from the machine learning community and spanning six years. We describe and analyze the data collection of over 3,000 published papers, their peer review texts and citation counts, and depict possible usage directions. The dataset aims at fertilizing novel interdisciplinary work between elds such as scientometrics, information retrieval, computational linguistics and natural language processing to study the scienti c publishing process.
Researchers around the globe continuously contribute invaluable information to
the world's knowledge by publishing their ndings in conferences and journals.
These ndings are subject to the scrutiny of the scienti c review process. Peer
reviewing is an essential component of the scienti c process. Leading conferences
and journals use peer reviewing to decide which manuscripts to include in their
proceedings and journals. The reviewing process is of vital importance, yet the
process itself is often subject to debate. For example, a recent experiment to
examine the consistency of the review process observed that reject/accept
decisions were following a disagreement rate of 26% on a random sample of a tenth
of the papers that went through the review process twice (e.g., [
Typically, reviews are accessible only to the authors of a manuscript, and
to the few selected individuals who organize the scienti c venue. Studies on the
qualitative and quantitative properties of peer reviews had been limited. Few
selected top-scienti c venues recently started to make the peer reviews publicly
accessibly. An example is NeurIPS (the conference on Neural Information
Processing Systems, previously named NIPS) and the OpenReview initiative. Such
initiatives contribute to opening up the largely covert process of peer reviewing.
This starts a recent surge of interest in the study of peer reviews, e.g., [
Once a paper is published, citations can be used to estimate the importance
of a paper, as it encodes the implicit judgement of the importance of a paper
by the community. Citation statistics hence can provide a valuable signal to
study the scienti c impact. E orts in understanding such signal typically resort
to modeling citation networks and hence typically refer to the past [
In this paper, we present a novel corpus that provides a possible link between these research strands and enables prediction of scienti c impact and the study of peer reviews. Our corpus called CiteTracked contains over 3,000 papers and over 12,000 reviews from the NeurIPS conference spanning the last six years. 2 2.1
The dataset contains 12,260 peer reviews and meta-data for a total of 3,427 papers published in the NeurIPS proceedings (http://papers.nips.cc) from 2013 to 2018. The meta-data includes information such as author names, abstract, title, a link to the original paper, and for a subset of editions also the paper presentation type (event type: oral, spotlight or poster) and author feedback.
An overview of the data set is provided in Table 1. First we notice that the data follows the general trend of increasing publication volume at computer science venues. There was an (almost) steady growth in papers, from 360 in 2013 to 679 in 2017, with a big step in 2018 (1,009 papers).3
For all years except 2015 and 2016, for each paper an average of 3 reviews per paper was solicited. In 2015, an average of 4 reviews per paper was implemented, while in 2016 this amounted to 6 reviews per paper. There is no further numerical scoring publicly available besides the review text, except for a single year (2016, in which reviewer con dence scores are available as well).4 3 Note that we had to exclude three papers from the 2018 edition due to review pages which were inaccessible. 4 We would like to note that beyond NeurIPS there are further venues such as ICLR which make review data publicly available, e.g., on the OpenReview platform. Col
We embarked on a manual e ort to collect citation statistics over time. CiteTracked is intended to be an on-going dataset collection e ort. The current release contains citation counts for all papers published up to 2018, i.e., for a total of all 3,427 scienti c papers citation counts of 5 time spans are available.
The goal is to collect citation counts at di erent time intervals, with at least
one such iteration per year for every paper (collected in a short time span) which
was originally a manual e ort and has now been semi-automatized. The recording
of the citation scores started in April 2016 (citations1). The citation scores were
further recorded in November 2016 (citations2), June 2017 (citations3), March
2018 (citations4), and June 2019 (citations5). Citation counts were collected via
the bibliographic database provided freely by Google Scholar. In contrast to
subscription-based services such as Web of Science of Scopus, Google Scholar
provides higher coverage.5 Citation statistics collected in this way provide an
alternative to within- eld citation networks [
lecting the reviews from this source and respective citation counts for papers is
currently beyond the scope of the current project. Recent work has started to collect
such peer review data, cf. [
In this section, we provide a rst quantitative and qualitative analysis of the corpus. We showcase the potential of using CiteTracked for data-driven analysis of linking peer reviews to scienti c impact. 3.1
In this section, we highlight the papers that received the most citations per year.
We analyze whether the presentation type of a paper is linked to higher impact.
Datasets like CiteTracked or PeerRead [
Peer Reviews for Citation Impact Another use case is to study whether reviews are predictive of citation impact. This is an aspect that, to the best of our knowledge, has not been studied yet. Therefore, in this paper we study whether we can successfully predict citation impact from peer reviews, and to what extent it is complementary to earlier work that relied on aspects of the paper itself.
In order to predict the impact of the scienti c papers, we discretize timenormalized citation statistics into low, medium and high impact papers based on a boxplot and outlier analysis. We use the Upper Outlier Threshold (UOT). UOT is de ned by adding up the Inter Quartile Range Rule (IQRR) to the third quartile. Papers with a growth rate above the UOT are therefore de ned as high impact papers. Papers with a growth rate that is below the UOT, are de ned as low/medium impact papers, which were further split up into low and medium impact papers based on an UOT analysis on the subsequent UOT analysis.
We create baselines which consider meta-data from the paper itself, namely
the title and the abstract as done in earlier studies. We train a Support Vector
Machine with commonly used features extracted from the text and titles of
the paper, i.e., word n-grams and character n-grams, including special features
motivated by earlier work such as the average word length of the paper title,
paper title length. In particular, this speci c feature set includes the use of
question marks in paper titles [
Table 4 shows the results. There are several take-aways. First of all, paper information is predictive of scienti c impact. A classi er that only uses the pa- Table 4. Results of predicting impact per title is able to achieve an average level of papers (F1 score). performance of .81 F1-score. This out- low mid high avg performs the random strati ed baseline title .91 .0 .26 .81 of .71 and con rms earlier ndings. A abstract .92 .0 .39 .83 closer look reveals that the model strug- reviews .93 .09 .49 .85 gles to predict the mid class. It falls all .92 .24 .48 .85 mostly back to the majority class (the low impact papers). Adding the paper abstract improves overall performance (from .81 to .83). Secondly and most importantly, the results show the potential of review texts. Review texts are predictive of scienti c impact. The performance of a model based on review texts is higher than using only abstract or title, thereby con rming our hypothesis that reviews constitute valuable information for scienti c impact prediction. A model which uses all information (title, abstract and reviews, indicated as `all' in Table 4) result in an overall similar performance to reviews alone, but it improves prediction F1-score for the di cult mid class. This investigation shows the potential of learning from peer review texts. 4
This paper introduces CiteTracked, a corpus of peer reviews from the NeurIPS conference enriched with citation statistics collected over several years. The current corpus contains 3,427 papers and over 12,000 reviews. We outline corpus collection, provide an initial analysis and discuss potential use cases that link work on bibliographic indicators, peer reviews and scienti c publication impact.
We would like to thank Stijn Eikelboom for help with improving the citation collection process and NVIDIA for supporting our research.