We investigate the TREC-CDS 2016 test collections as a new resource for citation context and citation-based IR experiments. The collection contains more than 1.25 million biomedical full-text articles in XML. We nd that a citation index can easily be extracted, and citation contexts easily be identi ed. We conduct initial experiments to determine the optimal citation context window size in this domain and collection. Surprisingly We nd that quite long citation contexts of more than 250 word yield the best performance when combined linearly with the fulltext and with moderate weight on the citation contexts.
In this work, we investigate the feasibility of extracting citation contexts from citing articles and using them in the retrieval of scienti c documents. Adding citation contexts of a document to its full-text may help the retrieval process by providing additional relevant keywords for indexing. Keywords from citation contexts may be valuable as they provide a di erent perspective on the cited text | that of the authors citing and using the cited document.
Prior research on this idea indicates that citation contexts can indeed
improve retrieval performance, e.g. [
The paper is structured as follows: Section 2 discusses related work. Section 3 presents the methods we used including analysis of the TREC-CDS 2016 test collection, and details on the extraction of citation contexts. Section 4 describes our experimental setup and our ndings. Section 5 presents discussion and conclusions. 2
In establishing the Science Citation Index in 1964 Eugene Gar eld created a retrieval system solidly based on the idea that citations form explicit links between papers that have particular points in common and can be used to search the scienti c literature [7, p.1]. A researcher would rely on the author's judgement to include references to other publications with shared subjects or topics and thus, through the network established from bibliographic references, identify other citing or cited papers with similar or new topics or subjects in common [7, p.2]. Continued as Clarivate's Web of Science, Gar eld's citation indexes is still based on the links between papers, but ignores the nature and meaning of these links. With the increasing availability of scienti c literature in full-text and as open source, we can now begin to investigate the nature of the links by studying the text where a given paper is mentioned. Existing research has demonstrated that the text surrounding citations often contains descriptions of the cited paper, the reason or function of the citation, or the disposition towards the cited paper [10, p.201]. As such, it may be possible to glimpse what the cited paper is about, or how the author has used the paper, by examining the surrounding associated text of a speci c citation. In other words, by examining the textual content of a citation, it is potentially possible to identify how and why a citation is made.
Analysis of Citation Contexts The purpose of citation context analysis, rst
proposed by Small[
Citation context analysis has been studied from a number of perspectives,
including citation summarisation [
Analysis of Citation Contexts and IR Performance Of particular interest
to the present work is attempts to enhance and improve retrieval of scienti c
documents. An early example is O'Connor who proposed to extract additional
indexing terms from citation contexts that cite a given paper [
Following a similar approach to Ritchie but at a larger scale, Dabrowska
and Larsen performed retrieval experiments on a test collection of over 430.000
full-text papers from a subset of the iSearch (Integrated Search) document
collection [
For our experiments we use the TREC 2016 Clinical Decision Support Track 1
data-set [
and is a free archive for full-text biomedical and life sciences journal articles. It contains at present more than 5 million full-text articles, most of which are also included in as bibliographical references in the 25+ million records of PubMed. Each article in the collection is represented as an NXML le and identi ed by an o cial PMCID.
The topics of the track simulate actual information needs of physicians and
are divided in three di erent types representing the most common generic clinical
questions [
Citation contexts are de ned as the textual passages or sentences surrounding
or containing the citations [
We downloaded the TREC-CDS 2016 collection and rst investigated the raw XML les to determine if (i) internal citations between the documents in the data-set can be readily identi ed, and to answer the question: (ii) how feasible is it to identify and extract citation contexts?
The documents in the TREC-CDS 2016 test collection are encoded in the Journal Archiving and Interchange Tag Set (JATS)3. All full-text documents in the test collection use the XML le type and use the tags de ned in the JATS DTD. The JATS include special tags for bibliographic references (<xref>) which are wrapped around cross-references in the full-text document. The conducted experiments utilised the cross-reference tags to determine the position and target reference for each in-text reference. Additionally, the JATS include special tags for the reference list to ease parsing of the list. This list was used to determine the target document for each in-text reference. The documents in the test collection have two unique and di erent identi ers that we could use to map documents to each other: (i) PubMed Central Identi er (PMCID) This identi er is assigned to all documents included in the PMC. As the test collection is a snapshot of a subset of the PMC, all documents will have this identi er. (ii) PubMed Identi er (PMID) This identi er is assigned to any record also included in PubMed. This identi er (and not the PMCID) has also been added by PMC sta in the JATS reference lists when pointing to documents included in PubMed.
As each document has a PMCID as well as a PMID (if it is in PubMed) in its header, we were able to match these two IDs as a basis for extracting a citation network and citation contexts. It is important to note that because not all documents in the PMC are also present in PubMed, not all documents in the test collection have a PMID. 87.622 (7%) of the documents did not have a PMID.
Table 1 gives a summary of the statistics of the extracted data. Of the 1.25 million documents just over a million have references (87.3%). 58,845 of the documents without references are abstract-only documents without full-text, and the remaining may be publication types that do not contain references (e.g. editorials, news items, etc.) The 1+ million documents with references contain more than 43 million references (40 references on average per article). Of the 1,255,260 documents in the collection, 567,650 documents (45.2%) received at least one citation from within the collection. 370,426 of these were cited between 2 and 100 times | see Table 1 for other ranges. More than 60 million citation contexts were identi ed (the same document can be mentioned more than once in the full-text in one document). 46 million of these (76.6%) have a target PMID pointing to a PubMed document, and 4.8 million (8%) could be matched to a PMCID in the TREC-CDS 2016 collection. These 4,833,813 citation contexts point to the 567,650 cited documents and form the core data-set used in our experiments. This means that each cited document has 8.5 linked citation contexts on average. A total of 37,707 documents were assessed for relevance in relation to the 30 topics. Some were retrieved and assessed for multiple topics. The number of unique PMCIDs in the QRELs is 28,349. Of these, 10,132 (36.1%) were cited and had at least one associated citation context. However, only 2,019 of these were assessed as De nitely relevant or Possibly relevant. This means that only 37% of the relevant documents also had citation contexts added.
Contexts total count Contexts with target PMID Usable contexts (i.e. with a PMCID) Documents with appended contexts Length of QREL Unique PMCIDs in QREL Unique and relevant documents in QREL Documents with appended contexts in QREL Relevant documents with contexts in QREL # of docs/refs/contexts
Citation Context lengths We limit ourselves to the simplistic, however,
widely used approach of using xed window sizes surrounding the citation. This
is done by considering the citation as a starting point and then extend the
context to include text around the citation with some determined xed or variable
distance on each side of the citation. Citation context text can be variable and
range from just a few characters, over phrase and clauses to sentences and
entire paragraphs. The task of identifying the optimal context length of citations
is di cult and complex, as the citing behaviour, characteristics and processes
of citations is di erent within papers and sections and across authors, scienti c
discourses, elds and domains [
Retrieval experiments As we are working with a test collection, our
experiments are solidly within the Cran eld tradition. We use the Indri IR system
to conduct the retrieval experiments, with Language Modeling and Dirichlet
smoothing [
To integrate citation contexts seamlessly into retrieval we took advantage of two features in Indri. First, we placed the full-text of the original article in a separate eld and added the di erent versions of extracted citation contexts in elds of their own. Second, we used the Indri query language to create a linear combination of the two. We used the #weight operator to assign weights to the full-text and citation context respectively in each query, with the weights adding up to 1. The following is an example of the query formatting (topic 1 from TREC-CDS 2016):
#weight (0.6 #combine [fulltext] (A 78 year old male presents with frequent stools and melena) 0.4 #combine [250 25] (A 78 year old male presents with frequent stools and melena))
where the same query string is matched against the full-text eld with a weight of 0.6, and against one of the citation context elds with a weight of 0.4 (the #combine operator is standard operator for combining beliefs in Indri ). In this way we can add the citation contexts as additional index terms to the cited document, and control their in uence relative to the full-text. We tested weights of 20, 40, 60, 80 and 100% in the main experiment. Retrieval results were evaluated using trec eval. For P@10 and MAP both De nitely relevant and Possibly relevant were counted as relevant | for NDCG De nitely relevant had a gain value of 2 and Possibly relevant a gain value of 1. P@10 represents a purely precision-oriented perspective on results whereas MAP and NDCG gives a perspective that balances precision and recall. It should be noted that retrieved documents that were not assessed were counted as non-relevant in all the measures. 4 The smoothing parameter for the baseline run was 24,000 for MAP and 30,000 for MAP and NDCG. 4.2
The aim of the experiments was to learn how much context to include to reach optimal retrieval performance, and to determine how much weight to put on them relative to the full-text. Table 2 shows the overall results for P@10, MAP and NDCG. The table shows the best result for each run with the optimal smoothing parameter and the weight between full-text and citation context that performs the best. Figures 1 and 2 below demonstrate the e ect of weighting and smoothing.
From Table 2 we see that the citation context runs all outperform the baseline
to some degree. The runs with more context added in general perform better.
The best performing run for P@10 is the one with 250 words added (19% over
the baseline), and for MAP and NDCG the run with 300 words added (4.6%
and 2.8% over the baseline respectively). The best performing word-based runs
outperform the sentence-based runs. An explanation may be that the
sentencebased runs in general would be shorter in term of the number of words - with
the longest of 5 sentences corresponding to 150 words or less. 5
5 We did not examine sentence length of the 4.8 million citation contexts, but other
studies of academic biomedical text nd around 25 words per sentence on average
[
With regards to whether adding text to the right of the in-text reference is bene cial, a more complex picture emerges: for early precision (P@10) runs without right-hand text performs better in almost all cases | especially for the better performing runs with more words. For MAP and NDCG the actual di erences are very small | for the runs with more context in the 100-250 word range performance is marginally better with right-hand text, but as more context gets added, right-hand context doesn't seem to have much in uence, as can be seen for the MAP performance (4.6% over baseline) in the 250-300 word range.
Overall, all results point to the fact that more context added leads to better performance. Given this nding even larger context windows should be experimented with to ensure that an upper-bound has indeed been reached at 300 words, which was the maximum context length studied in the present experiment.
The results in Table 2 represent the best smoothing and weight combinations. With respect to the linear combination the top 8 runs in relation to P@10 all have 40% weight on citation contexts - for MAP and NDCG the top runs all have a 20% weight on the citation contexts (not shown in Table 2). Figure 1 demonstrates the e ect of the linear combination of full-text and the added citation contexts for MAP for weights of 10, 15, 20, 25 and 40%. The shading of each line indicates the variation across the parameter tuning range, and the dashed line is the baseline. It can be seen that performance is very stable, and increases steadily as more weight is placed on the citation contexts from 10% up to 25%. This also holds across the di erent context length combinations with the 0.2633
MAP top performance being reached at 300 words as discussed above, with scores that are consistently over the baseline except for the shortest citation context windows. At 40% however, performance overall drops below the baseline and shows great diversity across contexts lengths, demonstrating that too much weight on the citation contexts can hurt performance and lead to erratic behaviour. It can also be noted that variation across the smoothing parameter range (shading) is not prohibitively large with little overlap between each type of weighting.
Figure 2 further illustrates the e ect of smoothing for contexts of 250 words (the best performing context length for P@10). Results are plotted for weights of 15, 20, 25 and 40% and for P@10, MAP and NDCG. A low or moderate level of variation across the tuning range is desirable as such an approach is less dependent on setting the tuning parameter correctly and can thus be considered more stable. It can be observed that performance is quite stable across the chosen smoothing range, with a bit more variation for P@10, which can be expected as it is a less stable measure. It can be clearly seen that a weight of 40% outperforms other weights for P@10 across the smoothing range. At a weight of 20% MAP and NDCG is very stable across the smoothing range. Higher performance can be achieved at 25% for MAP and NDCG but only at the lower smoothing values.
Finally, it should be noted that as expected, the citation distribution is skewed (1) - with 63% receiving no citations, the remaining 37% receiving more than 1 - and 18 documents receiving more than 1000 citations. With the experimental setup used this means that some documents have no additional representation, and that a sizable proportion have a great deals of text added to their representation - in some cases tens of thousands of words. The e ect of this on retrieval is at present unknown. 5
Generating a citation index and extraction of citation contexts It was unproblematic to identify internal citations documents in the TREC-CDS 2016 collection due to explicit tagging in the XML and the added PMIDs in the reference lists. Not all documents have references, but more than 1 million do | providing a rich testbed for citation-based IR. Further, 45% of the documents in the collection received at least one internal citation. The resulting citation index from RQ1 and the XML and JATS formats made it possible to identify citing documents and to identify the location of in-text references. 4,8 million citation contexts could thus be extracted and linked to the 567,650 cited documents. It is worth noting that only 37% of the relevant documents had one or more citation contexts added - this reduces the impact that citation contexts can have on IR performance in the TREC-CDS collection. This underlines that fact that retrieval based on citation contexts is inherently dependent on documents being cited, and that a test collection which has been created with citation and citation context based runs in the pooling process is really needed to fully investigate the true potential of these approaches.
We limited ourselves to a simple de nition of citation contexts extracting a number of words or sentences before and/or after each in-text reference. Tests of more advanced linguistically-based de nitions would be interesting, but can be a challenge given the large number of contexts. Somewhat to our surprise the best performance was found among the longest citation windows of 250-300 words - both for precision and recall-oriented measures. As argued this a large amount of text (a full page) that almost certainly goes beyond where a given cited document is discussed. This may indicate that identifying the exact extent of the actual citation context may not be of prime importance - and on the other hand leads to the question of why so much text from citing documents is bene cial for retrieval, and if even larger windows will be bene cial?
Compared to previous research this is much longer citation context windows than previously tested. With regards to how much weight to put on the contexts our ndings are in line with previous work of e.g. Ritchie (2009) and Dabrowska (2014). Best performance is achieved with moderate weight on the citation contexts of around 20% relative to the full-text - much more leads to decreased performance and erratic behaviour. As regards stability, results are quite stable across the smoothing range, indicating that this approach does not depend critically on getting the smoothing parameter right.
The present study mainly serves to introduce the TREC-CDS 2016 test
collection as an attractive ressource for the BIR community and those interested
in citation context analysis - and to conduct initial tests of the feasibility of IR
experiments using such features on this collection. Much more interesting work,
where it is tested if the context of citations are useful for semantically
categorizing a relationship and perhaps even an intention or an opinion between two
publications, can be built on top of this, e.g. along the lines of Ritchie [
We wish to thank the organisers of TREC-CDS for creating a great resource, and the four anonymous reviewers for insightful comments.