Bibliometrics is more and more interested in the full text processing and the study of the structure of scientific papers. The contexts of in-text references present in articles are particularly relevant for such studies. This work describes the construction of the InTeReC dataset, which is an in-text reference corpus that aims to promote experimental reproducibility and to provide a standard dataset for further research. The InTeReC dataset is a set of sentences containing in-text references together with all the data necessary for their recontextualization in papers using standard CSV format. This should encourage the implementation of natural language processing tools for Bibliometric studies and related research in information retrieval and visualization.
The assumption that the contexts of the bibliographic references present in a scientific article play an important role in characterizing the relationship between citing works and cited works have been accepted for many decades. Publications are connected to each other by citations and citations contexts categorize the semantic relations that exist between them. Whether the study of citation contexts relies mainly on linguistic clues or machine learning techniques, citation contexts for each research experiment need to be extracted from scientific corpora. Also, the extraction of citation contexts is a preliminary step to any statistical, distributional, syntactic or semantic analysis.
Sentences containing in-text references may contain relevant information
about the cited research and cited author’s research areas. Recently, He and
Chen [
In this paper, we propose a large scale dataset of citation contexts and
explain the methods that were used for its construction. Other similar initiatives
exist with various objectives. For example, the ESWC-14 Challenge: Semantic
Publishing3 – Assessing the Quality of Scientific Output (see [
In this paper we describe the creation of the InTeReC dataset [
The aim of this paper is to present the method of the construction of the InTeReC corpus, which is a standard in-text reference corpus taking into account the different elements relevant to the implementation of experimental protocols. The overall objective is to facilitate citation context analyses and various distributional analyses by providing a large dataset to the community. The InTeReC dataset also serves the purposes of reproducibility, interoperability and cumulative research. 2
The construction of the InTeReC dataset is based on several analyzes and experiments carried out in the recent years. In this section we summarize the methods that were used in order to propose a dataset that is reusable by the community for studying citation contexts.
Working with the full text of papers, we first classify the section titles in
order to identify the four major section types in the IMRaD sequence
(Introduction, Methods, Results and Discussion). This categorization aims to verify
the coherence of the corpus with the IMRaD structure. In many articles, the
four section types exist but not in the same order. For the InTeReC dataset, we
focused only on paper that follows the IMRaD structure, i.e. papers that contain
the four section types in the correct order. We then process the text content of
all paragraphs and segment them into sentences. In our approach, sentences are
considered as the basic textual units and are used to express the positions of
references in the article and in the section. This approach allows for example
to assign relative positions of all references and to obtain the distribution of
references along the text [
The links between the in-text references and the cited papers or bibliography items are preserved throughout the processing. 2.1
For this corpus, we have used the entire set of research articles published by PLOS5 up to September 2013. This initial corpus contains 90,071 articles.
As these 7 journals follow the same publication model but are in different scientific fields, our aim is to observe the different uses of bibliographic references in these fields and their relation to the structure of the articles.
PLOS provides access to the articles in the XML format. The set of XML elements and attributes that are used for the representation of journal articles are known as Journal Article Tag Suite (JATS), which is an application of Z39.962012. Technology evolves quickly and we have to take into consideration that JATS is a continuation of the NLM Archiving and Interchange DTD works by NCBI6. As this format is also used by PubMed, this work can easily be extended to processing the PubMed Open Access Subset which is a larger dataset. The JATS structure of an article consists of three main elements: front – body – back, and the textual content of the article is in the body element. It is further divided into sections and paragraphs. The front element contains some traditional metadata fields (title, authors, etc.) as well as the article type.
Different types of articles are present in the corpus, such as "Research article", "Synopsis", "Primer", "Essay", and the typology is given in the article’s metadata. We have focused on the "Research article" type, obtaining a total of 85,660 articles out of the initial 90,071 articles in the corpus.
lisher of seven peer-reviewed academic journals, mostly in the fields of Biology and Medicine. PLOS ONE, the publishers’ general journal covers, however, all fields of science and social sciences.
2.2
One of our objectives is to identify the rhetorical structure of the articles. The use of the IMRaD sequence (Introduction, Methods, Results and Discussion) is part of the editorial requirements of the PLOS journals and the large majority of articles include these four sections. In some articles however, the sections are not always in the same order.
Sections are represented as separate elements in the original XML files. The
research articles in the corpus contain a total of 404 311 sections. We categorized
them automatically by analyzing the section titles in order to match the existing
sections with one of the section types in the IMRaD structure [
Class Section type I Introduction M Methods R Results D Discussion (MR) Methods and Results (RD) Results and Discussion Total
We further restricted the set of sentences to be included in the InTeReC dataset by selecting only sentences that have a single citation and that contain at least one occurrence of the most frequent verbs that have been attested in citation contexts. These steps are explained in the following subsections.
Each paragraph was segmented into sentences by analyzing the punctuation of the text following a set of typographic rules. All the occurrences of symbols denoting sentence boundaries (point, exclamation mark, etc.) were examined and disambiguated. In fact, the occurrence of a point in a text does not necessarily mean a sentence end, because in many cases it can be part of an abbreviation, references, genus species, numeric values, etc. We used a set of finite-state automata in order to determine the contexts in which the points signal sentence ends. 2.3
Once we classified the sections, we examined the sequence of sections present in each article. To produce the InTeReC dataset, we focused only on articles where the order of the four sections is: "I,M,R,D". Considering merged sections, there are three possible article structures, that are listed in table 2. The last two columns of this table give the total number of sentences in the articles and the number of sentences that contain at least one in-text reference.
Article structure Articles Sentences Sentences with references I,M,R,D 44,370 7,656,518 1,704,326 I,M,(RD) 2,971 504,246 113,237 I,(MR),D 28 5,300 937 Total 47,369 8,166,064 1,818,500
The following processing was done on these 47,369 articles from which was selected the InTeReC dataset. 2.4
Our algorithm examines each sentence and counts the number of references present in the text. In fact, the input data is in the XML format where the references are represented in <xref> tags. Our algorithm covers all possible typographic variations for reference ranges and infers the missing data from the input XML. As a result we obtain the list of sentences in the text, where to each sentence we have associated a reference count as well as a list of reference identifiers corresponding to the bibliography entries.
We note that counting the <xref> tags are not a reliable method to obtain
the reference counts, especially if one is interested in multiple in-text references
(MIR) [
For this first version of the InTeReC dataset we have chosen to include only sentences that contain one single reference. These citation contexts establish links between only two works, the cited work and the citing article, and thus we can consider them as the simplest cases to study in terms of citation context analysis. 2.5
A series of experiments have been published around verbs occurring in citation
contexts and their distributions [
Bertin et al. [
All sentences were processed using the Part-Of-Speech tagger of python
NLTK7 [
The InTeReC dataset contains sentences that contain occurrences of the verbs in table 3 and their verb phrases have been identified. By keeping only sentences that contain these verbs, we eliminate many sentences that contain perfunctory citations because they only mention the cited work without explicitely identifying the its relation with the article. Thus we obtained the final set of 314,023 sentences for the dataset. 3
The InTeReC dataset contains a list of sentences in full text. Information is given on the position of the sentences relative to the article and the section
in which they appear, the section type with respect to the four main types of the IMRaD structure, as well as verb phrases that occur in the sentence. Each sentence contains one single in-text reference.
The dataset is published in the CSV format [
The format of the dataset has been chosen to facilitate the exploitation of the data and make it compatible and easily reusable for most types of processing. 4
Although this corpus takes many aspects into account, it is not exhaustive and has characteristics that underline the inherent limitations of this approach. For example, we have limited this work to level of the sentence, and thus do not take into account relevant citation contexts that span across sentence boundaries through the use of anaphora.
Many improvements are possible and they are currently receiving our attention in the evolution of our corpus. The first concerns the quantitative aspect with the extension of the corpus to new sources. For this, we can take into account for example PubMed8, arXiv9 and the CEUR Workshop Proceedings10.
The second evolution concerns a more qualitative aspect of the dataset and
adding semantic annotations. The main idea is to implement semantic annotation
in order to provide an training dataset for supervised learning tools. For this
purpose, we investigate tools to annotate segments with precision values close to
those of human annotators. Indeed, one challenge would be proposing semantic
annotations for ontologies such as CiTO (see [
Finally, the InTeReC dataset can be accessed and visualized using R-shiny11 interface in order to provide users a way to interact with the data and observe the distributional phenomena.
This dataset aims to facilitate the reproducibility of future research on in-text citation analysis and thus provide a common foundation for the development of a unified model of citation context analysis. 5
We thank Benoit Macaluso of the Observatoire des Sciences et des Technologies (OST), Montreal, Canada, for harvesting and providing the PLOS dataset.