In scientific writing, references are crucial in supporting claims, spotlighting evidence, and highlighting research gaps. However, where to add a reference and which reference to cite are subjectively chosen by the papers' authors; thus the automation of the task is challenging and requires proper investigations. This paper focuses on the automatic placement of references, considering its diverse approaches depending on writing style and community norms, and investigates the use of transformers and Natural Language Processing heuristics to predict i) if a reference is needed in a scientific statement, and ii) where the reference should be placed within the statement. For this investigation, this paper investigates two techniques, namely Mask-filling (MF) and Named Entity Recognition (NER), and provides insights on how to solve this task.
CEUR ceur-ws.org
Citing research papers is common practice to provide evidence, build upon existing knowledge, and substantiate findings. This activity relies on two main tasks: i) authors have to decide where to place a citation based on the written statements, and ii) the cited papers should align with the content of the factual information expressed in the sentences. The first task depends on the author’s experience in adding references to support claims and does not need knowledge about the state-of-the-art literature. The second task requires domain and state-of-the-art expertise to refer to the best and most timely literature. Although these two tasks are related to each other they can be studied independently; this paper focuses on the first task. In more detail, this paper proposes: • A model designed to solve a mask-filling problem using a generative approach. • A Named Entity Recognition (NER) model designed to label tokens that should precede a citation.
We analysed both methods as well as their typical errors. We also introduced a few heuristics that are able to solve some of the common issues and improve their performance. The proposed models are investigated on a manually annotated gold standard. In summary, the contribution of this paper is two-fold: • An analysis of a mask-filling and NER approach for the reference placement problem. • An error analysis and heuristics to solve the identified errors.
We provide the code as well as the data used in our analysis through a GitHub repository1.
The remainder of this paper is organized as follows. Section 2 presents the related work. Section 3 describes the proposed models, their results, and an error analysis. Finally, Section 4 concludes the paper and outlook our future research towards a fully-fledged knowledge-aware reference recommendation system
The task of deciding which statements need a citation is relatively recent. To start with, there
exist resources that provide information about why a reference is cited within a sentence. For
example, CiTO [
However, the task of determining whether a statement in a research paper needs a citation is less explored and only recently has raised the interest of the community working on Artificial Intelligence (AI) systems for the scholarly domain. For example, the problem is explored in [15] where a multi-layer perceptron model is proposed to measure the citation worthiness of scientific statements. Authors in [ 16] tackled the task with various scenarios, including sentence classification with and without context representation, and sequence modeling using contextual embeddings and BiLSTMs. While their results showed significant accuracy improvements with context, their focus remained on deciding whether a citation is necessary, not its precise placement within the text. Most recently, large language models have also been explored for this task. For instance, Vajdecka et al. [17] operated a Large Language Model (LLM) to predict whether a citation is needed in a sentence, achieving F1-scores between 75% and 89%. However, their approach does not pinpoint the optimal placement of the citation within the sentence. Additionally, LLMs can exhibit inherent instability even with clear instructions, making them unsuitable for tasks demanding consistent performance.
This section outlines the task investigated in our analysis, the proposed models, and the resulting evaluation.
The tasks investigated in this paper are the Citation Required Task and Citation Placement Task. Citation Required Task has the main goal of determining whether a sentence needs a citation. It can be seen as a binary classification task that given a scientific statement as an input returns 1 if at least one citation is needed, 0 otherwise. Citation Placement Task has the main objective to extend the Citation Required Task by correctly predicting the tokens that should be followed by citations. The reader notices that our work does not include the classification of the citation intents. These two tasks are addressed using mask-filling and NER approaches. Mask-Filling Approach. The first solution is based on a mask-filling approach where certain words on the original input are masked and the model has the assignment of predicting appropriate words that can be used to replace the masked ones. To solve the above tasks with this strategy, we adopted a generative approach. In more detail, given a sentence , its tokens = { 0, … , }, and a mask MASK, the mask was moved over the tokens to feed the model. More precisely, the model was iteratively presented with tokens at each iteration where 2 < < . For example, for a sentence 1 with tokens 1 = { 01 , … , 31 }, the model is fed with 1 0 1 1 , , in the first iteration, 1 0 1 1 , , 1 2 , in the second iteration, and 1 0 , 1 1 1 2 , , 1 3 , in the third iteration. When the model predicts a reference placeholder, this is recorded by our system and the next iteration is investigated.
NER Approach. The second approach investigated in this paper is a NER approach that usually classifies entities, such as names of persons, organizations, locations, dates, numerical values, and other relevant information, within a given text. In the context of citation placement prediction, the NER model is set to classify each token of an input sentence into REGULAR and CITATION tokens. The former indicates tokens that should not be followed by a citation. The latter indicates tokens which should be followed by a citation.
For our exploration, we focused on the following models: • GPT-2. The generative methodology uses the text-generation capabilities inherent in transformer models. Among these models, the Generative Pre-trained Transformer (GPT) family by OpenAI stands out as one of the most widely employed. In our approach, we specifically utilized the GPT-2 transformer [ 18]. GPT-2 is a generative language model pre-trained on approximately 40 GB of unlabeled data. Its function is to predict the subsequent token given an input phrase or word. Our choice fell on the variant of GPT-2 equipped with 117 million parameters and 12 layers. This selection was driven by GPT-2’s accessibility, integration within popular deep-learning libraries, and the possibility of running in low-setting machines. • BERT [19]. BERT, short for Bidirectional Encoder Representations from Transformers, is renowned for its encoder-only architecture, featuring stacked layers that refine text representations. Our use of the BERT-base, comprising 12 layers and 110 million parameters, underscores its usefulness in text classification tasks. BERT’s bidirectional processing captures contextual information from surrounding words, enabling a nuanced understanding of language.
This section describes the experimental setting, the evaluation of the models, and our preprocessing steps to further enhance the overall results.
To set the analysis of the models described above it is deemed to exploit models that have been trained on scholarly data. For this reason, two strategies are possible: i) use an already trained model, and ii) fine-tune a pretrained model on newly prepared data. For this, we referred to two datasets. The first dataset on which our analysis is based is arXiv-80 a dataset consisting of plain text containing citations from arXiv. A limitation of this dataset is that it contains citations in diferent formats, a characteristic that might mislead models for the above-mentioned tasks. Therefore, we also created a new dataset s2orc-9k, a dataset containing 9,000 papers about Computer Science from s2orc where citations have a unique format. These datasets were used to create and experiment with the following models: • NER-s2orc, a BERT-based model that has been fine-tuned on the NER task. • ArXiv-NLP GPT-24, a pretrained model on the arXiv-80 dataset with papers from the computational linguistic field. 4https://huggingface.co/lysandre/arxiv-nlp
Finally, we created a gold standard of 170 sentences from papers about Computer Science from the year 2023 that were manually labeled by three senior researchers with more than 100 papers in Computer Science. The annotators were asked to label tokens which must be followed by a citation. The computed inter-annotator agreement was 79.7%. This dataset serves for the evaluation of the Citation Required Task as well as Citation Placement Task. 3.3.2. Results We evaluated the models with precision, recall, and f1-score. While for the Citation Required Task we used standard metrics, for the Citation Placement Task we defined true positives as the correctly predicted tokens immediately preceding a citation, true negatives as the correctly predicted tokens that do not precede a citation, false negatives tokens that precede a citation but erroneously predicted and, finally, false positives as the erroneously predicted tokens that do not actually precede a citation. The results of the three experimented models are reported in Table 1 and Table 2 for the Citation Required Task and Citation Placement Task respectively. Both tables show that the generative approach is more suitable for the proposed task of obtaining a relatively high f1-score. In addition, both tables show that injecting well-formatted features during the fine-tuning of the generative approach further improved the model performance.
In addition, to the evaluation above we performed a manual inspection of the predicted citation placement from the various models. This revealed some error patterns that can be easily solved by some natural language processing (NLP) heuristics that implement common practice in governing paper citations. Here, the reader can find the kind of error and a description of the heuristic to solve it: • Citation placeholder between an acronym and its extended terms. The heuristic to solve this error takes the citation placeholder predicted and moves it after the acronym. • Citation placeholder after Table, Fig., Figure, etc . The heuristic simply removes the citation placeholder appearing after these keywords. • Citation placeholder after a verb . The heuristic labels the tokens of the input text with part-of-speech tags, and removes the predicted citation placeholders appearing after
These heuristics were applied to the results of the models above providing further improvements in solving the Citation Required Task and Citation Placement Task as shown in Table 3 and Table 4. Interestingly, it revealed a downgrading of the performance of the NER approach in terms of precision that nevertheless did not undermine the overall model, resulting in a higher f1-score of 0.162 for the Citation Required Task, and of 0.07 for the Citation Placement Task. For the generative approach, the heuristics increased the models’ performance in terms of precision and recall. This demonstrates that including insights from citation placement governance enhances the overall approach, leading to improved performance.
In this paper we have investigated two tasks for recognizing whether a scientific sentence
needs a citation, and if yes, after which token the citation should be placed. For this, we
have explored the NER and generative approaches obtaining promising results. We have also
provided some heuristics to better shape the output of the used models and incorporate common
practice in citation placement into our approach. However, our work needs to further develop
in a few directions. More precisely, we plan to complete our approach by investigating and
creating components to recommend potential papers that replace the citation placeholder. To
achieve our goal, we intend to utilize knowledge graphs [