We introduce the task of long-tailed detection of model entities from scientific documents. We use distant supervision using an external Knowledge Base (KB) to generate synthetic training data and use a simple entity replacement technique to improve performance significantly by addressing the problem of overfitting in small sized datasets for supervised NER baselines. We introduce strong baselines for this task which are evaluated on our annotated gold standard dataset. We also release the distantly supervised silver labels generated using the KB. We introduce this model as part of a starting point for an end-to-end automated framework to extract relevant model names and link them with their respective cited papers from research documents. We believe this task will serve as an important starting point to map the research landscape in a scalable manner, needing minimal human intervention.
Long tailed entities are named entities which rarely occur in text documents. For these types of entities, the task of Named Entity Recognition (NER) is non-trivial. Recent approaches have aimed at solving the problem of NER using supervised training using deep learning models. However, supervised learning techniques require a large amount of token-level labelled data for NER tasks. Annotating a large number of tokens can be time-consuming, expensive and laborious. For real-life applications, the lack of labelled data has become a bottleneck on adopting deep learning models to NER tasks.
Most scientific named entities can be classified as long-tailed entities because of the rarity
and domain-specificity of their occurrence. Recent work on NER in scientific documents has
been concentrated around detecting biomedical named entities [
NER has traditionally been treated as a sequence labelling problem, using CRF [
To address the above bottlenecks, we present a simple yet efective technique leveraging an external Knowledge Base and a large unlabelled corpora (both of which are cheap and easy to obtain) to generate our training dataset. We believe this simple technique can be easily extended to any other domain given the availability of a domain-specific Knowledge Base and unlabelled text corpora. Utilising this training set, we are able to establish a strong baseline for this task using a standard BERT-CRF model. In order to evaluate our performance for this task, we present a high quality human annotated gold standard evaluation dataset.
Using our trained models, we create an automated framework of detecting model names of
related work from research papers. We define related work as prior research work done by the
scientific community for the same or a similar related task that has been investigated by the
original paper. Our pipeline contains two steps: Firstly, we build a sentence intent classifier that
classifies whether a citation sentence contains information regarding related work or not. Then
we extract model names from the positively labelled sentences using our trained NER model
and link them to their respective citation mentions using a string distance based technique,
introduced by [
In order to create our whole set of gold labels for the evaluation test-set, we randomly sample abstracts from a large set of arxiv Research papers2. We also introduce randomly sampled papers from DBLP citation dataset to add to the diversity of train and test-set selection. 1https://github.com/paperswithcode/paperswithcode-data 2https://www.kaggle.com/Cornell-University/arxiv
Our labeling model was built upon SciBERT (Beltagy et al., 2019), a pre-trained language model based on BERT (Devlin et al., 2019) but trained on a large corpus of scientific text.
There are two models in the transformers, which can handle multilingual posts – multilingual-BERT[19] and XLM-Roberta [57].
For example, KG-BART encoded the graph structure of KGs with knowledge embedding algorithms like TransE (Bordes et al., 2013), and then took the informative entity embeddings as auxiliary input (Liu et al., 2021).
Considering our end goal of automating a high precision framework of extracting related model names and to minimise ambiguity, we consider only named models as model entities for this task. Few examples are - BERT+BiLSTM+CRF, KG-BERT (with overlap), LSTM + Attention, DeepWalk. We consider both single named entities and a combination of multiple model named entities for annotation. A few example sentences with model entities are displayed in Figure 1.
We consider strict span matches for the model entity names, and do not consider any partial matches or synonyms. We also consider plural variants of entity names as matches.
We aim to minimise ambiguity by considering only those model named entities that we can verify about in Google Scholar and Semantic Scholar. We follow the process of identifying a candidate model name and reviewing the existing Computer Science literature to verify whether it is a model name entity or not by identifying its usage in the literature. A simple criteria that we use is to observe if the model(or a variant of the model) has been mentioned in a Results table and compared with baselines/other related models, in previous literature. Only after this thorough review, we annotate a named entity as a model name. We discard any sentence if a model named entity within the sentence does not follow the defining criteria. Hence, we believe we reduce the ambiguity suficiently enough to allow for a single annotator for the entire annotation process. All the annotations has been done by a graduate NLP researcher who is also a co-author of this paper. The overall statistics of the training and test set has been provided in Table 2.
For the unlabelled corpus, we use the arxiv dataset containing 2˜27,000 abstracts from various domains of Computer Science. We use the Papers with Code (PwC) corpus as a reference Knowledge Base to obtain a total of 14,748 model entity names. We use this list of named entities and create a set of distant silver labels by extracting the corresponding sentences out of the arxiv dataset that contain the same entity mention. We aim for exact match while also considering plural forms of the entity words. We obtain a total of 7800 sentences that contain a model named entity mention.
We plot a model entity vs frequency of occurrence in the entire corpus of our obtained sentences. We provide the plot in Figure 2. We notice that the distribution is long-tailed in nature, which is consistent with our hypothesis about scientific named entities as discussed in the Introduction section. This means that there are certain pre-dominant popular models that occur most frequently in the literature. The distribution tapers down and takes a long-tailed form, where most of the entities have a much significant lower number of occurrence in the literature. This can be attributed to the wide-spread use of certain models (like CNN), in the existing Computer Science research literature.
However, such a skewed distribution is unsuitable for training supervised custom NER models. The models tend to memorise and overfit for certain named entities. Hence, we use a simple entity replacement technique to deal with this bottleneck. More specifically, we detect the entity span of the model named entity in the occurring sentence. Then, we replace this entity with another entity from the entire set of model entities obtained from the Knowledge Base. We execute the process keeping the number of entity distributions to atmost 2 to maintain uniformity. After the entire process is completed, the entire train sentences set is set with uniformly distributed entities.
• BiLSTM + CRF: This BiLSTM-CRF model captures the contextual representations and
encodes them into a bidirectional hidden state using BiLSTM. The CRF layer models
the dependency among a sequence of tokens by considering the entire sequence label
probability distribution.
• SciBERT + CRF: This model contains pre-trained SciBERT [
We evaluate the models on the gold test labels. We find that SciBERT model in combination with CRF provides the best performance. We also find that the entity replacement technique is particularly efective when dealing with long tailed entity distributions. We find that this simple technique ofers a significant boost in performance across all models. It is efective in countering the overfitting bottleneck and successfully prevents memorisation of named entities. We rely only on distant labels to obtain strong performance on gold labels. We illustrate our best performing model in Figure 3.
We train a classifier to detect whether a sentence contains relevant information regarding models that solve a similar task as specified in the target research paper. For a target scientific document, we define a relevant model name as a model that the author has cited, which solves a task that is similar or relevant to the original task that the target paper is solving. To create an 0.4 0.9
1.9 0.7 0.1
CRF 0.9 1.3 0.4 Sci-BERT 0.4 0.1 1.9 0.7 0.6 1.6 0.8 0.5 1.3 We present
SDP
LSTM
a
novel
network
automatically labelled dataset, we iterate over all sentences in the research corpora. If a sentence
contains the words - ‘Related Work’ or ‘Previous Work’ or ‘Baseline’, then we take 15 sentences
occurring after it. We assign positive labels to sentences containing model entity mentions by
referring to our KB. We consider the maximal span for entity matching between our unlabelled
text and KB. For creating negative samples, we randomly sample from all sentences and make
sure the above words are absent and it also does not contain model entity mentions. We keep
an equal distribution of positive and negative labels. An example of a positive and negative
label is shown in Figure 4.
5.1. Training the classifier
The most commonly used approach of averaging BERT embeddings or using the output of the
ifrst token (the [CLS] token) yields subpar sentence representations [
The authors have introduced a probabilistic framework based on Hidden Markov Random Fields (HMRFs) for semi-supervised clustering that combines the constraint-based and distance based approaches in a unified framework.
All processing units perform the same computation, specified by equation (1), and are locally connected to their three neighbours. be compared using dot product. It takes a sentence as input and returns the corresponding sentence-level representation as output. We use Sentence-BERT to encode the sentences and use Logistic Regression as our binary classifier to train it on 15,518 labelled sentences, containing both citation and non-citation sentences. Positive and negative samples are equally distributed. The sentence dataset size is kept small to avoid compromising on the quality of the labels. The train-test split followed is 75-25. The testset accuracy (which, again, consists of both citation and non-citation sentences) is 86.41%.
For the entity citation linking, we iterate between all possible extracted entities and citation
combination and get their closeness score, which is the string distance between an entity and
the citation occurrence. We first take all the citations and keep the closest entity per citation.
Then, we take all the entities and keep the closest citations per entity. This linking process is
able to accurately link most of the extracted entities with their closest citations, as demonstrated
by [
We show two end-to-end pipelines in this paper. First, we show the entire training process for both model entity extraction and sentence intent classification. We use the same unlabelled corpora and Knowledge Base(KB) for both training processes. The automatic data labelling
Sentences The optimized 4-layer BiLSTM model was then calibrated and validated for multiple prediction horizons.
Furthermore, case studies show that SIMCLDA can effectively predict candidate lncRNAs for renal cancer.
Longformer's attention mechanism is a drop-in replacement for the standard self-attention.
Bi-LSTM SIMCLDA Longformer
MODEL MODEL MODEL Unlabelled corpora
Sentences Knowledge
Base
The authors have
used...
All processing units...
Entity Replacement The optimized 4-layer BiLSTM model
was then calibrated and ...
The optimized 4-layer TransE model was then calibrated and ...
Weak Labels Distantly labelled TrDaiinstSanetnlytelnacbeeslled TrDaiinstSanetnlytelnacbeeslled Train Sentences Sentence-BERT
Classification Layer Predicted Labels
The authors have used ...
All processing units ...
Positive(+) Negative(-)
B-Model I-Model O O O O
CRF Layer SciBERT Embeddings Distantly Labelled
Training Data process using the external KB followed by entity replacement is shown in Figure 5. This transformed dataset is used as distantly supervised training labels for input to the SciBERT-CRF NER model. Also, the sentence intent classifier approach is illustrated, where the KB is utilised to obtain weak binary labelled sentences from ’Related Work’ section to train the classifier.
For the automated framework, we use a two stage pipeline. It takes a scientific research paper as input and obtains the citation sentences from it. Using the trained sentence intent classifier, we segregate the sentences into positive and negative labels. Only positively labelled sentences are passed into the next stage of the pipeline. We use the trained SciBERT-CRF NER model to extract entity mentions from the sentences. The model mentions are then linked with their respective citations using our Entity-Citation linker. The framework is illustrated in Figure 6.
Citation Sentences
The authors use CNN [
embeddings
The ImdB dataset [
Trained Sentence Intent
Classifier (Sentence-BERT + Binary
Classifier) Predicted Labels (green denotes +, red denotes -)
The authors use
CNN [
Entity-Citation
Linker
The authors use CNN [
BERT [
B-Model
The authors use CNN [
CRF Trained NER Model (Model Name Extractor)
Predicted Entities ( with
citation link)
CNN [
We conduct error analysis for model entity extraction, sentence intent classification and entity citation linking. Some precision error is introduced into the model because for the training set we consider the maximum span of each entity and the I-Model entity occurrence (a token that lies inside a named entity) is high. We find in our evaluation dataset, the number of B-Model entities is massively more, which leads to the model misclassifying an O as an I for few sentences.
Also, due to the usage of citation sentences in the evaluation dataset, our model recognises the citation marker occurring right after the entity as an I-Model. Also, most of the citation sentences in the evaluation dataset has a large number of named entities occurring adjacently, as seen in many citation contexts. The model, which is trained on sentences from abstracts only, is unable to recognise all of them as entities sometimes.
For the sentence intent classification, our classifier often recognises sentences containing dataset names as a positive label. This can be attributed to the fact that citation sentences that refer to diferent datasets often have a similar structure to those citing model names of prior work. Lastly, for the entity citation linker, sometimes an entity that is associated with a citation marker occurs in the initial part of a sentence and its not the closest to the citation. This can lead to missed out or incorrect linking.
We use PyTorch framework to implement our NER model. We use the pre-trained SciBERT tokenizer and embeddings as input to a dropout layer with a dropout probability of 0.5 to prevent overfitting. We use a learning rate of 1e-5 and train all models for 20 epochs. We pass the output from the dropout layer through a linear layer with input dimension same as the hidden dimension of SciBERT embeddings (768) and output dimension same as the number of labels (4). We train the BiLSTM-CRF model for 20 epochs. We annotate the evaluation dataset in the standard CoNLL BIO format. For Sentence-BERT, we use pretrained models available in Pytorch. We use the DBLP corpus consisting of 4˜3K papers as our unlabelled research corpora to obtain the distant labels for training the classifier. For the Knowledge Base (KB), we use PwC public data corpus. For the CRF layer, we use allennlp3 models library. We use regular expressions to extract citation sentences from papers that are written in the Springer LNCS/LNAI format. 10. Conclusion and future work We have introduced a novel task of long-tailed model entity recognition from scientific documents. We test our gold standard evaluation set on multiple baselines. We also find that a simple strategy of entity replacement works well on small labelled datasets for distant supervision. We hope to extend this technique to diferent types of entities with low labelled data availability. We integrate our model in the automated pipeline framework to extract model names from scientific research documents and link them to their respective citations. For future work, we aim to utilise this pipeline on a large research corpus to obtain a map of benchmarked model names linked with their respective papers on a much larger scale. We believe our work will serve as an important starting point for mapping the entire research landscape of computer science.