Datasets play an important role in data-driven scientific research. With the development of scientific research, more and more new datasets have been constructed. It is important to recognize dataset entities correctly, especially when it comes to unusual long-tail dataset entities. However, it is very dificult to obtain high quality training corpus in named entity recognition. We obtained our data based on a distant supervision method along with two data augmentation methods. We then use a BERT-BiLSTM-CRF model to predict long-tail dataset entity. By applying data augmentation methods, we achieve a highest F1-score of 0.7471.
With the development and rise of artificial intelligence, data-driven
research has become a new paradigm. Therefore, the value of
datasets has been paid more and more attention. Commonly used
datasets are, for example, Wordnet, DBpedia, MovieLens, etc.
Currently, dataset entity recognition research is still in the exploratory
stage, especially when it comes to long-tail entity recognition.
Longtail entities are entities that have a low frequency in the document
collections and usually have no reference in existing Knowledge
Bases[
In this paper, we propose a dataset long-tail entity recognition model based on distant supervision method along with two data enhancement ways to expand the training corpus. We finally achieve a best F1-score of 0.7471 using data augmentation methods.
There are many researchers focusing on domain entity recognition.
Aguilar G et al.[
In this paper, BERT + BiLSTM + CRF model structure is adopted, as shown in figure 1. The training corpus is firstly pre-trained by BERT layer to get the word embedding vectors. Word embedding vectors trained by BERT model contains context, syntax and semantic information, carried by a dynamic vector. In diferent contexts, the embedding for the same word may be diferent, which is capable of carrying the context information of sentence. Then the embedding vectors are fed into the BiLSTM layer. We use vanilla LSTM and two way BiLSTM. Finally, the CRF layer takes the BiLSTM outputs and decode them into labels we need for final annotation. Unlike softmax, CRF can capture context tagging information and improve the final annotation performance.
During training, we feed the ground truth annotations to decoder
and minimize the loss in sequence level. We use adam optimizer
and the learning rate is scheduled using warming-up mechanism
following Vaswani et al. [
The size and diversity of training data can make a huge diference when training a deep learning model. Therefore, to improve Copyright 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). long-tail dataset entity recognition performance, we not only used BERT model to capture potential semantic features of sentences but also adopted two data augmentation methods in NLP field: entity replacement and entity mask.
Entity replacement (see figure 2): the entity words in the original training corpus are randomly replaced with entities that in our dataset entity dictionary but did not appear in the training corpus.
Entity mask (see figure 3): the entity words in the original training corpus are replaced with “unknown words” which are generated randomly.
We randomly take 20%, 50% and 100% of our original training corpus and applied the above two data augmentation methods respectively. Through data augmentation process, see figure 4, we ifnally obtained six more augmented training corpus to train our model.
We collected full-text academic papers from ACL and ACM. We then parsed all the PDF files into XML files using tools developed in our own lab. We used NLTK to segment sentences and 10747988 sentences is obtained. And we created a dataset entity dictionary of 10873 dataset entity words by crawling commonly used datasets in the computer science field from Kaggle and other websites. By applying a distant supervision matching method, we obtained a training dataset with 70313 annotated sentences. We also had human annotators labeled 200 sentences. The dataset entity mentions in the 200 sentences are those that are infrequent and never appear
Training data Precision 0.7201 0.8167 0.8049 0.8156 0.8421 0.7385 0.7209
Recall 0.6293 0.6853 0.6923 0.6503 0.6713 0.6713 0.6503
F1 0.6716 0.7452 0.7444 0.7237 0.7471 0.7033 0.6838 in our 10873 dataset entity dictionary, therefore it is reasonable to regard these entities as long-tail entity.
We conducted total seven experiments including one original corpus training and six data augmentation experiments, see table 1. The experimental results show that the prediction results of the model are greatly improved by adopting data augmentation methods on six experiments. The best F1-score 0.7471 is obtained by using entity mask(20%) and two entity replacement(20%, 50%)also achieve F1-scores above 0.74.
The results demonstrate the usefulness of data augmentation methods on long-tail entity recognition. It also shows that data augmentation doesn’t always work better when we augment more data. We achieved best results on both methods by augmenting only 20% of the training corpus. Therefore, we suggest conducting more experiments when applying data augmentation methods as it may have diferent results when it comes to diferent tasks.
In this paper, a distant supervision method is used to obtain a large number of training data, and data augmentation is used to expand the training data. The model performance in long tail entity recognition is considerably improved by adopting data augmentation mechanism, which has great theoretical and practical value. Data augmentation shows great potential in improving the performance of long tail entity recognition. In the future work, we plan to conduct more experiments and employ methods such as active learning to further improve long-tail dataset entity recognition performance.