In this paper, we present participation in the Named Entity Recognition and Classification evaluation organized by CAPITEL at IberLEF 2020. We mainly aim to investigate the eficiency of data augmentation in transfer learning for NER. In addition to the oficial training set, we use Spacy to annotate the Spanish news monolingual corpus to create an augmented dataset. We perform experiments on two sets of data under 4 experimental settings, the experimental result show that pre-training on the augmentation set and then fine-tuning on the oficial set (i.e. cascadingly fine-tuning) could improve the performance compared to fine-tune on any of them separately or mixed.
Fine-tuning a pre-trained language model in downstream tasks has become a standard paradigm for
many NLP tasks like sentiment classification, NER or POS-tagging, because models for these tasks
can be easily designed as the combination of a unified encoder and a task specific classifier. [
Here we choose to use multilingual-BERT [
We use cross-entropy as the loss function. Note that BERT uses WordPiece tokenizer [
As described by the organizers, the CAPITEL dataset is composed of Spanish news and has been
annotated with Person (PER), Location (LOC), Organization (ORG), and Other (OTH). The corpus has
been annotated with the BIOES format [
To create the augmented dataset, 38,000 sentences were sampled from the WMT news translation corpus. Then, we use Spacy to annotate the corpus and create the augmented dataset which contains 11,000 PER, 6,000 LOC, 4,000 ORG and 6,000 OTH. Note that the “MISC" annotation of Spacy is similar to the CAPITEL’s OTH, so it can be directly converted.
Table 1 shows the detail of two datasets employed in experiments, where the cap and aug represents for the CAPITEL and the augmented set respectively. From the table we can see that the distribution of each entity type for CAPITEL and augmented is diferent, which means the augmented set should be used carefully to prevent from introducing such bias.
Basically, our experiments are conducted under four settings: • Cap Fine-tuning on all of the oficial training data. • Aug Fine-tuning on all of the augmented training data. • Mix Fine-tuning on the mixture of 10000 augmented data and all of the oficial data. • Cascade Pre-training on the all of the augmented training data and fine-tuning on the oficial set.
The first experiment can be considered as a strong baseline, which shows that fine-tuning a PLM with in-domain data is already able to achieve an acceptable performance.
The second experiment shows that fine-tuning solely on the augmented set could have a negative impact on the performance. We consider that the distribution of each entity in the CAPITEL and augmented data is diferent which could be a reason. Another reason might comes from the noise of the annotation. Unlike some data augmentation method achieved by corrupting input features which doesn’t introduce bias on the mapping of to , Spacy annotation is not the golden truth, we tried to use the Spacy to tag the CAPITEL training set which achieves 51.18%, 72.03% 57.79%, 50.43% of F1 score for the PER, LOC, ORG and the Macro respectively. This means the augmented data could mislead the model to learn incorrect pattern, thus should be used carefully.
The third experiment aims to evaluate the performance of fine-tuning on the mixture of two set. Unfortunately, same as the second experiment, the noise of the augmented data still brings negative influence on the performance, although could be fixed to a certain extent by the correct pattern in the oficial data, which means that the mixture paradigm only applies to high-quality augmented data.
Despite the performance was unexpected for the second experiment, we decided to continue experiments fine-tuning the trained model with the CAPITEL training set. We can see that further finetuning on the unbiased clean data successfully fix the biased estimation of the noisy augmented data, at the same time, the performance improvement might comes from the more generalized knowledge learned from the correct annotation of the augmented data.
In this paper, we present our work in the CAPITEL NER evaluation and investigate the method of improving performance with transfer learning and data augmentation. We find that cascadingly finetuning a pre-trained model on the augmented set and oficial set could significantly improve the performance. Our submission is based on this strategy and achieves the third place.