Introduction

Named Entity Recognition (NER) is a widely studied Natural Language Processing (NLP) task. Briefly, the task involves annotating any mentions of entities (usually proper names) occurring in running text. The most common annotated corpora for NER focuses on four types of named entities: Locations, Organizations, Persons and Other (Miscellaneous) entities. Spanish NER has been well-studied, as it was one of the languages proposed in the CoNLL NER shared tasks [1,2].

As for many other NLP tasks, current best performing models for NER are those based on large pre-trained language models which allow to build rich representations of text based on contextual word embeddings. These approaches are based on character-based models like Flair [3] or masked language models like BERT [4]. Furthermore, multilingual versions of these models have been trained: the multilingual version of BERT [4] was trained for 104 languages. More recently, XLM-RoBERTa [5] was trained for 100 languages.

These publicly available deep learning multilingual models for text excel in tasks involving highresourced languages such as English, but their performance drops when applied to low-resource languages [6]. This may occur for a number of reasons. First, each language has to share the quota of substrings and parameters with the rest of the languages represented in the pre-trained multilingual model. As the quota of substrings partially depends on corpus size, this means that larger languages such as English or Spanish are better represented than lower resourced languages such as Basque [6]. Moreover, multilingual models also seem to behave better for structurally similar languages [7].

In our submission for the CAPITEL 2020 NER task [8] we leverage both these multilingual models as well as other monolingual models trained specifically for Spanish. Furthermore, we project the annotations provided by each system into a target final prediction. The projection of several source annotations into a target is loosely inspired by a method originally designed for projection of annotations across languages [9]. Our projection method indicates that substantial gains in performance (around 1.3 points in F1 score) can be obtained by projecting annotations from various heterogeneous sources into a final target prediction. Our submission obtained the best score, substantially outperforming other participants of the CAPITEL 2020 NER task.

Related Work

Deep learning methods in NLP rely on the ability to represent words as continuous vectors on a low dimensional space, called word embeddings. The first approaches generated static word embeddings [10,11], namely, they provided a unique vector-based representation for a given word independently of the context in which the word occurs. This means that polysemy cannot be represented. Thus, if we consider the word 'bank', static word embedding approaches will generate only one vector representation even though such word may have different senses, namely, 'financial institution', 'bench', etc.

In order to address this problem, contextual word embeddings were proposed. The idea is to be able to generate different word representations according to the context in which the word appears. Currently there are many approaches to generate such contextual word representations, but we will focus on those that have had a direct impact, in terms of performance, for the Named Entity Recognition task. First, Flair [3] representations are built following a LSTM-based architecture and trained as language models. Second, the models based on the transformer architecture [12] and of which BERT is perhaps the most popular example [4].

The multilingual counterpart of BERT, called mBERT, is a single language model pre-trained from corpora in more than 100 languages. Another standout model is XLM-RoBERTa [5] also based on the transformer architecture which provides a pre-trained language model for 100 languages trained on 2.5 TB of Common Crawl text. Both mBERT and XLM-RoBERTa enable to perform transfer knowledge across languages [13,14,7], although in this paper we will use them in a monolingual setting for Spanish NER.

Flair

Flair refers to a system based on a BiLSTM architecture [15] and to a specific type of characterbased contextual word embeddings. Flair (embeddings and system) have been successfully applied to sequence labeling tasks obtaining state-of-the-art results for a number of Named Entity Recognition (NER) and Part-of-Speech tagging benchmarks [3].

Flair embeddings consist of sequences of characters. More specifically, sentences are processed into sequences of characters and feed into a character-level Long short-term memory (LSTM) model. For each sentence, a forward LSTM language model processes the its sequence of characters from the beginning of the sentence to the last character of the word we are modeling. Furthermore, a backward LSTM performs the same operation going from the end of the sentence up to the first character of the word. The extracted hidden states contain information propagated from the end and the beginning of the sentence up to the first and the last character of the target word. Finally, the resulting two hidden states are concatenated to generate the final embedding.

Pooled embeddings are a type of Flair embeddings which consider global information in order to generate the final word embedding [16]. In this approach embeddings are kept into a memory which is later used in a pooling operation to obtain a global word representation. This representation will be the concatenation of all the local Flair contextualized embeddings obtained for a given word. It should be consider that pooling operation is involved in the process of fine-tuning the Flair pre-trained models, not in the process of training the language models themselves. We use the default pooling operation, min, which computes a vector of all element-wise minimum values [16].

Transformers

LSTM-based language models such as the one presented in the previous section cannot capture longrange sequence information. Furthermore, they are quite hard to train at a large scale (see [17], especially Figure 7). In order to address these issues, the Transformer architecture was proposed [12], based on multi-headed self-attention and positional encoding. The most popular Transformer is BERT [4], which pre-trains a Transformer encoder on the Masked Language Model (MLM) and Next Sentence Prediction (NSP) tasks. BERT is composed by stacked layers of Transformer encoders [12]. More specifically, in this paper we will use the BERT BASE configuration which contains 12 Transformer encoder layers, a hidden size of 768 and 12 self-attention heads for a total of 110M parameters.

The MLM task is designed as follows: For a input sequence of 𝑛 tokens 𝑥 1 , 𝑥 2 , ..., 𝑥 𝑛 , 15% are selected as masking candidates. From those candidates, 80% of them are masked (they are replaced with the [MASK] token), 10% are replaced by a random word and the last 10% is left unchanged. For the NSP task, two segments are selected from the training corpus, 𝐴 and 𝐵. In 50% of the cases 𝐵 is the true next segment for 𝐴. For the rest, 𝐵 is just a random segment. The model is trained to optimize the sum of the means of the MLM and NSP likelihoods.

It should be noted that the benefits of the NSP task during the pre-training process has been questioned [18,19,20]. Thus, other transformer proposals such as RoBERTa train without the NSP task, showing strong performance on the same downstream tasks.

XLM-RoBERTa relies exclusively on the MLM objective. The biggest update that XLM-Roberta offers is a significantly increased amount of training data, 2.5TB of Common Crawl clean data [5]. As for BERT, in this paper we use the base version of XLM-RoBERTa. The reason being that their base versions fit for fine-tuning into a standard GPU card with 12GB of RAM.

Experimental Setup

Named entities were originally annotated using the BIO encoding which identifies the Beginning, the Inside and the Outside of named entities. Later on the BILOU model1 was proposed to mark tokens as the Beginning, the Inside and the Last tokens of multi-token entities as well as Unit-length entities [21]. Although the CAPITEL corpus is originally released using the BILOU model, we experiment with both type of encodings.

The CAPITEL (Corpus del Plan de Impulso a las Tecnologías del Lenguaje) has been developed by the PlanTL, the Royal Spanish Academy (RAE) and the Secretariat of State for Digital Advancement (SEAD) of the Ministry of Economy. These organizations signed an agreement for developing a linguistically annotated corpus of Spanish news articles, with the objective of extending the language resource infrastructure for the Spanish language. CAPITEL is composed of contemporary news articles and contains annotations for Universal Dependencies and Named Entities. The NER portion of the corpus contains around one million words.

For the experiments performed for this paper, we use a number of publicly available models: Additionally we trained the following monolingual language models for Spanish:

1. Flair-GW: Flair character-based language model trained on the Spanish Wikipedia and the Gigaword 3rd edition corpus, containing around 11GB of text. 2. Flair-Oscar: Flair language model trained on the OSCAR Spanish corpus [23], which contains 157GB of Common Crawl text cleaned and deduplicated.

The Flair embeddings for Flair-GW and Flair-Oscar were trained with the following parameters: Hidden size 2048, sequence length of 250, and a mini-batch size of 100. The rest of the parameters were left in their default setting. For Flair-GW, training was done for 5 epochs over the full training corpus. The training took around 5 days in a Nvidia Titan V GPU. With respect to Flair-Oscar, only one epoch was performed, requiring around a month to complete it.

Results

Table 1 reports only the best results obtained during the experimentation. Each of the S1-S8 results is the average of five randomly initialized runs. Flair models were trained using the default parameters, although we experimented adding FastText embeddings to the Flair and Pooled embeddings. We used 10 percent of the training data for development of the Flair models. In the case of the Transformer models described in the previous section, we used the full training set for hyperparameter fine-tuning. For XLM-RoBERTa we used a maximum sequence length of 128, mini-batch 16, 5e-5 learning rate, and 4 epochs. For mBERT and BETO best results were obtained using the same hyperparameters as for XLM-RoBERTa but increasing the sequence length to 256.

Out of the many experiments performed with the three Flair language models (Official, GW and Oscar), the best performing language model in every possible configuration was the Flair-Oscar model combined with the FastText embeddings trained on Wikipedia. In fact, Flair-Oscar was the best single system by a substantial margin. Apart from this, S2 and S3 show the small gains obtained by adding the 10 percent used for development for the final evaluation. Furthermore, S3 was trained when the progress of training the language model was at half epoch, whereas S4 was trained using the final Oscar language model based on one epoch. Finally, S5 is the same model as S1 but using BIO encoding instead of the original BILOU encoding from the CAPITEL corpus. The best overall invididual system was S4, significantly outperforming the multilingual and monolingual Transformer models.

With respect to the transformer models, it can be seen that in general their results are lower than those obtained by the Flair-Oscar models. During the development phase they all performed very closely although in the final, official results XLM-RoBERTa was slightly superior to the rest. Furthermore, results also show that mBERT performed worst and that XLM-RoBERTa obtains very similar results to the monolingual models.

The last three rows of Table 1 report the three best projections. Once we had the best 8 systems, we proceeded to project their predictions by means of any possible combination of the 8 systems. The best three systems were picked based on two criteria: the F1 score obtained on the development data and the number of No-agreements recorded by each projection.

The projections were performed using 5 predictions as source. We tested various strategies and the one we finally used to report the final results was, interestingly enough, the simplest of them all. It uses a very simple methodology based on the number of agreements between the predicted labels of the 5 source annotations: if agreement is >= 3 then project, otherwise, project "O".

As we could not compute F1 scores on the official test set released by the shared task, we simply picked the projection which recorded fewer No-agreements. This corresponds to the best overall system (P3), which uses S3, S4, S6, S7 and S8 as source to obtain the final prediction.

Concluding Remarks

In this paper we have described the experiments performed for our participation in the CAPITEL 2020 shared task on Named Entity Recognition. Even though the best results are obtained by the Flair-Oscar monolingual models, our results indicate that multilingual pre-trained models such as XLM-RoBERTa are performing increasingly close to monolingual models for a large-resourced language such as Spanish. Furthermore, we also show the benefits of projecting named entity annotations from various heterogeneous sources in order to substantially improve performance (around 1.3 points in F1 score over the best individual system).