Introduction

Information Extraction (IE) is the process of obtaining structured data from sources which can not be interpreted directly by machines, like texts [23]. This is particularly important considering the amount of textual information which is exchanged every minute on the internet [34]. Named Entity Recognition (NER) is the Natural Language Processing (NLP) task which focus on identifying and classifying named entities from this unstructured textual information, making them interpretable and accessible to different communication channels.

When dealing with multiple domains, a NER prediction model needs to be able to handle not only the difference of lexicon between them, but also the difference of morphological features. This adds an additional layer of complexity to this task, requiring a more scalable model to perform well in this challenge.

This paper describes our participation in IberLEF (Iberian Languages Evaluation Forum), Task 1: Named Entity Recognition [31]. We present a system

Related Work

The first Deep Learning architectures to be applied in NER models were based on CNNs [5,32], and later on Recurring Neural Networks (RNN) [9,11,4,17,22]. The reason why Deep Learning models perform well on NLP tasks is because they learn latent features from words, as well as the interactions between them, during the training of specific tasks, such as NER.

Collobert et al. [5] proposed a model based on a Multilayer Perceptron with a convolutional layer, and the following works for NER were mostly based on bidirectional LSTMs, with a few differences between them. Huang et al. [11] used a biLSTM-CRF network with manually selected features, combined with features from SENNA [5] word embeddings. Chiu and Nicols [4] used a biLSTM model without the CRF layer for classification, and had their best results with character level features extracted from a CNN layer, concatenated with SENNA embeddings. Lample et al. [17] and Ma and Hovy [22] used similar approaches based on biLSTM-CRF models, with the difference that [17] used a biLSTM to extract character level features, combined with Word2Vec [24] representations, while [22] used a CNN to extract the character level features, that were combined with GloVe [29] embeddings. These works show that biLSTM-CRF networks became a standard architecture for NER models (as well as for other NLP sequential classification tasks). Following works focused on representation of the words, instead of the actual NER model. Language models have been the primary architecture for contextualized word representations.

Peters et al. [30], Devlin et al. [7] and Akbik et al. [1] developed different architectures for contextual word representations based on bidirectional language models and evaluated their performance on the NER task (as well as on other NLP tasks). Both papers, [30] and [1] used a biLSTM-CRF baseline NER model for evaluating their representation models, while [7] evaluated his model by adding a neural layer to the language model, performing the NER classification with it. The ELMo (Embeddings from Language Model) representations from [30] are provided by the biLM language model, which is based on 2 biLSTM networks, with 2 layers each, and the model's input is a character level representation provided by a CNN network. In another way, [7] created BERT, a language model based on the Transformer [36] architecture, which is based only on the neural mechanism of attention. The author from [1] created a language model on character level, in a way that his objective was not to predict words, but characters. The architecture of his CharLM model is also based on a biLSTM network. Table 1 lists the models presented on this section with their respective F-Score performance on the English benchmark from CoNLL-2003 [35].

For Portuguese language, the first work that used a Deep Learning approach was from Dos Santos and Guimarães [32], who adapted the architecture from [5] Work Benchmark F-Score Year Akbik et al. [1] CoNLL [6] HAREM-Tot 69.14% 2018 Chiu and Nichols [4] CoNLL-2003 91.62% 2016 Ma and Hovy [22] CoNLL-2003 91.21% 2016 Lample et al. [17] CoNLL-2003 90.94% 2016 Huang et al. [11] CoNLL-2003 90.10% 2015 Dos Santos e Guimarães [32] HAREM-Sel 71.23% 2018 HAREM-Tot 65.41% Collobert et al. [5] CoNLL-2003 89.59% 2011 Table 1. NER models using Deep Learning architectures for English and Portuguese languages, both evaluated using the CoNLL script [35]. The English language results are reported on the CoNLL-2003 [35] benchmark, and the Portuguese ones are reported on the HAREM [33] benchmark. and proposed CharWNN. For this work, besides using character level features from CNN, the authors also used word embeddings that were pre-trained using the Word2Vec tool [38]. Da Costa and Paetzold [6] and Quinta de Castro et al. [3] used a BiLSTM-CRF architecture with minor differences between them. [6] concatenated character level features from a BiLSTM network with FastText [13] word embeddings, prior to passing this concatenation through another BiLSTM network. [3] used a similar approach from [17] and concatenated the character level features from a BiLSTM network with the representations of a second BiLSTM, which processed pre-trained Wang2Vec [20] embeddings.

Proposed Model

In this work, we propose a system based on different deep learning architectures, similar to that was used by [30]: a Bidirectional Long Short-Term Memory (BiL-STM) [10] NER model with a Conditional Random Fields (CRF) [16] sequential classifer; fed by the contextual word representations from an ELMo [30] language model, combined with character level representations from a Convolutional Neural Network (CNN) [8,18]. Our system differs from [30] in the way that we do not use pre-trained word embeddings, and we use two different ELMo models, one for the general domain of Portuguese language, and one for the police domain.

The ELMo embeddings are obtained using the biLM (bidirectional Language Model) [30] architecture. This architecture is based on 2 BiLSTM networks, each of them responsible for one direction in the bidirectional language model: one for keeping a representation while making predictions in the forward direction of the text and one for the reverse direction. The first layer from the biLM model produces character level features from the training words using two CNNs, one for each direction of the text, each of them with 2048 convolutional filters. They produce a representation with a total dimension of 4096, which is fed to the first BiLSTM layer of the biLM model. Each layer of the model (the CNN and the two BiLSTMs) projects the input it receives to a vector of dimension 1024. These 3 projections represent the ELMo embeddings which are produced by the biLM model. The size of the biLM training vocabulary determines the amount of words that will be predicted in the Softmax layer of the model, as shown in figure 1. The BiLSTM-CRF architecture used in this work is the same from the Al-lenNLP framework [2], following a parameterization similar to the one described in [30] for the NER task. The CNN network used for producing character level features from words used embeddings with dimension 16 and 128 convolutional filters of size 3, with the ReLU [12,26] activation function. The BiLSTM network used for encoding the words has 2 layers, with 200 hidden units each. Figure 2 shows the dimensionality of the word representations obtained from the CNN and the 2 ELMo embeddings used. The 2 ELMo we use were trained in two separate domains: for the general Portuguese domain we used a Portuguese Wikipedia [37] dump, and for the police domain we used a 1.6 billion word corpus created from public documents from Brazil's Labor Courts [15]. The Portuguese ELMo model we trained is publicly available at https://allennlp.org/elmo. For the Iber-LEF evaluation, we performed the fine tuning of this ELMo in this combined dataset, following [30].

Experimental Setup and Results

For the Portuguese NER task, IberLEF specified the evaluation of models in three different domains: general, police and clinical. For the specific domains only person names (PER category) are annotated, while the general domain dataset is annotated with 5 different categories: person, place (PLC), organization (ORG), value (VAL) and time (TME). The following public corpora were used for the model proposed in this work: WikiNER [27], LeNER-Br [21], HAREM I [33] and MiniHAREM [28] golden collections, and Paramopama [14]. We also used a private legal corpus provided by the Datalawyer company, consisting of 76 annotated documents from the Brazilian Labor Court. The only dataset annotated with all five categories is HAREM. These corpora have the following categories annotated in them:

- Since only the HAREM datasets contains all the categories needed for the IberLEF evaluation, we adopted a semi-supervised approach training for an initial NER model to perform the self-training of the final model. This training had the following procedure:

1. For each one of the datasets, we ignored all the entities that were not annotated as one of the 5 relevant categories for this evaluation. Their annotation was removed; 2. We merged the datasets from HAREM, LeNER None of the existing annotations was removed or overriden during the bootstrapping of the datasets. Only words that prior to this process had no category associated to them were classified as either Time or Value, according to the bootstrap model.

Models Evaluation

Prior to submitting the NER model with word representations from 2 ELMo and a CNN (henceforth referred to as 2xELMo+CNN), we performed the training of two other models, with different types of word representation: (i) ELMo+CNN and (ii) ELMo+CNN+Wang2Vec [19]. These two models use only the general domain ELMo. We performed the training of these three models using the same configuration, and performed an additional evaluation of them in the following datasets: MiniHAREM, test datasets from Datalawyer Company and LeNER-Br, and the full datasets from Paramopama and WikiNER. For all of them, except MiniHAREM, we evaluated both variants: with and without bootstrapped Time and Value entities. The best model with the best F-Score was ELMo+CNN+Wang2Vec, followed by 2xELMo+CNN.

We also evaluated the three models in all nine datasets (MiniHAREM, Datalawyer, LeNER-Br, Paramopama and WikiNER, with these last four being evaluated in the original dataset, and the bootstrapped dataset). The 2xELMo+CNN had the best results for the MiniHAREM dataset, as well as for the datasets in the police domain (Datalawyer and LeNER-Br datasets). ELMo+CNN had the best results for Paramopama and WikiNER. After grouping these evaluation results by model, the best mean F-Score was from the 2xELMo+CNN variant. Since 2xELMo+CNN performed better in the police domain (which is relevant for the IberLEF evaluation), we chose this model for the task evaluation.

Table 2 presents the results obtained from the IberLEF evaluation. We point out that the only corpus we did not use from HAREM to train our models was the one from HAREM II [25], which is the one used in the general domain evaluation. We also did not have any access to any type of clinical documents or embeddings, so our model contained no type of adaptation for this specific domain.

Corpus

Concluding Remarks

For the Portuguese NER task of the Iberian Languages Evaluation Forum, we experimented with different systems based on deep learning architectures, for both NER model and word representations. For the NER model we used the BiLSTM-CRF architecture, which became a reference for sequential classification NLP tasks. For word representations we experimented with character level features from Convolutional Neural Networks, Wang2Vec pre-trained word embeddings, and the ELMo embeddings from a biLM language model. We evaluated different models with different types of word representations in 5 different corpora, and submitted a system based on 2 different ELMo, combined with character level features. Our model was trained in a semi-supervised scenario, in order to account for the lack of certain types of categories in the used corpora.

Our main contribution is the use of ELMo embeddings for the Portuguese NER task, which have not been reported so far in the related literature. Our pre-trained ELMo model is publicly available at https://allennlp.org/elmo.

For future work, instead of training a single NER model with different ELMo representations for different domains, we will experiment with an ensemble of different models, each one trained separately in a different domain.