These working notes describe the participation of Vicomtech NLP team in the CAPITEL task, which is part of the IberLEF 2020. CAPITEL task included two sub-tasks: Named Entity Recognition and Classification (NERC) and Universal Dependency (UD) Parsing for Spanish news articles. A specific system has been designed for each task based on BERT architectures. Both systems have been tested with diferent settings, where the best ones have been selected for being submitted to the shared task. The resulting systems show robust and competitive systems in both tasks with simple architectures.
(NEs) in Spanish news articles. The following NE categories will be evaluated: Person (PER), Location (LOC), Organization (ORG) and Other (OTH). The metrics used for the evaluation will be Precision, Recall, and F-measure (Micro and Macro average).
Universal Dependency Parsing sub-task aims to challenge participants to apply their systems or solutions to the problem of Universal Dependency parsing of Spanish news articles. The metrics used for the evaluation would be the Unlabelled Attachment Score (UAS) and Labelled Attachment Score (LAS). UAS stands for the percentage of words that have the correct head and LAS, the percentage of words that have the correct head and dependency label.
We have participated in both sub-tasks with two diferent systems making use of simple architectures with BERT at their base.
These working notes are organised as follows: Sect1iodnescribes the systems presented in both tasks, with all the details concerning architecture and training setup. Se2cstihoonws the results obtained in the participation of both tasks and Sect3iodnraws some conclusions and future work.
This section provides a description of the systems that we have developed to participate in CAPITEL’s task. In the face of the widespread success of Transformer-based architectu2r]es [ in virtually all Natural Language Processing (NLP) tasks, Vicomtech has implemented both systems, one that learns to recognise and classify entities and other to establish relations between sentence elements, based on BERT3[]. The first task, NERC, is approached using a traditional sequence-labelling approach relying on BERT contextual representation for words. The second task, syntactic dependency parsing, is also based on BERT at its core, combining the semantic representation of the tokens to detect the syntactic relations among them. We have tried the same architecture with diferent pre-trained BERT models.
The NERC system is a Deep Learning model based on BERT (Bidirectional Encoder Representation of Transformers). The model is the simplest approach in which BERT can be used to perform NERC. It makes use of BERT to encode the input, obtaining a contextual embedding for each input token. These contextual embeddings are the input to a fully connected feed-forward layer that helps to classify each token as one of the possible output tags. Fi1gusrheows a diagram with the architecture.
We have experimented with two diferent pre-trained BERT models in our experiments. On the one hand, we have used the BERT-Base Multilingual Casehdttp(s://github.com/google-research/ bert/blob/master/multilingual.m) dshared by Google. On the other hand, we have used BET4O],[ a BERT-base architecture pre-trained only using Spanish texts.
The dependency parsing system is, again, a Deep Learning model relying on BERT. The model uses BERT to encode the input, obtaining contextual embeddings. Then, a tensor operation is performed over the contextual embeddings to obtain an all-vs-all combination of token vectors. This generates × combined embeddings that represent all the possible token pa irbse,ing the length of the input sequence.
The resulting token-pair representations are then passed to several classification layers to make predictions about the relation between the tokens in each pair. First, the pairs are categorised by a binary classifier that decides whether the tokens that form the pair are connected by a relation (an arc of the dependency tree). The logits resulting from the relation classifier are concatenated with each token-pair embeddings. The resulting representation is passed to a ifnal classification layer to obtain the type of relation for each token pair among the Universal Dependencies types.
Note that this all-vs-all token combination strategy has an exponential computational cost w.r.t. the length of the sequence. This approach could not be applied to full documents. However, the scope of dependency parsing is limited to individual sentences, and since the only operations with the all-vs-all pairs consist of concatenation and a simple matrix multiplication (the feed-forward classification layer) the overall computational cost is feasible.
Figure2 shows a diagram of the described architecture.
Similar to the NERC task, we have experimented with the BERT multilingual pre-trained model, and with its Spanish pre-trained counterpart, BETO.
The input for the task comes already tokenized. However, these tokens are not equivalent to what a BERT model expects. Each pre-trained BERT model needs the tokens as they are obtained after using its own tokenizer. Otherwise the input would be meaningless to the pre-trained model. This poses the additional challenge of keeping the alignment between the resulting tokens and the provided labels.
For that, each original token is retokenized with the corresponding BERT tokenizer. This results in additional tokens, since BERT uses WordPiece tokenization that breaks words into word-pieces (e.g. ”Jim Henson” would become ”Jim Hen ##son”). The provided labels are mapped to the head of each token (i.e. the first piece of a sub-word) and the rest of the subtokens are assigned with an special label ’X’. A mapping indicating the correspondence between head subwords and original tokens is stored, so the original token space can be rebuilt at the end of the process, resulting in a token and label sequences of the same length than the original.
While using BETO, we realised than the special ’[UNK]’ token was being hit too often. This token is the default representation used for out of vocabulary values (OOV). When using more traditional tokenization approaches it is usual to encounter OOVs in NLP tasks, due to the limited size of the whole-word vocabularies. But with modern tokenization approaches like WordPiece or BPE this is less common. We discovered that BETO tokenizer’s WordPiece vocabulary was missing some common punctuation marks like semicolon ’;’, or percentage symbol ’%’. Also, we noticed that any word containing certain diacritic marks, like ’cigüeña’, ’piragüista’ or ’Düsseldorf’, were automatically marked as unknown. The same happened with the words containing the character ’Ç’, rather common in Catalan or French nouns.
Having so many unknown values is inconvenient because that means that all the occurrences of such words will share the same vector representation. The contextual information coming from the surrounding tokens may alleviate the problem, but relevant information is being lost.
In order to deal with these issues we manually added missing punctuation marks and symbols to the BETO vocabulary, using the unused slots of the vocabulary that are reserved for the addition of new words. The newly added symbols would have a randomly initialised value because they were not part of the BETO pre-training, but at least this gives them the chance of learning a meaningful representation during the fine-tuning of the model for the downstream task. The problem with the diacritics was solved replacing the ofending characters by their diacritics-free counterparts, e.g. ’Düsseldorf’ was converted into ’Dusseldorf’.
The gold labels for the NERC task follo wIOaBES tagging scheme, which indicates if a token is at theBeginning, I nside, Outside orEnd of a named entity, or if it is an entity spanning a Single token.
This means that for each given entity type , there are four possible label−s: , − , − or − . Sometimes the model selects a tag that correctly predict the entity type, but it is incorrect with respect the IOBES tagging. Some of these mistakes can be corrected with a simple post-processing step.
A must be followed by a n tag or an tag, otherwise it becomes an d tag. If an tag is followed by a n tag, then the must become an tag, and vice versa.
We have experimented with two diferent pre-trained BERT models as the core for the
semantic representation of the input tokens: BERT-Base Multilingual Cashetdtp(s://github.com/
google-research/bert/blob/master/multilingual).m(mdBERT) and BETO [
The training of the diferent variants was carried out on 2 Nvidia GeForce RTX 2080 GPUs with ∼11GB of memory. We applied the AdamW optimiser5[] with a base learning rate of 2−5 , combined with a linear LR scheduling to warm-up the learning rate during the first 5,000 training steps.
For each trained model, the training monitored the weighted F1-score for the model predictions against the development set (i.e. the entity tags for the NERC system, and the syntactic dependency relations for the dependency parsing system). They were run for a maximum of 500 epochs with an early-stopping patience of 150 epochs. Finally, we chose the model checkpoints that had the best development metrics.
Regarding UD task, Tabl2e shows the top-results of the UD task evaluated on the test set and the results of our training evaluated on the development set. Our system achieves the second position in this task out of four diferent submissions. The results show very similar scores in both metrics, UAS and LAS compared to the winner of the task (0.06 points less in LAS). Again, as for the NERC task, our submitted system is based on BETO instead of mBERT because it achieved better results in the development set.
In these working notes we have described our participation in the CAPITEL shared task, for the two available subtasks: NERC and dependency parsing based on Universal Dependencies (UD). We have presented the deep-learning-based architecture of our systems, which rely on pre-trained BERT models as the base for semantic representation of the texts. We have tried diferent pre-trained BERT models, multilingual-BERT and Spanish-BERT (BETO). Despite the presented systems are simple and domain agnostic they obtain high scores. For the NERC subtask our system is the 4th best performing submission, and our team achieves the 2nd position among five participants. For the UD subtask our system ranks the 2nd achieving only 0.06 points less than the best performing system.
The described systems can almost be considered baselines based on BERT. As future work, we may experiment with other novel transformer architectures, and additional in-domain pre-training or more sophisticated pre-training objectives. We would also experiment with additional layer on top of these basic architectures (from the well-known CRF for NERC to additional self-attention layers). Also, in particular for the NERC model, researching and designing an extensible way of injecting world-knowledge about existing entities would be very interesting. This work has been supported by Vicomtech and partially funded by the project DeepReading (RTI2018-096846-B-C21, MCIU/AEI/FEDER,UE).