These working notes describe the two Named Entity Recognition (NER) systems designed by Team Recognai for the CANTEMIST (CANcer TExt Mining Shared Task - tumor named entity recognition) NER track. While the first system tries to maximise the performance with respect to the F1-score, the second system tries to maximise its eficiency with respect to model size and speed while maintaining acceptable performance.
Our data preprocessing was minimal and consisted of two major steps:
1. As a first step we transformed the given brat annotations2of the train, dev1 and dev2
data sets to commonly used BILOU tags[
We used the same spaCy sentence splitter for splitting the test and background data into sentences and feed them to our models to obtain the submitted predictions.
No data augmentation or external data was used for the training of our systems.
The goal of this system was to maximise its performance with respect to the F1-score.
To achieve the goal of maximal performance, we use a pretrained transformer-based masked
language model provided by the Huggingface Transformers library[
The XLM-RoBERTa[
We introduce this model in our embedding layer to obtain contextualized word embeddings. Since XLM-R applies subword tokenization, we simply sum up the subword vectors when necessary to end up with embeddings at word level.
To facilitate the model’s identification of words that are likely entities regardless their context (like technical names of cancer types), we extend our embeddings with character features. Another reason to add character features in general, is to make the model more robust against typographical errors.
The character feature consists of the last hidden outputs of a bidirectional Gated Recurrent Unit (GRU) that is fed with the characters of the respective word.
The stacked embeddings are then passed on to the bidirectional LSTM layer in which we seek after a task specific contextualization of our embeddings.
The hidden states of the LSTM layer are finally fed into a token classification head. This head consists of a linear transformation of the hidden dimension of the LSTM layer to the number of 1https://www.recogn.ai/biome-text 2http://brat.nlplab.org 3https://spacy.io possible BILOU tags, and a subsequent Conditional Random Field (CRF) model that predicts the sequence of BILOU tags for the input.
The single components of the system and there approximate sizes in terms of number of parameters are summarised in Table 1.
Experimenting and optimisation of some hyperparameters were done with the train and dev1 data set. We used the AdamW algorithm implemented in Pytorch for optimising all parameters of our model with a learning rate of 10−5.
For an estimation of the model performance we used the train and dev1 data set as training data and the dev2 set as validation data. For the final submitted predictions we trained our model on the combined set of train, dev1 and dev2. We stopped the final training after 10 epochs, a number estimated from previous training runs, to prevent overfitting.
The goal of this system was to maximise the model’s eficiency with respect to the model size and speed while maintaining acceptable performance.
In contrast to our XLM-R system, we opted for a more light-weight solution regarding the
pretrained component. For this reason we chose the pretrained Spanish word vectors provided
by FastText[
To assure a well generalised word vocabulary, we only add words to it that appear at least 2 times in our training data set.
Furthermore we add character features to our embeddings to make our model more robust against typos. The character feature consists of the last hidden outputs of a bidirectional GRU that is fed with the characters of the respective word.
The stacked embeddings are then passed on to the bidirectional LSTM layer in which we seek after the contextualization of our embeddings.
The hidden states of the LSTM layer are finally fed into a token classification head. This head consists of a linear transformation of the hidden dimension of the LSTM layer to the number of possible BILOU tags, and a subsequent CRF model that predicts the sequence of BILOU tags for the input.
The single components of the system and there approximate sizes in terms of number of parameters are summarised in Table 2.
Experimenting and optimisation of hyperparameters were done with the train and dev1 data set
and performing Hyperparameter Optimization of both architecture parameters (e.g., encoder
hidden sizes) and training hyperparameters (e.g., learning rate). For hyperparameter
optimization, we used the integration of biome.text with the Ray Tune library[
For an estimation of the model performance we used the train and dev1 data set as training data and the dev2 set as validation data. For the final submitted predictions we trained our model on the combined set of train, dev1 and dev2. We stopped the final training after 4 epochs, a number estimated from previous training runs, to prevent overfitting.
This work was supported by the Spanish Ministerio de Ciencia, Inonvacion y Universidades through its Ayuda para contratos Torres Quevedo 2018 program with the reference number PTQ2018-009909.
A GitHub repository was created at https://github.com/recognai/cantemist-ner that contains the data sets as well as the data preparation, training and evaluation notebooks.