The CANTEMIST track included three subtasks for the automatic assignment of codes related with tumour morphology entities to Spanish health-related documents: CANTEMIST-NER, CANTEMISTNORM and CANTEMIST-CODING. For CANTEMIST-NER, we trained Spanish biomedical Flair embeddings on PubMed abstracts and then trained a BiLSTM+CRF Named Entity Recognition tagger on the CANTEMIST corpus using the trained embeddings. For CANTEMIST-NORM, we adapted a graph-based model that uses the Personalized PageRank algorithm to rank the eCIE-O-3.1 candidates for each entity mention. As for CANTEMIST-CODING, we adapted X-Transformer, a state-of-the-art deep learning Extreme Multi-Label Classification algorithm, to classify the clinical cases with a ranked list of eCIEO-3.1 terms in a multilingual and biomedical panorama. The results obtained were a F1-score of 0.749 and 0.069 for the CANTEMIST-NER and the CANTEMIST-NORM subtasks, respectively, and our best scoring submission achieved a MAP score of 0.506 in the CANTEMIST-CODING subtask.
There are several benefits arising from the application of Natural Language Processing (NLP)/Text
Mining approaches to clinical text, like for example, the improvement of the decision-making
process in clinical context. The use of electronic health records is associated with less
doctorpatient interaction [
In the present work, we describe the participation of the LasigeBioTM team in CANTEMIST
(“CANcer TExt Mining Shared Task – tumor named entity recognition") competition [
O-3.1 codes for each document.
For the CANTEMIST-NER substask we used the Flair framework [
In the Named Entity Recognition (NER) task, state-of-the-art approaches usually have a
BiLSTMCRF architecture, which was initially proposed by Huang et al. [
The state-of-the-art approaches in Entity Normalization (also called Disambiguation or
Linking) include graph-based models [
As to XMLC, several machine learning solutions have been developed in the last decade [
Our team has already made an adaptation of X-Transformer for the biomedical panorama in the BioASQ MESINESP competition that occurred earlier this year. In MESINESP, the goal was indexing a large dataset of biomedical articles written in Spanish using DeCS terms. In the ifnal scoreboard, our approach using X-Transformer has achieved high scores in the precision measures, surpassing most competing systems in those measures.
The goal of this task was to recognise and locate tumour morphology entities in Spanish
healthrelated documents. We used the Flair framework [
We trained new Flair contextualised embeddings [
The 4 splits contained a total of 130,000 articles and 155,366,645 tokens, of which 143,387,385 corresponded to training tokens.
We generated a language model for each split with hidden_size = 1024, nlayers = 1, dropout = 0.1, using the following training parameters:
We trained forward and backward embeddings in each split using a single NVIDIA Tesla P4 GPU, since Flair does not have multi-GPU support in the current version, and interrupted the training after a variable number of epochs that ranged from 71 in the backward embeddings training in split 1 and 99 in the backward embeddings training in split 2.
We converted the train, development 1 (dev1) and development 2 (dev2) sets of the CANTEMIST corpus2 to the IOB2 format3 using the Flair sentence segmenter jointly with the Flair tokenizer. Each token was tagged with the label “B-MOR_NEO" if corresponded to the beginning of an annotation, the label “I-MOR_NEO" if corresponded to the inside of an annotation, and the label “O" if it was outside of any annotation. The content present in the train, dev1 and dev2 sets originated, respectively, the files “train.txt", “dev.txt" and “test.txt". The corpus was then loaded into a Flair “ColumnCorpus" object to allow the further training of the NER tagger. 2https://temu.bsc.es/cantemist/?p=4338 3https://en.wikipedia.org/wiki/Inside%E2%80%93outside%E2%80%93beginning_(tagging)
1. “base": uses Flair embeddings (es-forward and es-backward) trained on Spanish Wikipedia + Spanish FastText embeddings. 2. “medium": uses Flair embeddings (es-forward and es-backward) trained on Spanish Wikipedia + Spanish FastText embeddings + PubMed Flair embeddings trained in 1 PubMed split. 3. “large": uses Flair embeddings (es-forward and es-backward) trained on Spanish Wikipedia + Spanish FastText embeddings + PubMed Flair embeddings trained in 2 PubMed splits. 4. “pubmed": uses PubMed Flair embeddings trained in 4 PubMed splits.
We considered the default architecture for the sequence tagger: BiLSTM with a CRF decoding
layer, hidden_size = 256. The training parameters were set to:
• learning_rate = 0.1
• mini_batch_size = 32
• max_epochs = 55
• patience = 3
Due to the lack of time, we only selected the “medium" model for training, as we considered
it was the safest approach: to leverage trained biomedical embeddings jointly with general
available embeddings. After the training, we applied the model to predict the labels in the 5232
documents belonging to the background + test set of the CANTEMIST corpus and to create an
annotation file in the BRAT format for each text document.
The goal of this task was to perform NER and the normalization or disambiguation of the
recognised entities to the eCIE-O-3.1 terminology. We applied the model previously developed
for the NER task to recognise the entities and adapted the PPR-SSM model [
In each document, the first step was to apply the NER tagger to generate the NER output files
and then to retrieve the ten best eCIE-O-3.1 candidates for each recognised entity through
string matching, more concretely, according to the edit distance. The model then built a
disambiguation graph with the eCIE-O-3.1 candidates for all present entities in the document.
Two candidates/nodes were considered linked in the graph if they were linked in the eCIE-O-3.1
hierarchy. For each candidate was calculated the extrinsic information content (IC), which is a
measure of rareness: the IC of a given entity is high if that entity has few entries in an external
dataset [
The model applied the Personalized PageRank (PPR) algorithm over each disambiguation graph.
PPR is a variation of PageRank [
We also explored the use of more than one terminology in the candidate retrieval and graph
building. We considered the CIE-10-ES (“Classificación Internacional de Enfermedades 10. ª
Revisión, Modificación Clínica" 4) and the Spanish DeCS (“Descriptores en Ciencias de la Salud")5
terminologies. For each entity, besides the ten best eCIE-O-3.1 candidates, we also retrieved the
ifve best CIE-10-ES and the five best DeCS candidates through string matching and built the
disambiguation graph accordingly. We considered that a eCIE-O-3.1 candidate and a CIE-10-ES
or a DeCS candidate were linked if they were present in the same sentence of any document
belonging to the CANTEMIST corpus. We applied the python implementation6 of MER [
The goal of this task was the classification of clinical cases in Spanish by returning a list of ranked eCIE-O codes related to the content of each clinical case. To tackle this challenge, we decided to use X-Transformer, a state-of-the-art deep learning XMLC solution and apply it to multilingual biomedical panorama.
Some modifications in the algorithm code were required. The first one was made in the vectorization of the labels of the training and test sets. We have chosen to use all possible labels, including the labels that were not present in the train or test sets. This change was needed since the algorithm would fail to work correctly if the number of labels between sets did not match.
Another modification was the inclusion of BETO [
A pipeline was developed for this task as it can be seen in Figure 1. After retrieving the data from the competition organisers, the first step of this pipeline was merging each separate text ifle that composed the training and development datasets into a single file for each dataset, so that in the end we could have only two files, one for train and another for test. Then, using the ’.ann’ files given for the other two tasks of the CANTEMIST competition and that were associated with each clinical case, we extracted all labels that were attributed to each document and appended them to the beginning of the corresponding clinical case separated by a tab character (’\t’). This way, X-Transformer could distinguish between the labels and the text. The text was then stemmed using a Snowball stemmer8.
The next step was creating a vocabulary file containing all labels used in the datasets, which was required as input by X-Transformer. Each line of this file had an eCIE-O code and its corresponding internal identifier, which corresponds to a number from 0 to N, where N is the total number of eCIE-O codes minus 1. This internal identifier is a label standardization method that allows X-Transformer to classify the text using labels from any kind or domain. For the creation of this file, we used a file containing a list of valid eCIE-O codes that was provided by the competition in their evaluation scripts folder, from which we retrieved all codes and corresponding descriptions. Then, we included the eCIE-O codes descriptions that were present in the ’.ann’ files and that were not present in the list of codes retrieved, adding them to existing descriptions if they had diferences. Then, the codes without any description were removed, since they would be of no use to classify the clinical cases using X-Transformer. In the end, our vocabulary file consisted of a total of 4360 eCIE-O codes.
7https://github.com/dccuchile/beto 8https://www.nltk.org/_modules/nltk/stem/snowball.html
In addition, we have also created a label mapping file that contained the correspondence between the eCIE-O code and its numeric identifier in the vocabulary file. For example, the term ‘Células tumorales benignas’, which has the corresponding eCIE-O code ‘8001/0’, is the eleventh element in the vocabulary file, thus it’s numeric identifier will be ‘10’. This label mapping file will later be used to map the predictions from their numeric identifiers to the corresponding eCIE-O codes required for the task.
The results of X-Transformer are given in the form of sparse matrices, with a number of rows equal to the number of clinical cases that compose the test set, and the number of columns corresponding to the possible labels. The prediction for each clinical case was retrieved, comprising a top K of most relevant labels and their confidence values ranging from -0.99 to 1. We used K=20 labels per clinical case. Then, using a Python script, we converted the predicted labels from their numeric identifiers to their corresponding eCIE-O codes using the label mapping file previously created. The script also discarded each label with a confidence score under a threshold chosen to achieve the highest Precision, Recall or F1 scores. For each of these measures, a ’.tsv’ file was created with the predictions for each clinical case in the format required by the competition. A fourth ’.tsv’ file was also created for the score threshold equal to 0, which was used as a baseline score. In the end, the files were used as input for the evaluation script given by the CANTEMIST competition so that we could retrieve the Mean Average Precision (MAP) scores for the predictions of our models.
In the test and background sets given by competition organizers, there were no eCIE-O codes indexing the clinical cases, so we had to put a placeholder label on each document, because X-Transformer was not prepared to run on unlabelled data. We also had to artificially adapt the size of the given test and background sets by splitting the sets into a total of 48 smaller sets of 250 clinical cases each. This procedure was necessary so that the files could have the same size as the test sets used on the trained X-Transformer models, which had a total of 250 articles. The first 109 lines of each of those files was composed by 109 clinical cases from the test and background sets to classify, and the remaining 141 came from the dev1 set, which was already classified with eCIE-O codes. These 141 clinical cases were used as an additional validation set to define the confidence threshold values of our submissions.
In a first iteration, we trained 4 models using the 501 indexed clinical cases that composed the train set, and the dev2 set that comprised a total of 250 indexed clinical cases, was used as test set. One of our models was trained using BERT Base Multilingual Cased and another was trained using BETO, the Spanish version of BERT.
The other two models were trained with two X-Transformer models that were previously
developed by us for the Spanish biomedical domain using biomedical articles in Spanish retrieved
from the IBECS, LILACS and PubMed databases, along with a list of keywords identified for
each article using MER[
In a second iteration, we trained 4 additional models following the same characteristics of the previous ones, but using a larger train set composed by the 501 clinical cases that composed the CANTEMIST-CODING train set, plus 249 additional clinical cases from the dev1 set that were already classified. This way, we expected to achieved better results, since the models were trained with additional clinical cases. Summarizing, these four models had the following characteristics: • Model 5: BERT base Multilingual Cased finetuned with 750 clinical records. • Model 6: BETO finetuned with 750 clinical records. • Model 7: Spanish Biomedical X-Transformer finetuned with 750 clinical records. • Model 8: Spanish Biomedical X-Transformer large finetuned with 750 clinical records.
All models were trained using the default parameters of X-Transformer, except for the eval and train batch sizes which were both changed from their original values of 64 and 32 to 4 due to hardware constrains. We have also set the number of gradient accumulation steps to 2 to compensate for the small batch size. Each model was trained for 12 epochs, on a single NVIDIA Tesla P4 GPU.
In order to choose which model predictions to submit, we decided to evaluate the predictions made by each model on the dev2 set using the evaluation script given by the competition organizers. As was explained before, each model had 4 ’.tsv’ files as output, with each file containing the predictions with a confidence score superior to the confidence score threshold defined to best precision, recall or F1-score, and the baseline score, which corresponds to the confidence score threshold set to 0, which was the middle of the X-Transformer confidence score scale. Then, each file was used as input for the evaluation script given by the competition organizers.
The results for each model can be seen in Table 1. As it can be seen, the highest MAP scores are achieved when the predictions are focused on achieving the highest recall scores especially when using the models trained with the Spanish Biomedical X-Transformer models. We can notice that the models that use BETO seem to achieve higher scores when compared with the ones that used BERT Multilingual. In addition, we notice that there is not a clear diference between the usage of more clinical cases to train the models, with some models achieving slightly higher scores in MAP if the evaluation was focused on precision or in the F1-score, while in other models the score was inferior when compared with the models that used lesser articles.
Taking this into consideration, we decided to choose the predictions focused on recall of 5 distinct models to submit for the CANTEMIST-CODING task so we could also compare their performance in the competition. The chosen models were Models 2, 3, 4, 5 and 7. We then retrieved the predictions made by the models for each of the 48 text files that contained the test and background sets data. Then, for each of the resulting prediction files, the first 109 lines corresponding to the predictions made for the test and background sets were stored in the ’.tsv’ ifles, while the other 141 lines which corresponded to the predictions of the labelled cases from the first development set, were used to find the confidence score threshold that achieved the best recall score, and that would be used to choose which predictions would be stored in the ifnal ’.tsv’ file.
The results for the CANTEMIS-NER subtask are available in Table 2.
Our model obtained a F1-score of 0.749 in this subtask. Some errors prevented a higher performance, such as those related with the span of some detected entities. For example, in the document “cc_onco94.ann", the mention “linfoma" is correctly recognised by the NER tagger, but the processing script attributed the span “1690 1697", whereas the correct span would be “1691 1698". Besides, in some cases the NER tagger only recognised incomplete entity mentions. Model
Model 1 BERT Base Multilingual Cased
Model 2 BETO
Model 3 Spanish Biomedical X-Transformer
Model 4 Spanish Biomedical X-Transformer large
Model 5 BERT Base Multilingual Cased (750 CC) Model 6
BETO (750 CC)
Model 7 Spanish Biomedical X-Transformer (750 CC)
Model 8 Spanish Biomedical X-Transformer large (750 CC)
Focus Baseline
F1 Precision
Recall Baseline
F1 Precision
Recall Baseline
F1 Precision
Recall Baseline
F1 Precision
Recall Baseline
F1 Precision
Recall Baseline
F1 Precision
Recall Baseline
F1 Precision
Recall Baseline
F1 Precision
Recall
For example, in the document “cc_onco89.ann", the NER tagger recognised the entity “implantes mediastínicos", whereas the full entity mention would be “implantes mediastínicos pleurales".
For future work, it would be interesting to train the other models beside the “medium" (“base", “large" and “pubmed") in the CANTEMIST corpus and to apply them to the test set to verify if they obtain a higher performance. Besides, we only trained the Flair embeddings during less than 100 epochs, so with more epochs, probably the performance of the NER tagger would be higher. The NER tagger itself could also be trained for more epochs, up until 150, according to a Model medium
Model 1.single-ont 2.multi-ont suggestion by the authors of Flair. We will also address the errors associated with the span of the recognised entities.
The results for the CANTEMIS-NORM subtask are available in Table 3.
The model “multi-ont" obtained a F1-score of 0.061, which represents a slight improvement of +0.001 comparing to the “single-ont" model. Still, the performance of the model was too low, which is mainly related with the candidate retrieval step. Since we used string matching for candidate retrieval and selected the top candidates according to the edit distance between candidate/entity mention, most of the candidates lists did not contain the correct codes for the respective entity mentions. For example, the correct code for the entity mention “cáncer" would be the eCIE-O-3.1 code 8000/6, corresponding to the concept “Neoplasia metastásica". However, neither the “single-ont" model nor the “multi-ont" model were able to retrieve this candidate code for the entity mention, because the edit distance between “cancer" and “Neoplasia metastásica" is too high. Besides, the lack of a synonyms list in the eCIE-O-3.1 terminology further exacerbates the problem. Another aspect that is worth mentioning is the fact that the performance of the normalization model is always dependant on the performance of the NER tagger, so an incorrect output returned by the latter will hinder the results outputted by the former.
In order to improve the normalization model we will explore alternative methods for candidate generation, like for example, the use of word embeddings, both for entity mentions and for the terminology concepts. Instead of just considering that two entities in the same sentence are related, we will also use a proper relation extraction tool to get more semantically meaningful relations between concepts, either belonging to the same terminology, or belonging to diferent terminologies (for example, a relation between a eCIE-O-3 and a DeCS concept), which will densify the disambiguation graph and improve the disambiguation precision.
The results for the CANTEMIS-CODING subtask are available in Table 4.
Looking at our results, we can observe that, contrarily to what we expected, the best scoring model was Model 5 which, in our preliminary evaluation, had achieved the lowest MAP score of the five models submitted for this task. The lowest MAP scores were achieved by Models 3, 4 and 7, which were trained using the Spanish Biomedical X-Transformer models and that Model Model 2 Model 3 Model 4 Model 5 Model 7 had achieved the highest MAP scores in the preliminary evaluation. We can also notice that the precision and F1 scores were lower than the recall scores, but that was expected, since our submissions contained the predictions that used a confidence score threshold focused on achieving the highest recall scores.
As a proposal for a future work, we could try to develop additional models with a Transformer architecture, like the Biomedical X-Transformer models, and use them to train other X-Transformer models expecting to improve the number of correctly identified eCIE-O codes. These models could be trained using scientific biomedical articles in Spanish or even the clinical cases given by the CANTEMIST competition.
Another possible solution could pass by preprocessing the clinical case files before running them through X-Transformer by reducing the amount of text from each clinical case. This could be achieved by using automatic text summarization tools to leave only the essential information about each clinical case. Then, using NER tools, we could retrieve key entities and/or related terms and include them in the summarized text. This way, by reducing the original clinical case to a smaller and more objective text, along with identified key terms and entities given by the NER tools, it is expected that the X-Transformer model will be able to achieve better results since it has a smaller and more concise string of words, than a larger amount of text with the key topics more diluted.
We obtained a F1-score of 0.749 and 0.069 for the CANTEMIST-NER and the CANTEMIST-NORM subtasks, respectively, and a MAP of 0.506 for the CANTEMIST-CODING subtask. The code to run the developed models is available in our GitHub page: https://github.com/lasigeBioTM/ CANTEMIST-Participation.
To improve the NER tagger, we intend to resume the training of the Flair embeddings up until 2000 epochs and to generate a larger language model, with 2048 hidden layers instead of 1024, and to train the tagger over more text, by including the development 2 set in the training process. For the normalization model, we intend to explore the use of word embeddings in the candidate generation process and the use of a relation extraction tool to build better disambiguation graphs. As to improvements in the classification model, we intend to explore additional X-Transformer models trained with new models using biomedical data in Spanish and through the combination with other NLP techniques, such as automatic text summarization and NER, in order to further improve the results achieved by the algorithm.
This project was supported by FCT through funding of the DeST: Deep SemanticTagger project, ref. PTDC/CCI-BIO/28685/2017, and the LASIGE ResearchUnit, ref. UIDB/00408/2020