This paper presents the contribution of the NOVA LINCS team to the task MultiCardioNER at BioASQ12. BioASQ is a long-running challenge that focuses mostly on biomedical semantic indexing and question answering (QA). The specific task of MultiCardioNER focuses on the multilingual adaptation of clinical NER systems to the cardiology domain. We leverage a state-of-the-art spanish pre-trained model to perform NER on the DisTEMIST and DrugTEMIST datasets provided by this task, aided by the additional clinical cases corpus, CardioCCC, for validation. Experiments were done both with just the entity type to which each of those datasets refers to, and with the additional entity types from other datasets that use the same documents, in order to determine if any advantage could be obtained by leveraging the knowledge of the additional entities. The models trained on the combined dataset achieved a very slight and not significant boost in F1-score when compared to their one-entity counterparts, with all of them lacking heavily in recall, due to pre- and post-processing errors. However, one run achieved the highest precision of the task (0.9242). Code to reproduce our submission is available at https://github.com/Rodrigo1771/BioASQ12-MultiCardioNER-NOVALINCS.
The MultiCardioNER [
The datasets provided by each of the two MultiCardioNER subtasks, namely DisTEMIST and
DrugTEMIST, both make use of and annotate over the same set of documents, the SPACCC corpus [
This shared task utilizes two main training datasets, as previously mentioned: the DisTEMIST and DrugTEMIST datasets. Each of these corpora is composed by the same 1000 documents that belong to
DrugTEMIST-Dev the Spanish Clinical Case Corpus (SPACCC), a manually classified collection of clinical case reports written in the Spanish language. While DisTEMIST is focused on extracting disease mentions, thus having ENFERMEDAD as its sole and primary label, DrugTEMIST focuses on drug and medication extraction, with FARMACO being its entity type. Also previously mentioned is the fact that two former subtasks, MedProcNER and SympTEMIST, also make use of the SPACCC corpus to extract medical procedures and symptoms, having PROCEDIMIENTO and SINTOMA as their entity types, respectively.
Additionally, the organization also provided a separate cardiology clinical case reports dataset, CardioCCC, to be used for the domain adaptation part of the task. It contains a total of 508 documents, split into 258 documents initially intended for validation, and 250 for testing. As expected, these documents only include annotations for disease and medication mentions, the entity types considered for this competition. Nevertheless, we used this dataset both for training and validation of the model. Some statistics on all of these datasets are shown in Table 1.
After combining the four mentioned datasets (apart from CardioCCC) into a single dataset that includes the annotations of all four entity types, and given the task objective of adapting clinical models to the cardiology domain specifically, we joined this new combined dataset with CardioCCC, and split it so that 80% of examples (sentences) were used for training and 20% for validation. This way, we can include cardiology clinical case reports in the training of the model. Table 2 shows some statistics on every training and validation set.
Note that the three training sets are all equal between themselves in terms of the example sentences that they contain. Likewise, the two validation sets have the same exact sentences between them. The only thing that changes is the annotated entity types in those examples.
For this task, we leveraged the capabilities of the pre-trained model bsc-bio-ehr-eb1 [
We conducted two experiments. The first experiment compares the model when fine-tuned on the 1https://huggingface.co/PlanTL-GOB-ES/bsc-bio-ehr-es combined dataset with the four entities against the model when fine-tuned on just the DisTEMIST dataset. The second experiment also compares the model when fine-tuned on the combined dataset, but this time against the model when fine-tuned on just the DrugTEMIST dataset. This brings the total of runs submitted to four: • Track 1 (DisTEMIST): • Track 2 (DrugTEMIST), Spanish subset: – 1_bsc-bio-ehr-es_distemist_4: bsc-bio-ehr-es trained on the combined dataset with the 4 entity types (ENFERMEDAD, FARMACO, PROCEDIMIENTO, SINTOMA), and validated on the validation set for DisTEMIST (only with ENFERMEDAD). – 2_bsc-bio-ehr-es_distemist_1: trained and validated on the training and validation sets, respectively, for DisTEMIST (only with ENFERMEDAD). – 3_bsc-bio-ehr-es_drugtemist_4: bsc-bio-ehr-es trained on the combined dataset with the 4 entity types (ENFERMEDAD, FARMACO, PROCEDIMIENTO, SINTOMA), and validated on the validation set for DrugTEMIST (only with FARMACO). – 4_bsc-bio-ehr-es_drugtemist_1: trained and validated on the training and validation sets, respectively, for DrugTEMIST (only with FARMACO).
During training, the best checkpoint was kept, when evaluated on the validation set after each epoch. The model’s hyperparameters for all four runs were the following: • Learning rate: 5e-05 • Total train batch size: 16 • Epochs: 10
The results for each run obtained on our validation set, both during the development of our approach and after submitting it (the latter obtained using the oficial evaluation library 2, after it was released) are presented in Tables 3 and 4, respectively. Additionally, the oficial gold standard test set results are shown in Table 5. The naming convention for each run is as follows: {run_id}\_{model_name}\_{dataset}\_{number_of_entity_types_in_training}.
First of all it is important to point out the way in which we obtained our results on the validation set (Table 3), as well as the final submitted predictions, for the runs where the model was trained on the combined dataset (specifically runs 1 and 3), as those models naturally predict more entities than the entity relevant for the track. Thus, on those particular runs, after obtaining the predictions (be it the final test set predictions or the predictions used to further calculate the validation results), the ones that corresponded to entity types that did not match the entity type of the track were ignored. This way, only predictions of the entity type related to the track were considered. For example, in 2https://github.com/nlp4bia-bsc/multicardioner_evaluation_library
run 1, the predictions related to the entity types FARMACO, PROCEDIMIENTO and SINTOMA were ignored and only ENFERMEDAD was included in the final metric assessment seen on Table 3 and in the submitted predictions. Likewise, in run 3, the predictions related to the entity types ENFERMEDAD, PROCEDIMIENTO and SINTOMA were ignored and only FARMACO was considered.
The significant diference between the recall scores on our internal evaluation and on the oficial evaluation of the runs is evident. Usually, in a situation of high precision and low recall, it means that the system is only retrieving a small portion of the relevant entities (low recall), but when it does retrieve one, it identifies it correctly the vast majority of the time (high precision). Our first hypothesis to try to explain what happened was that this was due to our use of the BIO tagging schema. In this tagging schema, a sequence containing an entity, for example, of type ENFERMEDAD like "hipertensión pulmonar severa", should be classified as "B-ENFERMEDAD I-ENFERMEDAD I-ENFERMEDAD". Our reasoning was that the model could not generalize for entities that were not present in the training data, specially since we picked the checkpoint that obtained the best token classification and observed some overfitting when comparing the training and validation losses, shown in Figure 1. This could lead to the model under or over-classifying entities: • Under classification of "hipertensión pulmonar severa" - "B-ENFERMEDAD I-ENFERMEDAD O" (missing the "severa"). • Over classification of "estenosis píloro - duodenal de carácter extrínseco" - should be correctly classified as "B-ENFERMEDAD I-ENFERMEDAD I-ENFERMEDAD I-ENFERMEDAD O O O" but is overclassified as "B-ENFERMEDAD I-ENFERMEDAD I-ENFERMEDAD I-ENFERMEDAD I-ENFERMEDAD I-ENFERMEDAD I-ENFERMEDAD" (including the additional information "de carácter extrínseco").
This under and over-classification could then lead to a low recall, where the correct entities are not extracted and the number of false negatives grows. However, this would also lead to low precision, as both under and over-classification of entities would also lead to the extraction of incomplete entities which in turn leads to more false positives, and our models achieved good and even great precision. To achieve great precision and bad recall, our models would have to extract a low number of entities as to not increase the number of false positives, and they do as demonstrated by Tables 6 and 7.
Then, through further error analysis, we realized that this might be happening due to our data parsing approach. Because of how we parsed the datasets (same approach as PlanTL-GOB-ES’s when parsing 6 10
2
Epoch 3_bsc-bio-ehr-es_drugtemist_4 6
Epoch 4_bsc-bio-ehr-es_drugtemist_1 Epoch 6 8 8
Training Loss Validation Loss Best Model Epoch
10
CANTEMIST 3 or PharmaCoNER 4 for NER in order to fine-tune the same model that we used [
This conflict of approaches might explain the discrepancy between the quality of the results, particularly the recall, obtained on our validation set with our evaluation method, and the quality of the results obtained with the oficial evaluation library, both on our validation set and on the test set: as just mentioned, the examples that were fed to the model during training were individual sentences, meaning that they were overall much shorter (average length of 21 tokens, 124 chars) than the ones fed to the model when evaluating it on the test set (entire documents, average length of 1013 tokens, 5754 chars). Thus, the small size of the training examples might have induced the model to only extract entities up until a certain span when fed longer examples (i.e. documents). Furthermore, and possibly more relevant, the maximum input sequence length of this model is 512 tokens, which means that when obtaining the test set predictions to submit, with the entire documents as input, the tokens that went over this limit were not classified by our models.
These two reasons explain the low number of entities extracted, as the majority of the misses occur near the end of the documents. On the other hand, the results using our evaluation method, during the model’s training, did not show this low recall because the model was also evaluated on individual sentences. The information presented in Tables 8 and 9 supports our claims, as the average start span of (true and false) positive, i.e retrieved entities, is much lower than the average span of false negatives, i.e. relevant but not retrieved entities.
We believe that this is the reason why our approach showed consistent results across the board on our internal validation, but failed to replicate them on the oficial testing: basically the input length of the model when evaluating it on a document basis being smaller than many documents, with the additional nuance that the diference in length between the training and the test examples (the first much shorter than the second) may have induced the model to only predict entities until a certain span. We plan to re-classify the dev and tests sets using sentences instead of the full document in order to verify if the recall would improve and become closer to our internal evaluation.
Nevertheless, and focusing on the objective of experiment itself, we can observe that training with the combined dataset did not show any substantial improvements regarding the F1-Score. In fact, both tasks showed a barely significant disadvantage on the validation set: for task 1, a drop of 0.38pp on our evaluation method and of 0.85pp using the oficial evaluation library, and on task 2 a drop of 0.92pp for our evaluation method and of 0.76pp using the oficial library. On the test set, while the precision for all runs is quite good, even achieving the best precision score for any team on task 2 with run 3 (0.9242), the recall achieved by all runs is very low, which results in poor F1-scores, more specifically 20.64pp and 16.62pp below the mean for tasks 1 and 2 respectively, for our best run from each task. Furthermore, when checking for the main objective of the experiments, we observe again no substantial advantage for training with the combined dataset, only increasing the F1-Score by 0.95pp for task 1 and 0.80pp for task 2.
We employ the state-of-the-art transformer model bsc-bio-ehr-es and explore the hypothesis of if training with more entity types within biomedical domain would be beneficial for extracting a specific type, with the results indicating it does not make a significant diference. Furthermore, with the scores obtained on the test set, we show that this model can achieve results with high precision and with minimal fine-tuning. However, the model’s poor recall on the test set is noteworthy and likely caused by the the way in which the data is parsed for training and evaluation.
In the future, we intend to perform hyperparameter optimization on this model, to try to improve the scores that we obtained for the MultiCardioNER task (both on the DisTEMIST and DrugTEMIST subtracks), and apply English and Italian biomedical pre-trained language models to the corresponding subsets of the DrugTEMIST dataset.
This work is supported by NOVA LINCS ref. (https://doi.org/10.54499/UIDB/04516/2020) and ref. (https://doi.org/10.54499/UIDP/04516/2020) with the financial support of FCT.IP.
UIDB/04516/2020 UIDP/04516/2020 cessing, Association for Computational Linguistics, Dublin, Ireland, 2022, pp. 193–199. URL: https://aclanthology.org/2022.bionlp-1.19. doi:10.18653/v1/2022.bionlp-1.19. [7] Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, V. Stoyanov, RoBERTa: A Robustly Optimized BERT Pretraining Approach, 2019. URL: http://arxiv.org/abs/1907. 11692, arXiv:1907.11692 [cs]. [8] A. Miranda-Escalada, E. Farré-Maduell, M. Krallinger, Named Entity Recognition, Concept Normalization and Clinical Coding: Overview of the Cantemist Track for Cancer Text Mining in Spanish, Corpus, Guidelines, Methods and Results, 2020. doi:10.5281/zenodo.3773228. [9] A. G. Agirre, M. Marimon, A. Intxaurrondo, O. Rabal, M. Villegas, M. Krallinger, PharmaCoNER: Pharmacological Substances, Compounds and proteins Named Entity Recognition track, in: Proceedings of The 5th Workshop on BioNLP Open Shared Tasks, Association for Computational Linguistics, Hong Kong, China, 2019, pp. 1–10. URL: https://www.aclweb.org/anthology/D19-5701. doi:10.18653/v1/D19-5701.