This paper presents a transformer-based approach for disease Named Entity Recognition (NER) in Spanish clinical texts using the DisTEMIST dataset and drug multi-lingual NER in Spanish, English and Italian clinical texts using the DrugTEMIST dataset. For the disease NER task, we use CLIN-X-ES, a BERT-based model pretrained on Spanish clinical texts and additional pretrained on a custom dataset, and fine-tuned on token classification, achieving F1 score 0.8049 on the test set. For the drug NER task, we experiment with language-specific clinical models as well as general domain multilingual models and achieved the best results with the language-specific models. For Spanish we fine-tuned the CLIN-X-ES model and our best model showed 0.9238 F1 score, for English we fine-tuned BioLinkBERT which scored F1 - 0.9223, and for Italian we pretrained the CLIN-X-ES model with a custom Italian dataset and achieved F1 score - 0.8838. Our system placed first on the English and Italian tracks in the drug subtask in MultiCardioNER challenge.
Clinical narratives are a valuable source of healthcare information, but it requires design of special
NLP models for efective data extraction. Automated identification of key terms such as diseases,
medications, and procedures within clinical documents is known as clinical named entity recognition
(NER). This process plays a significant role in clinical natural language processing (NLP) by facilitating
the extraction of structured data from clinical narratives for subsequent analysis and interpretation
by downstream healthcare applications. MultiCardioNER1 [
This paper describes our approach for disease and drug NER which we submitted for the MultiCardioNER challenge. Our code is available on GitHub 2. The contributions of this paper are as follows: • Developed a system for disease entity recognition in Spanish and performed diferent experiments with BERT-based models, achieving 0.8049 F1 score on the DisTEMIST dataset; • Developed a system for drug entity recognition in Spanish, English, and Italian which achieved state-of-the-art (SOTA) results on English and Italian DrugTEMIST dataset and very competitive results on Spanish; • We investigated and compared the performance of multilingual BERT-based models vs languagespecific models for named entity recognition; • We adapted a clinical Spanish RoBERTa model to the Italian language and showed the best result on the drug NER task for Italian;
The state-of-the-art methods for biomedical named entity recognition are predominantly using deep
learning based models [
Specifically for the task of NER for medication extraction the SOTA methods are based on BiLSTM+CRF [14] reporting F1-score 0.93 for the best performing system for NER tasks over MIMIC-III3 dataset. Another approach is based on Question-Answering (QA) for medication event extraction [15] translating the NER task to span identification task in QA, reporting NER F1-score 0.98. The classical models like SVM, CRF and rule-based models [16],[17] show comparable results. In the n2c2 shared task on medication event extraction in clinical notes [18] the top score model for NER task scores strict F1 0.97 using transformer based pretrained LLM, using a BERT-based model (RoBERTa-large-PM-M3-Voc) with classification layer and BILOU tags.
For the task of NER for diagnoses, the SOTA methods are also based on transformers [19], [20] of various architectures. Most widely used are domain and task adaptations of the transformer architecture, such as BERT, RoBERTa and ELECTRA. Additional enhancement using knowledge bases in deep learning models [21] can help in dataset annotation and expansion.
The organizers of MultiCardioNER have organized multiple challenges in the area of information
extraction from Spanish clinical texts for diferent entity types - diseases, procedures, symptoms, etc.
Transformer-based approaches are very commonly used for NER tasks in diferent languages. In
previous challenges for diseases using the DisTEMIST dataset, the top teams have used BERT-based
models trained for token classification like Spanish RoBERTa
(PlanTL-GOB-ES/roberta-base-biomedicalclinical-es [22]) ([23], [24], [25]), mBERT and the Spanish BETO [26]. In a similar challenge for named
entity recognition of medical procedures in Spanish text, competitors used an architecture including
the same Spanish RoBERTa or XLM-RoBERTa model and adding BiLSTM and CRF layers on top ([
3.1. Subtask 1 MultiCardioNER corpus for disease named entity recognition task consists of two data subsets of diferent nature: DisTEMIST and CardioCCC. The DisTEMIST dataset consists of 1000 clinical cases of diferent medical specialities (incl. oncology, otorhinolaryngology, dentistry, pediatrics, primary care, allergology, radiology, psychiatry, ophthalmology and more). CardioCCC is a collection of 508 cardiology clinical case reports, which are longer on average than the DisTEMIST reports. The CardioCCC dataset contains 508 documents, split in 258 for development and 250 for testing. Following the suggestions by the organizers, we used DisTEMIST as a train set, leaving CardioCCC documents for validation. Additionally, a custom dataset was used for clinical domain adaptation (see Section 3.2). Each team had to provide their model predictions on a dataset containing the CardioCCC test data and a large unlabelled background dataset. At the time of submission, the organizers had not released which examples were part of the test set. Therefore the number of documents in the prediction set is a lot higher than the train or validation sets, however only a small portion was used for evaluating the model performance by the organizers. 3.2. Subtask 2 Similar to Subtask1, the MultiCardioNER corpus for the drug prediction task consists of two data subsets of diferent nature: DrugTEMIST and CardioCCC. The diference between the datasets is quite substantial. DrugTEMIST is a task-specific adaptation of DisTEMIST dataset that consists of 1000 clinical cases of diferent medical specialities. CardioCCC is a collection of 508 cardiology clinical case reports, meaning that reports are longer on average. Similarly to subtask 1, we used DrugTEMIST as a train set, leaving CardioCCC documents for validation.
One of the important aspects of the dataset is the sparsity of annotations. For DrugTEMIST only 9% sentences contain drugs, while for CardioCCC the share is even lower - 6%.
To adapt foundation models for the clinical domain, we collected language-specific datasets with raw texts. The data statistics can be found in Table 2. 1. Wikidata concepts related to medicine: We ran a SPARQL query over WikiData4 extracting labels that are included in the following medical ontologies and classifications: ICD-11 5; ICD-10,
ICD-10 CM 6, Symptom Ontology7, eMedicine8, DiseasesDB, MedlinePlus9, MONDO10 , Human Disease Ontology11, SNOMED CT12, UMLS13. 2. Wikipedia articles related to medical concepts: Based on the extracted WikiData concepts, we went through the concept list and downloaded the texts of WikiPedia articles that were created for those concepts. For some concepts the articles did not exist. We used Mediawiki API 14 for text extraction. 3. Abbreviation lists found online: For each of the languages, we browsed for medical abbreviation lists available online. Some of them were extracted from the open-source articles, others were scraped from websites. 4. Drug descriptions: As drug descriptions usually contain quite useful information on symptoms, side efects and dosages, we leveraged multilingual drug description lists 15. 5. EMA medical documentation: We leveraged a parallel corpus of the European Medical Agency
Documentation16. 6. Machine-translated data: To enrich the amount of medical data, we decided to use machine translated datasets. For this purpose, we used medical abstracts corpus17.
As drug lists are dynamic and changing over time, specifically designed dictionaries were created based on various oficial drug sources to support model-based drug extraction. For the English drug dictionary, we used OMOP Standardized Vocabulary V5.0 18 (incl. ATC, RxNorm), DrugCentral19 (FDA Approved Drugs, EMA Approved Drugs, PMDA Approved Drugs, PMDA+EMA+FDA Approved Drug), DrugBank20, DailyMed21 (NIHS human drugs), Top25022, UnatedHealthcare23, and Drugs.com24 (My Med List). For Spanish, we used Centro de información online de medicamentos de la AEMPS - (CIMA)25 (incl. ATC Spanish version and Arbol Medicamentos DSCA Spanish). For Italian, we used ATC and Lists of Class A and Class H medicinal products of Italian Medicine Agency26. The drug names and synonyms are aggregated in three dictionaries for English, Italian and Spanish. The total number of generic and brand names of drugs included in the cleaned lists, after removing duplication are presented on Table 3. In addition to the drug dictionaries were used some procedures names for lab test from LOINC27 for all languages in order to disambiguate some drug mentions from lab tests for measuring levels of some minerals and vitamins, like Vitamin D, Magnesium, Calcium, etc. 6https://www.eciemaps.sanidad.gob.es/browser/metabuscador 7https://raw.githubusercontent.com/DiseaseOntology/SymptomOntology/main/src/ontology/symp.owl 8https://emedicine.medscape.com/ 9https://medlineplus.gov/ 10https://obofoundry.org/ontology/mondo 11https://www.disease-ontology.org/ 12https://www.snomed.org/ 13https://www.nlm.nih.gov/research/umls/index.html 14https://www.mediawiki.org/wiki/API:Main_page 15https://www.ema.europa.eu/en/medicines/human 16https://live.european-language-grid.eu/catalogue/corpus/12729 17https://github.com/sebischair/medical-abstracts-tc-corpus 18https://www.ohdsi.org/web/wiki/doku.php?id=documentation:vocabulary 19https://drugcentral.org/ 20https://go.drugbank.com/ 21https://www.dailymed.nlm.nih.gov/dailymed/ 22https://clincalc.com/PronounceTop200Drugs/ 23https://www.uhc.com/member-resources/pharmacy-benefits/prescription-drug-lists 24https://www.drugs.com/mednotes/ 25https://cima.aemps.es/cima/publico/nomenclator.html 26https://www.aifa.gov.it/en/liste-farmaci-a-h/ 27https://loinc.org/
As clinical documents are quite lengthy, especially in the case of the CardioCCC corpus, we decided to
split the documents into sentences using the following tools for diferent languages:
• English - MedSpaCy Sentence Splitting 28
• Italian - Tint Sentence Splitting 29 [
Afterwards, we used the Brat tool 31 [
4.1. Subtask 1
Our system was built using transformer-based models that were either multilingual or adapted to
Spanish and that were preferably adapted to the biomedical domain as well. The models we used were:
• PlanTL-GOB-ES/roberta-base-biomedical-clinical-es [22]: Biomedical pretrained
language model for Spanish. This model is a RoBERTa-based model trained on a
biomedicalclinical corpus in Spanish collected from several sources.
• CLIN-X-ES [
• mDeBERTa v3 32: A multilingual version of DeBERTa.
Additionally, some of the models were further pretrained on medical data from the Custom Medical Text Dataset described in Section 3.3. Approximately 0.08 of the tokens are annotated entities per sentence which corresponds to a very sparse annotation setting. Hence the majority of tokens to be evaluated by the model will be negative examples. Therefore, we tried to fine-tune our models with diferent class weight ratios (positive to negative samples) to try to make up for the class imbalance. The ratios chosen were inversely proportional to the prevalence of the class. 28https://github.com/medspacy/medspacy 29https://github.com/dhfbk/tint 30https://github.com/PlanTL-GOB-ES/SPACCC_Sentence-Splitter 31http://brat.nlplab.org 32https://huggingface.co/microsoft/mdeberta-v3-base 4.2. Subtask 2 The drug name extraction task was reformulated as a drug Named Entity Recognition (NER) task. As the setting for this task was multilingual, we focused on two major approaches: building a single multilingual model capable of making predictions on all languages by leveraging knowledge transfer between languages during training, and training language-specific models which are independent from each other. In addition, as the drug name list is dynamic, we experimented with adding drug names from oficial drug registries such as DrugBank as a gazzetteer. The gazetteer collection is described in Section 3.4.
In particular we experimented with the following methods: • Multilingual model
For the foundation multilingual model we used a FacebookAI/xlm-roberta-base33 backbone. We experimented with training the original model and also performing domain adaptation. The adapted model was trained on English, Spanish and Italian medical datasets on the Masked Language Modeling objective using the dataset from Section 3.3. In addition, we experimented with a multilingual model pretrained on several NER datasets which has shown good results on similar tasks: numind/NuNER-multilingual-v0.134 • Language-specific models
As a set of language-specific models we focused on michiyasunaga/BioLinkBERT-base35 for English, PlanTL-GOB-ES/roberta-base-biomedical-clinical-es36 for Spanish and dbmdz/bert-base-italian-cased37 for Italian. As the models for English and Spanish were originally pretrained on medical data, we expected them to perform well as is, whilst for the Italian model, we conducted additional domain adaptation. Furthermore, as Italian medical vocabulary is relatively close to Spanish, we conducted language adaptation of the Spanish model on the Italian medical dataset too. • Drug Gazetteer
The gazetteer was applied before model predictions by finding exact match of drug names in the text. Dictionary statistics and description can be found in Section 3.4. 33https://huggingface.co/FacebookAI/xlm-roberta-base 34https://huggingface.co/numind/NuNER-multilingual-v0.1 35https://huggingface.co/michiyasunaga/BioLinkBERT-base 36https://huggingface.co/PlanTL-GOB-ES/roberta-base-biomedical-clinical-es 37https://huggingface.co/dbmdz/bert-base-italian-cased
As described in Section 3.2, the dataset is highly imbalanced and as a potential solution for boosting the precision of predictions, we experimented with training a binary classification model to identify sentences that contain drug annotations and sort out empty sentences. FacebookAI/xlm-roberta-base after domain adaptation was used t train the classifier model. Furthermore, as many drug names included punctuation marks, we added a post-processing step joining drug names divided by symbols / and +.
Figure 1 shows the overall architecture of the approach. Depending on the configuration we either include or do not include filtering and dictionary-based annotations.
5.1. Subtask 1 For the Named Entity Recognition task we employed a standard approach for token classification task. To simplify the setting and to avoid truncation due to limits in the input sequence length, we trained on the split sentences (see Section 3.5). Standard token classification pipeline from Huggingface Transformers was used. For the pretraining, we used a standard masked language modeling pretraining objective. In general, the models achieved better recall than precision. The models that achieved the best results were the models that had domain adaptation for both Spanish and clinical language domains. Despite that DeBERTa is generally a model that achieves better performance than XLM-R, the best model in our experiments was CLIN-X-ES, a XLM-R-based model pretrained on a Spanish clinical corpus. It is notable that DeBERTa-base achieved better performance than DeBERTa-large probably because the dataset was quite small. Using pretraining generally gave small improvements or no improvements, but pertained models converged much faster - on average it took them between 2-6 epochs less to converge. 5.2. Subtask 2 For the second subtask the setup was quite similar to the first one. During the first set of experiments we compared the multilingual candidate models. The results are reported at Table 6. It could be observed that in-domain pretraining positively influences the overall performance of the model. For each of the languages, the F-score improves by circa 2%.
As for the monolingual model comparison reported at Table 7, we can observe that in general monolingual models exhibit slightly better performance for all the target languages except for Italian. Curiously, a Spanish model trained on medical data and fine-tuned on Italian medical data performs better compared to Italian foundation model adapted to the clinical domain.
Table 8 reports results of the submitted systems on the validation dataset. In general, language-specific models show better performance for this task. Filtering consistently increases precision but slightly reduces recall for medical XLM-R model pipelines for all the languages. The result is quite natural as by ifltering out extra sentences we are adding more false negative examples to the final prediction. The most significant diference is observed for Italian.
The final evaluation on the test set showed that medical XLM-R is a SOTA result for Italian and BioLinkBERT is the best approach for English.
As for the final submission, the best candidate models remained unchanged. Although on the validation set filtering out empty sentences proved to be a beneficial strategy, on the test set results of the pipeline without filtering are generally better.
After the annotated test data was released, we conducted error analysis and found the following patterns. First, most of the errors related to recall are the cases of drugs that include numbers and special symbols. However, there were cases where the model predicted drugs names that were missing in the annotations (e.g. it was the case with Angiotensin-converting enzyme (ACE) inhibitors).
Based on the performance evaluation one can see that adding dictionary-based matching decreases precision of the models. This can be explained by several factors rooting from the nature of the dictionary we used. In the oficial drug registries there are quite a lot of drugs with ambiguous brand names (e.g. vita - life in Italian), which increases the number of false positive predictions. In addition, there are drug names that are homonymous with laboratory test measures, for instance sodium, vitamin K, etc. Rule-based matching does not rule out such cases. Lastly, oficial names of the medications include dosages and concentrations, while in the challenge data those were not included in the annotations (e.g. oficial name lidocain 2%, annotated name lidocain).
The issue with drug dosage and medication concentration is relevant for model predictions too. While the Italian models rarely included concentrations in the predictions, for English and Spanish this was often the case. Such inconsistencies originate from training data diferences. The final models were trained on 2 diferent datasets (DrugTEMIST, CardioCCC) and those appeared to be annotated diferently. While in DrugTEMIST dosages and concentrations were consistently included in the drug span, for CardioCCC this is not the case. Moreover, combined drug names were also annotated inconsistently being either split into 2 drugs, or combined into one. Figure 2 shows an example of drug annotations which do not include the respective dosages as part of the labelled span. The inconsistency of labelling between the train, dev, and test sets is a source of errors for the final trained model.
In this paper we presented transformer based models for drug and disease named entity recognition in multilingual clinical texts. The experiments for disease NER showed that the domain-adapted and language-specific model CLIN-X-ES scored 0.819 F1 and outperformed DeBERTa-based models. For the drug NER task, the best scores for English and Spanish were achieved using monolingual models BioLinkBERT and RoBerta-base-biomedical-clinical-es respectively. For Italian, in contrast, the multilingual model XLMR_med outperformed the monolingual one by a small margin. Although dictionaries showed contribution to the recall metric, the ambiguities in the drug names that contain some common vocabulary or the common confusion with lab test results cause a significant drop in the precision. We experimented with diferent approaches to attempt to tackle the label sparsity issue adjusting class weights during training as well as adding a classification step which predicts whether the sentence contains a drug name. The approach using a classifier as a filtering step showed improved performance on the validation set, however, did not work so well on the actual test set. As future work, using hybrid solution with the help of LLMs can improve the system performance and address the issues with the disambiguation of the term usage in context.
This work was partially supported by the European Union-NextGenerationEU, through the National Recovery and Resilience Plan of the Republic of Bulgaria [Grant Project No. BG-RRP-2.004-0008] and by Horizon Europe research and innovation programme project RES-Q plus [Grant Agreement No. 101057603], funded by the European Union. Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or the Health and Digital Executive Agency. Neither the European Union nor the granting authority can be held responsible for them.
The methods described in this paper were submitted as part of the MultiCardioNER challenge and were validated only on the challenge datasets. Further investigation on diferent datasets is needed to explore the generalizability of the approach. The specific labeling approach on the datasets impacts the model performance, as in the case of drug NER the inconsistencies of dosage labelling was a source of errors. In diferent settings, the labelling guidelines used might be diferent and therefore the presented approach may not perform as well. [8] S. Archana, J. Prakash, An efective undersampling method for biomedical named entity recognition using machine learning, Evolving Systems (2024) 1–9. [9] X. Wang, C. Yang, R. Guan, A comparative study for biomedical named entity recognition,
International Journal of Machine Learning and Cybernetics 9 (2018) 373–382. [10] A. Yazdani, D. Proios, H. Rouhizadeh, D. Teodoro, Eficient joint learning for clinical named entity recognition and relation extraction using fourier networks: A use case in adverse drug events, arXiv preprint arXiv:2302.04185 (2023). [11] Y. Lou, X. Zhu, K. Tan, Dictionary-based matching graph network for biomedical named entity recognition, Scientific Reports 13 (2023) 21667. [12] S. Ghosh, U. Tyagi, S. Kumar, D. Manocha, Bioaug: Conditional generation based data augmentation for low-resource biomedical ner, in: Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, 2023, pp. 1853–1858. [13] I. Bartolini, V. Moscato, M. Postiglione, G. Sperlì, A. Vignali, Data augmentation via context similarity: An application to biomedical named entity recognition, Information Systems 119 (2023) 102291. [14] Q. Wei, Z. Ji, Z. Li, J. Du, J. Wang, J. Xu, Y. Xiang, F. Tiryaki, S. Wu, Y. Zhang, et al., A study of deep learning approaches for medication and adverse drug event extraction from clinical text, Journal of the American Medical Informatics Association 27 (2020) 13–21. [15] T. Tsujimura, K. Yamada, R. Ida, M. Miwa, Y. Sasaki, Contextualized medication event extraction with striding ner and multi-turn qa, Journal of Biomedical Informatics 144 (2023) 104416. [16] S. Doan, N. Collier, H. Xu, P. H. Duy, T. M. Phuong, Recognition of medication information from discharge summaries using ensembles of classifiers, BMC medical informatics and decision making 12 (2012) 1–10. [17] S. Doan, H. Xu, Recognizing medication related entities in hospital discharge summaries using support vector machine, in: Proceedings of COLING. International conference on computational linguistics, volume 2010, NIH Public Access, 2010, p. 259. [18] D. Mahajan, J. J. Liang, C.-H. Tsou, Ö. Uzuner, Overview of the 2022 n2c2 shared task on contextualized medication event extraction in clinical notes, Journal of Biomedical Informatics 144 (2023) 104432. [19] X. Liu, Y. Zhou, Z. Wang, Recognition and extraction of named entities in online medical diagnosis data based on a deep neural network, Journal of Visual Communication and Image Representation 60 (2019) 1–15. [20] M. Polignano, M. de Gemmis, G. Semeraro, et al., Comparing transformer-based ner approaches for analysing textual medical diagnoses., in: CLEF (Working Notes), 2021, pp. 818–833. [21] S. Silvestri, F. Gargiulo, M. Ciampi, Iterative annotation of biomedical ner corpora with deep neural networks and knowledge bases, Applied sciences 12 (2022) 5775. [22] C. P. Carrino, J. Armengol-Estapé, A. Gutiérrez-Fandiño, J. Llop-Palao, M. Pàmies, A. GonzalezAgirre, M. Villegas, Biomedical and clinical language models for spanish: On the benefits of domain-specific pretraining in a mid-resource scenario, 2021. arXiv:2109.03570. [23] V. Moscato, M. Postiglione, G. Sperlì, Biomedical spanish language models for entity recognition and linking at bioasq distemist, in: Working Notes of Conference and Labs of the Evaluation (CLEF) Forum. CEUR Workshop Proceedings, 2022, pp. 315–324. [24] F. Borchert, M.-P. Schapranow, Hpi-dhc @ bioasq distemist: Spanish biomedical entity linking with pre-trained transformers and cross-lingual candidate retrieval, in: Conference and Labs of the Evaluation Forum, 2022. URL: https://api.semanticscholar.org/CorpusID:251471783. [25] M. Chizhikova, J. Collado-Montañez, P. López-Úbeda, M. C. Díaz-Galiano, L. A. U. López, M. T. M.
Valdivia, Sinai at clef 2022: Leveraging biomedical transformers to detect and normalize disease mentions, in: Conference and Labs of the Evaluation Forum, 2022. URL: https://api.semanticscholar. org/CorpusID:251471813. [26] J. Reyes-Aguillón, R. del Moral, O. Ramos-Flores, H. Gómez-Adorno, G. Bel-Enguix, Clinical named entity recognition and linking using bert in combination with spanish medical embeddings, in: Conference and Labs of the Evaluation Forum, 2022. URL: https://api.semanticscholar.org/
For fine-tuning the models we used the following hyperparameters settings:
• learning rate: We used AdamW[
• learning rate scheduler: linear
• learning rate: We used AdamW optimizer with 5-e5 learning rate. • number of epochs: 3 epochs due to resource limitations. • weight decay: 0.01. • batch size: Initialized to 8 due to GPU memory limitations. • learning rate scheduler: linear