Clinical narratives are a valuable source of healthcare information, but it requires design of special NLP models for effective data extraction. Automated identification of key terms such as diseases, medications, and procedures within clinical documents is known as clinical named entity recognition (NER). This process plays a significant role in clinical natural language processing (NLP) by facilitating the extraction of structured data from clinical narratives for subsequent analysis and interpretation by downstream healthcare applications. MultiCardioNER 1 [1] is a shared task part of CLEF BioASQ [2], which aims to detect diseases in Spanish clinical texts as well as drugs in a multilingual setting, including Spanish, English and Italian. The organizers have provided annotated datasets for training and evaluation of the systems -DisTEMIST for disease NER and DrugTEMIST for drug NER. The shared task consists of two subtask -Subtask 1 addressing disease recognition in Spanish, and Subtask 2 addressing drug recognition in multiple languages. In both subtasks, the challenge is to recognize terms in cardiology reports.
This paper describes our approach for disease and drug NER which we submitted for the Mul-tiCardioNER challenge. Our code is available on GitHub 2 . The contributions of this paper are as follows:
• Developed a system for disease entity recognition in Spanish and performed different experiments with BERT-based models, achieving 0.8049 F1 score on the DisTEMIST dataset; • Developed a system for drug entity recognition in Spanish, English, and Italian which achieved state-of-the-art (SOTA) results on English and Italian DrugTEMIST dataset and very competitive results on Spanish; • We investigated and compared the performance of multilingual BERT-based models vs languagespecific models for named entity recognition; • We adapted a clinical Spanish RoBERTa model to the Italian language and showed the best result on the drug NER task for Italian;
The state-of-the-art methods for biomedical named entity recognition are predominantly using deep learning based models [3], [4]. The most recent NER approaches for clinical documents also include some hybrid models like dictionary guided attention based model [5] and transfer learning [6], [7].
Besides classical approaches like machine learning [8], hidden Markov models (HMM) and conditional random fields (CRF) [9], an interesting application of Fourier Networks for NER and relation extraction were proposed in [10]. Another direction of research is using models based on knowledge graphs [11] as additional source for information. The limited availability of annotated data, imbalanced training datasets and lack of resources in low resource languages triggered another direction in the NER model development using data augmentation techniques to tackle these issues [12], [13]. Specifically for the task of NER for medication extraction the SOTA methods are based on BiL-STM+CRF [14] reporting F1-score 0.93 for the best performing system for NER tasks over MIMIC-III3 dataset. Another approach is based on Question-Answering (QA) for medication event extraction [15] translating the NER task to span identification task in QA, reporting NER F1-score 0.98. The classical models like SVM, CRF and rule-based models [16], [17] show comparable results. In the n2c2 shared task on medication event extraction in clinical notes [18] the top score model for NER task scores strict F1 0.97 using transformer based pretrained LLM, using a BERT-based model (RoBERTa-large-PM-M3-Voc) with classification layer and BILOU tags.
For the task of NER for diagnoses, the SOTA methods are also based on transformers [19], [20] of various architectures. Most widely used are domain and task adaptations of the transformer architecture, such as BERT, RoBERTa and ELECTRA. Additional enhancement using knowledge bases in deep learning models [21] can help in dataset annotation and expansion.
The organizers of MultiCardioNER have organized multiple challenges in the area of information extraction from Spanish clinical texts for different entity types -diseases, procedures, symptoms, etc. Transformer-based approaches are very commonly used for NER tasks in different languages. In previous challenges for diseases using the DisTEMIST dataset, the top teams have used BERT-based models trained for token classification like Spanish RoBERTa (PlanTL-GOB-ES/roberta-base-biomedicalclinical-es [22]) ( [23], [24], [25]), mBERT and the Spanish BETO [26]. In a similar challenge for named entity recognition of medical procedures in Spanish text, competitors used an architecture including the same Spanish RoBERTa or XLM-RoBERTa model and adding BiLSTM and CRF layers on top ( [27], [28]). In the task of symptoms NER, ensemble of transformer models for Spanish clinical text achieved the best result -F1 0.74 (strict) [29].
MultiCardioNER corpus for disease named entity recognition task consists of two data subsets of different nature: DisTEMIST and CardioCCC. The DisTEMIST dataset consists of 1000 clinical cases of different medical specialities (incl. oncology, otorhinolaryngology, dentistry, pediatrics, primary care, allergology, radiology, psychiatry, ophthalmology and more). CardioCCC is a collection of 508 cardiology clinical case reports, which are longer on average than the DisTEMIST reports. The CardioCCC dataset contains 508 documents, split in 258 for development and 250 for testing. Following the suggestions by the organizers, we used DisTEMIST as a train set, leaving CardioCCC documents for validation. Additionally, a custom dataset was used for clinical domain adaptation (see Section 3.2). Each team had to provide their model predictions on a dataset containing the CardioCCC test data and a large unlabelled background dataset. At the time of submission, the organizers had not released which examples were part of the test set. Therefore the number of documents in the prediction set is a lot higher than the train or validation sets, however only a small portion was used for evaluating the model performance by the organizers.
Similar to Subtask1, the MultiCardioNER corpus for the drug prediction task consists of two data subsets of different nature: DrugTEMIST and CardioCCC. The difference between the datasets is quite substantial. DrugTEMIST is a task-specific adaptation of DisTEMIST dataset that consists of 1000 clinical cases of different medical specialities. CardioCCC is a collection of 508 cardiology clinical case reports, meaning that reports are longer on average. Similarly to subtask 1, we used DrugTEMIST as a train set, leaving CardioCCC documents for validation. One of the important aspects of the dataset is the sparsity of annotations. For DrugTEMIST only 9% sentences contain drugs, while for CardioCCC the share is even lower -6%.
To adapt foundation models for the clinical domain, we collected language-specific datasets with raw texts. The data statistics can be found in Table 2. The data was collected using the following sources:
1. Wikidata concepts related to medicine: We ran a SPARQL query over WikiData4 extracting labels that are included in the following medical ontologies and classifications: ICD-11 5 ; ICD-10, ICD-10 CM6 , Symptom Ontology7 , eMedicine 8 , DiseasesDB, MedlinePlus9 , MONDO 10 , Human Disease Ontology 11 , SNOMED CT 12 , UMLS 13 . 2. Wikipedia articles related to medical concepts: Based on the extracted WikiData concepts, we went through the concept list and downloaded the texts of WikiPedia articles that were created for those concepts. For some concepts the articles did not exist. We used Mediawiki API 14 for text extraction. 3. Abbreviation lists found online: For each of the languages, we browsed for medical abbreviation lists available online. Some of them were extracted from the open-source articles, others were scraped from websites. 4. Drug descriptions: As drug descriptions usually contain quite useful information on symptoms, side effects and dosages, we leveraged multilingual drug description lists 15 . 5. EMA medical documentation: We leveraged a parallel corpus of the European Medical Agency Documentation 16 . 6. Machine-translated data: To enrich the amount of medical data, we decided to use machine translated datasets. For this purpose, we used medical abstracts corpus 17 .
As drug lists are dynamic and changing over time, specifically designed dictionaries were created based on various official drug sources to support model-based drug extraction. For the English drug dictionary, we used OMOP Standardized Vocabulary V5.0 18 (incl. ATC, RxNorm), DrugCentral 19 (FDA Approved Drugs, EMA Approved Drugs, PMDA Approved Drugs, PMDA+EMA+FDA Approved Drug), DrugBank 20 , DailyMed 21 (NIHS human drugs), Top250 22 , UnatedHealthcare 23 , and Drugs.com 24 (My Med List). For Spanish, we used Centro de información online de medicamentos de la AEMPS -(CIMA) 25 (incl. ATC Spanish version and Arbol Medicamentos DSCA Spanish). For Italian, we used ATC and Lists of Class A and Class H medicinal products of Italian Medicine Agency 26 . The drug names and synonyms are aggregated in three dictionaries for English, Italian and Spanish. The total number of generic and brand names of drugs included in the cleaned lists, after removing duplication are presented on Table 3. In addition to the drug dictionaries were used some procedures names for lab test from LOINC 27 for all languages in order to disambiguate some drug mentions from lab tests for measuring levels of some minerals and vitamins, like Vitamin D, Magnesium, Calcium, etc.
As clinical documents are quite lengthy, especially in the case of the CardioCCC corpus, we decided to split the documents into sentences using the following tools for different languages:
• English -MedSpaCy Sentence Splitting 28• Italian -Tint Sentence Splitting29 [30] • Spanish -SPACCC Sentence splitter 30Afterwards, we used the Brat tool31 [31] for data transformation from BRAT to CONLL format. The dataset statistics after pre-processing are shown in Table 4.
Our system was built using transformer-based models that were either multilingual or adapted to Spanish and that were preferably adapted to the biomedical domain as well. The models we used were:
• PlanTL-GOB-ES/roberta-base-biomedical-clinical-es [22]: Biomedical pretrained language model for Spanish. This model is a RoBERTa-based model trained on a biomedicalclinical corpus in Spanish collected from several sources. • CLIN-X-ES [32]: This model is based on the multilingual XLM-R transformer (xlm-roberta-large) further pretrained on a Spanish clinical corpus. • DeBERTa v3 [33]: A transformer-based model with disentagled attention trained using ELECTRA style pretraining. Both base and large versions were used and also a version of DeBERTa that was further pretrained on clinical data. • mDeBERTa v3 32 : A multilingual version of DeBERTa.
Additionally, some of the models were further pretrained on medical data from the Custom Medical Text Dataset described in Section 3.3. Approximately 0.08 of the tokens are annotated entities per sentence which corresponds to a very sparse annotation setting. Hence the majority of tokens to be evaluated by the model will be negative examples. Therefore, we tried to fine-tune our models with different class weight ratios (positive to negative samples) to try to make up for the class imbalance. The ratios chosen were inversely proportional to the prevalence of the class.
The drug name extraction task was reformulated as a drug Named Entity Recognition (NER) task. As the setting for this task was multilingual, we focused on two major approaches: building a single multilingual model capable of making predictions on all languages by leveraging knowledge transfer between languages during training, and training language-specific models which are independent from each other. In addition, as the drug name list is dynamic, we experimented with adding drug names from official drug registries such as DrugBank as a gazzetteer. The gazetteer collection is described in Section 3. 4.
In particular we experimented with the following methods:
•
For the foundation multilingual model we used a FacebookAI/xlm-roberta-base33 backbone.
We experimented with training the original model and also performing domain adaptation. The adapted model was trained on English, Spanish and Italian medical datasets on the Masked Language Modeling objective using the dataset from Section 3.3. In addition, we experimented with a multilingual model pretrained on several NER datasets which has shown good results on similar tasks: numind/NuNER-multilingual-v0.134 • Language-specific models
As a set of language-specific models we focused on michiyasunaga/BioLinkBERT-base35 for English, PlanTL-GOB-ES/roberta-base-biomedical-clinical-es36 for Spanish and dbmdz/bert-base-italian-cased 37 for Italian. As the models for English and Spanish were originally pretrained on medical data, we expected them to perform well as is, whilst for the Italian model, we conducted additional domain adaptation. Furthermore, as Italian medical vocabulary is relatively close to Spanish, we conducted language adaptation of the Spanish model on the Italian medical dataset too.
The gazetteer was applied before model predictions by finding exact match of drug names in the text. Dictionary statistics and description can be found in Section 3.4. As described in Section 3.2, the dataset is highly imbalanced and as a potential solution for boosting the precision of predictions, we experimented with training a binary classification model to identify sentences that contain drug annotations and sort out empty sentences. FacebookAI/xlm-roberta-base after domain adaptation was used t train the classifier model. Furthermore, as many drug names included punctuation marks, we added a post-processing step joining drug names divided by symbols / and +.
Figure 1 shows the overall architecture of the approach. Depending on the configuration we either include or do not include filtering and dictionary-based annotations.
For the Named Entity Recognition task we employed a standard approach for token classification task. To simplify the setting and to avoid truncation due to limits in the input sequence length, we trained on the split sentences (see Section 3.5). Standard token classification pipeline from Huggingface Transformers was used. For the pretraining, we used a standard masked language modeling pretraining objective. In general, the models achieved better recall than precision. The models that achieved the best results were the models that had domain adaptation for both Spanish and clinical language domains. Despite that DeBERTa is generally a model that achieves better performance than XLM-R, the best model in our experiments was CLIN-X-ES, a XLM-R-based model pretrained on a Spanish clinical corpus. It is notable that DeBERTa-base achieved better performance than DeBERTa-large probably because the dataset was quite small. Using pretraining generally gave small improvements or no improvements, but pertained models converged much faster -on average it took them between 2-6 epochs less to converge.
For the second subtask the setup was quite similar to the first one. During the first set of experiments we compared the multilingual candidate models. The results are reported at Table 6. It could be observed that in-domain pretraining positively influences the overall performance of the model. For each of the languages, the F-score improves by circa 2%.
As for the monolingual model comparison reported at Table 7, we can observe that in general monolingual models exhibit slightly better performance for all the target languages except for Italian. Curiously, a Spanish model trained on medical data and fine-tuned on Italian medical data performs better compared to Italian foundation model adapted to the clinical domain.
Table 8 reports results of the submitted systems on the validation dataset. In general, language-specific models show better performance for this task. Filtering consistently increases precision but slightly The final evaluation on the test set showed that medical XLM-R is a SOTA result for Italian and BioLinkBERT is the best approach for English.
As for the final submission, the best candidate models remained unchanged. Although on the validation set filtering out empty sentences proved to be a beneficial strategy, on the test set results of the pipeline without filtering are generally better.
After the annotated test data was released, we conducted error analysis and found the following patterns. First, most of the errors related to recall are the cases of drugs that include numbers and special symbols. However, there were cases where the model predicted drugs names that were missing in the annotations (e.g. it was the case with Angiotensin-converting enzyme (ACE) inhibitors). Based on the performance evaluation one can see that adding dictionary-based matching decreases precision of the models. This can be explained by several factors rooting from the nature of the dictionary we used. In the official drug registries there are quite a lot of drugs with ambiguous brand names (e.g. vita -life in Italian), which increases the number of false positive predictions. In addition, there are drug names that are homonymous with laboratory test measures, for instance sodium, vitamin K, etc. Rule-based matching does not rule out such cases. Lastly, official names of the medications include dosages and concentrations, while in the challenge data those were not included in the annotations (e.g. official name lidocain 2%, annotated name lidocain).
The issue with drug dosage and medication concentration is relevant for model predictions too. While the Italian models rarely included concentrations in the predictions, for English and Spanish this was often the case. Such inconsistencies originate from training data differences. The final models were trained on 2 different datasets (DrugTEMIST, CardioCCC) and those appeared to be annotated differently. While in DrugTEMIST dosages and concentrations were consistently included in the drug span, for CardioCCC this is not the case. Moreover, combined drug names were also annotated inconsistently being either split into 2 drugs, or combined into one. Figure 2 shows an example of drug annotations which do not include the respective dosages as part of the labelled span. The inconsistency of labelling between the train, dev, and test sets is a source of errors for the final trained model.
In this paper we presented transformer based models for drug and disease named entity recognition in multilingual clinical texts. The experiments for disease NER showed that the domain-adapted and language-specific model CLIN-X-ES scored 0.819 F1 and outperformed DeBERTa-based models. For the drug NER task, the best scores for English and Spanish were achieved using monolingual models BioLinkBERT and RoBerta-base-biomedical-clinical-es respectively. For Italian, in contrast, the multilingual model XLMR_med outperformed the monolingual one by a small margin. Although dictionaries showed contribution to the recall metric, the ambiguities in the drug names that contain some common vocabulary or the common confusion with lab test results cause a significant drop in the precision. We experimented with different approaches to attempt to tackle the label sparsity issueadjusting class weights during training as well as adding a classification step which predicts whether the sentence contains a drug name. The approach using a classifier as a filtering step showed improved performance on the validation set, however, did not work so well on the actual test set. As future work, using hybrid solution with the help of LLMs can improve the system performance and address the issues with the disambiguation of the term usage in context.
This work was partially supported by the European Union-NextGenerationEU, through the National Recovery and Resilience Plan of the Republic of Bulgaria [Grant Project No. BG-RRP-2.004-0008] and by Horizon Europe research and innovation programme project RES-Q plus [Grant Agreement No. 101057603], funded by the European Union. Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or the Health and Digital Executive Agency. Neither the European Union nor the granting authority can be held responsible for them.
The methods described in this paper were submitted as part of the MultiCardioNER challenge and were validated only on the challenge datasets. Further investigation on different datasets is needed to explore the generalizability of the approach. The specific labeling approach on the datasets impacts the model performance, as in the case of drug NER the inconsistencies of dosage labelling was a source of errors. In different settings, the labelling guidelines used might be different and therefore the presented approach may not perform as well.
For fine-tuning the models we used the following hyperparameters settings:
• learning rate: We used AdamW [34] optimizer with between 1-e5 and 5-e5 learning rate.
• number of epochs: We experimented between 5-20 epochs depending on how long it took the model to converge. • batch size: Initialized to 8 with gradient accumulation steps 1-2 giving an effective batch size of 8-16 due to GPU memory limitations). • learning rate scheduler: linear For pretraining we used the following hyperparameter settings:
• learning rate: We used AdamW optimizer with 5-e5 learning rate.
• number of epochs: 3 epochs due to resource limitations. • weight decay: 0.01.
• batch size: Initialized to 8 due to GPU memory limitations.
• learning rate scheduler: linear