Performance disparities exist in Named Entity Recognition (NER) systems across languages due to variations in available human-annotated data. We participated in the MultiDrug subtask of MultiCardioNER, a shared task focusing on multilingual NER for cardiology, to compare the efectiveness of fine-tuning BERT-based monolingual and multilingual language models, and prompting Large Language Models (LLMs) for drug entity recognition across multiple languages. Our findings demonstrate that monolingual BERT models pretrained on biomedical corpora generally outperform their multilingual counterparts. However, for languages lacking access to a broader range of pretrained models, combining the translation capability of LLM [1, 2, 3, 4] with the best-performing pretrained monolingual BERT model yielded superior results. This approach efectively reduces the resource disparity while leveraging domain-specific knowledge captured by the monolingual BERT model. Our best systems in the MultiCardioNER track yielded F1-scores of 0.9277 for Spanish, 0.9107 for English, and 0.8776 for Italian. We highlight the comparative advantages of domain-specific fine-tuning and LLM-powered language translation for multilingual drug NER.
Named entity recognition (NER) is one of the central tasks in natural language processing (NLP),
particularly in specialised domains such as biomedicine [
For the MultiDrug subtask [
The DisTEMIST [
As a domain-specific dataset for drug NER, CardioCCC dataset is comprised of 508 cardiology clinical
case reports. The original Spanish documents were annotated under the same guidelines used for
DisTEMIST[
For systems based on the fine-tuning BERT-based language models, we transformed the datasets into BIO2 (Begin, Inside, Outside) format [14]. After tokenisation, original annotations indicating the start and end ofsets of drug mentions were converted so that each token was labeled with one of the following tags: • B (Begin): start of a named entity. • I (Inside): tokens within the entity.
• O (Outside): tokens that are not part of any entity.
BIO2 format is particularly efective for encoder-based language models like BERT [ 15], which predict the classification for each token individually [ 16]. This allows for accurate identification of the boundaries of each drug entity.
Our strategy for the MultiDrug subtask was structured around two key dimensions: assessing the comparative efectiveness of fine-tuning encoder-based language models versus prompting generative LLM for NER, and exploring diferent multilingual processing approaches. For the latter, we evaluated the results from multilingual BERT model compared to those achieved by translating NER results from one language to others.
Our main strategy involved fine-tuning BERT-based language models [ 15, 18, 20] specifically for drug NER. We explored a range of models, testing both general-purpose and domain-specific pretrained language models of various parameter sizes. Only the training set from DrugTEMIST corpora was used for fine-tuning. We then selected the best-performing models based on their performance on the development set derived from the CardioCCC corpus.
As a baseline approach, we utilised pretrained monolingual BERT-based models in Spanish, English, and Italian. This method ensures that each model is attuned to the linguistic nuances specific to a language, but has limitations in addressing resource disparities across diferent languages [ 20]. Each of these models was trained on the respective monolingual training set. As an alternative approach, we ifne-tuned a multilingual BERT model on a combined training set of all three languages.
In System 1, in view of the multilingual nature of the given dataset, we opted for using a pretrained multilingual BERT-based language model. Specifically, we employed “google-bert/bert-base-multilingual-cased” [15]. To enhance its performance for drug entity recognition, we further fine-tuned the model on the BIO2-formatted data in English, Italian, and Spanish provided by the organiser. Each document was split into individual sentences and a batch size of 32, learning rate of 8e-5, weight decay of 1e-5, and 5 epochs were employed during fine-tuning. Hyperparameter optimisation was not performed.
Given the limited range of available multilingual BERT models, particularly the lack of biomedical domain-specific pretrained models, we investigated the efectiveness of fine-tuning independent monolingual models in Systems 2 and 3. This approach aimed to improve performance by leveraging language-specific pretrained biomedical models. However, availability of pretrained models difered across languages. While a wider range of selection was available for English and Spanish, a limited range of pretrained models available for Italian. Consequently, for Italian, we opted for “dbmdz/bert-base-italian-xxl-cased” [19]. To evaluate the eficacy of both general and domainspecific models available for the other two languages, we used the SeqEval library. This resulted in using “allenai/scibert_scivocab_cased” [18] for English and “PlanTL-GOB-ES/bsc-bio-ehr-es” [17] for Spanish. Hyperparameter search was not performed. Following the approach used for multilingual BERT, we applied sentence-level split of the data, a batch size of 32, learning rate of 8e-5, weight decay of 1e-5, and 5 epochs for all monolingual models.
In light of recent advancements in general-purpose LLMs, we explored their potential in biomedical multilingual NER. We employed two distinct strategies. The first strategy involved directly using the LLM for NER, bypassing the fine-tuning process. This approach aimed to assess the efectiveness of prompting LLM for multilingual NER tasks without domain-specific adaptation. The second strategy leveraged the best-performing monolingual BERT-based model for NER, while utilising LLM as a translation module to convert the NER results in a given language into the target language. This approach investigated the potential use of LLM to bridge the resource gaps among diferent languages while capitalising on the domain-specific knowledge captured by pretrained monolingual BERT models.
In addition to fine-tuning BERT-based models for drug NER, we also experimented with directly using generative LLM for NER task. This approach involved prompting the LLM to produce drug annotations from clinical case reports without the conventional fine-tuning process. While this method takes advantage of the contextual understanding and embedded domain knowledge of LLMs [21, 22], it is recognised in existing literature that LLMs often fall short in achieving comparable performance on specific non-generative tasks such as NER [23].
Figure 1 illustrates an example of prompts used for LLM drug entity recognition. Our experiments were conducted in a zero-shot setting on the English development set, where we supplied the complete clinical text and tasked the LLM with identifying all drug mentions. We included an instruction in the prompt to output the predictions in JSON format for eficient post-processing. We experimented with three LLMs, Meta-Llama-2-7B, Meta-Llama-3-8B, and gpt-3.5-turbo. Llama-2-7B produced results that were significantly lower than the average scores of fine-tuned models, with a precision of 0.6689, recall of 0.1964, and F1 score of 0.3037. This outcome was expected due to the limited parameter size and the zero-shot setting. On the other hand, GPT-3.5 and Llama-3-8B demonstrated performances comparable to the average results from fine-tuned models (avg. precision: 0.8373; avg. recall: 0.8779; avg. F1: 0.8564). Comparing the two LLMs, GPT-3.5 achieved better recall (precision: 0.8236; recall: 0.8538; F1: 0.8384), while Llama-8B showed higher precision and F1 values (precision: 0.8767; recall: 0.8303; F1: 0.8529). LLaMA-3-8B achieved higher precision (0.8767) than the average precision of fine-tuned models (0.8373), which is noteworthy given that the LLaMA-3-8B model has significantly fewer parameters compared to GPT-3.5 and was used in a zero-shot setting. Further studies with a broader range of LLMs, as well as exploring few-shot settings and additional prompting methods, will help better understand the potential use of LLM in drug NER. {"role": "system", "content": "Your job is to review a clinical note that potentially contains mentions of drug names."}, {"role": "user", "content": "Find all mentions of drug names in the following clinical note. Output your response in JSON format with keys 'drug 1', 'drug 2', and so on.
Clinical Note: ANAMNESIS 46-year-old Spanish woman.
Married, with an 18-year-old daughter.
Good family support. ..."}
Figure 1 – Example LLM Prompt for NER 3.2.2. Entity Translation {"role": "user", "content": "I have Spanish drug names 'fluoxetina', 'clonazepam'. Find the corresponding drugs in English in the following clinical note. Output your response in JSON format, where the keys are the given Spanish drug names ('fluoxetina', 'clonazepam'), and the values are the corresponding drug names in English found in the clinical note. Note that for every Spanish drug name, there is always at least one mention of the corresponding drug in English.
Clinical Note:
Francisca Valero, a 33-year-old stock market analyst, married with two children,
was brought to the emergency department (ED) after 10 days of what her husband
described as ..."}
multilingual NER tasks. Figure 2 shows an example of prompts used for the LLM-powered translation.
In System 4, we generated predictions for the Italian test set by translating the predictions from the
best-performing Spanish monolingual model into Italian using GPT-3.5. For the English test set, we
retained predictions from the English monolingual model. In System 5, predictions for both the English
and Italian test sets were derived from the Spanish predictions and translated into the respective
languages using GPT-3.5. This approach tests the feasibility of using an LLM-based translation system
for multilingual NER tasks, especially when there is a disparity in available resources for each language,
as an alternative to directly applying LLM for NER. [
text FARMACO FARMACO FARMACO FARMACO FARMACO FARMACO ground truth labels. System selection for final submission was based on performance on the development set (Table 2). The precision, recall, and F1 metrics are calculated based on a strict matching with the ground truth. In this approach, an exact match between the predicted and ground truth intervals is required to consider a true positive. The relaxed F1 score is computed based on a more lenient criterion. A true positive is counted if either the start of the prediction falls between or coincides with one of the start and end intervals present in the ground truth, or if a ground truth start falls between or coincides with one of the start and end intervals present in the prediction. This approach allows for some degree of imprecision in the predicted intervals while still considering them as correct. The relaxed F1 aims to understand the model predictions and if post-processing rules can be found to reduce the diference between the strict and relaxed scores.
• Spanish: Our monolingual BERT model achieved the top recall and F1 score (0.9277) among all participant submissions (mean F1: 0.6373; median F1: 0.8502). • English: While our monolingual BERT model showed the highest precision, its F1 score (0.9107) was marginally lower than the best overall system (best F1: 0.9223; mean F1: 0.7101; median F1: 0.8768). • Italian: In Italian, where model selection may be limited, we combined LLM translation with our Spanish monolingual system. This combined approach achieved the highest precision, while our monolingual Italian system showed the highest recall. Ultimately, the F1 score of our best Italian system (0.8776) was somewhat lower than the top-ranked submission (best F1: 0.8842; mean F1: 0.6506; median F1: 0.8421).
These results highlight the efectiveness of fine-tuning monolingual BERT models, particularly for high-resource languages with large amount of available domain-specific training data. The combination of LLM translation and monolingual models from other languages shows promise for low-resource languages (i.e., Italian) and requires further exploration.The results of our five submissions on the test set can be found in Table 3 alongside the best score for Precision, Recall and F1, the mean F1 and the median F1. bsc-bio-ehr-es cased bsc-bio-ehr-es cased + pp bert-base-spanish-wwm-uncased BERT multilingual base model cased BERT multilingual base model cased + pp bert-base-italian-xxl-cased bert-base-italian-xxl-cased + pp BERT multilingual base model cased BERT multilingual base model cased + pp BERT multilingual base model cased BERT multilingual base model cased + pp
While the relaxed F1 scores in Table 2 are encouraging, we analysed the entities that overlapped partially with the gold standard annotations, suggesting a potential mismatch between system predictions and the human-annotated ground truth. We observed that the annotation guidelines provided to human annotators allowed for the inclusion of consecutive drug entities separated by specific delimiters like ‘/’, ‘and’ (corresponding to ‘y’ in Spanish and ‘e’ in Italian), ‘of’ (corresponding to ‘de’ in Spanish and ‘di’ in Italian), ‘+’, and ‘/ ’. This specific format caused performance degradation for our system.
Interestingly, combining entities separated by ‘/’ and ‘of’ (with its translation in Spanish and Italian) resulted in improved performance, while the impact of other delimiters was negligible. Additionally, we implemented a rule to include the entire span of a word entity even if our model only predicted a subset of the words. This adjustment aimed to further reduce the discrepancy between predicted and gold standard annotations.
It is worth noting that our strategy did not involve training on the validation set. Only the training split of the dataset, which is derived from the DrugTEMIST dataset, was used for fine-tuning, while the development and test sets were obtained from CardioCCC. There was no significant diference between the models’ performances on the validation and test sets, suggesting that our systems likely did not exhibit signs of overfitting. Our analysis highlights the importance of considering relaxed F1 scores when evaluating NER systems. It also suggests that incorporating post-processing rules specific to the annotation guidelines employed can improve performance, particularly when dealing with specific entity formatting conventions.
In this study, we conducted a comparative analysis of multilingual drug NER in English, Italian, and Spanish. We evaluated the efectiveness of four distinct approaches: fine-tuning multilingual BERT, ifne-tuning monolingual BERT models for each language, zero-shot LLM prompting, and a hybrid method that combines monolingual BERT and LLM as a translation module. Our findings revealed that fine-tuning monolingual BERT models generally outperformed other approaches. Specifically, our system for the Spanish test set, based on fine-tuning a BERT-based monolingual Spanish language model pretrained on a biomedical corpus, achieved the top ranking among all participant submissions.
On the other hand, for Italian, where there is limited availability of domain-specific human-annotated data and consequently a smaller range of pretrained models, a hybrid approach of combining bestperforming monolingual model’s prediction and LLM translation demonstrated greater eficacy compared to results achieved by a monolingual Italian BERT model alone. This suggests that LLM can be used as a translation module to enhance the cross-lingual applicability of fine-tuned monolingual models.
Promising results were obtained with zero-shot LLM prompting, where LLaMA-3-8B achieved higher precision (0.8767) than the average precision of fine-tuned models (0.8373) on English development set. Future work will involve experimenting with a broader range of LLMs and prompting methods in few-shot settings.
C.L. was supported by the United Kingdom Research and Innovation (grant EP/S02431X/1), UKRI Centre for Doctoral Training in Biomedical AI at the University of Edinburgh, School of Informatics. For the purpose of open access, the author has applied a creative commons attribution (CC BY) licence to any author accepted manuscript version arising. J.M.P. and A.D.L. are supported by the CoDiet project. The CoDiet project is funded by the European Union under Horizon Europe grant number 101084642 and supported by UK Research and Innovation (UKRI) under the UK government’s Horizon Europe funding guarantee [grant number 101084642]. [13] S. Lima-López, E. Farré-Maduell, M. Krallinger, DrugTEMIST Guidelines: Annotation of Medication in Medical Documents, 2024. URL: https://doi.org/10.5281/zenodo.11065433. doi:10.5281/zenodo. 11065433. [14] L. Ramshaw, M. Marcus, Text chunking using transformation-based learning, in: Third Workshop on Very Large Corpora, 1995. URL: https://aclanthology.org/W95-0107. [15] J. Devlin, M.-W. Chang, K. Lee, K. Toutanova, Bert: Pre-training of deep bidirectional transformers for language understanding, arXiv preprint arXiv:1810.04805 (2018). [16] R. Sharma, D. Chauhan, R. Sharma, Named entity recognition system for the biomedical domain, in: 2022 17th Conference on Computer Science and Intelligence Systems (FedCSIS), IEEE, 2022, pp. 837–840. [17] C. P. Carrino, J. Llop, M. Pàmies, A. Gutiérrez-Fandiño, J. Armengol-Estapé, J. Silveira-Ocampo, A. Valencia, A. Gonzalez-Agirre, M. Villegas, Pretrained biomedical language models for clinical NLP in Spanish, in: Proceedings of the 21st Workshop on Biomedical Language Processing, Association for Computational Linguistics, Dublin, Ireland, 2022, pp. 193–199. URL: https://aclanthology.org/2022.bionlp-1.19. doi:10.18653/v1/2022.bionlp-1.19. [18] I. Beltagy, K. Lo, A. Cohan, Scibert: A pretrained language model for scientific text, in: EMNLP, Association for Computational Linguistics, 2019. URL: https://www.aclweb.org/anthology/D19-1371. [19] S. Schweter, Italian bert and electra models, 2020. URL: https://doi.org/10.5281/zenodo.4263142.
doi:10.5281/zenodo.4263142. [20] K. Hakala, S. Pyysalo, Biomedical named entity recognition with multilingual BERT, in: K. JinDong, N. Claire, B. Robert, D. Louise (Eds.), Proceedings of the 5th Workshop on BioNLP Open Shared Tasks, Association for Computational Linguistics, Hong Kong, China, 2019, pp. 56–61. URL: https://aclanthology.org/D19-5709. doi:10.18653/v1/D19-5709. [21] S. Wang, X. Sun, X. Li, R. Ouyang, F. Wu, T. Zhang, J. Li, G. Wang, Gpt-ner: Named entity recognition via large language models, arXiv preprint arXiv:2304.10428 (2023). [22] D. Ashok, Z. C. Lipton, Promptner: Prompting for named entity recognition, arXiv preprint arXiv:2305.15444 (2023). [23] I. Jahan, M. T. R. Laskar, C. Peng, J. X. Huang, A comprehensive evaluation of large language models on benchmark biomedical text processing tasks, Computers in Biology and Medicine (2024) 108189.