Biomedical Information Extraction from Natural Language Processing (NLP) is one of the newest challenges driving innovation in the biomedical scientific field. In this work, we present our implementation pipeline for the GutBrain shared task covering both Named Entity Recognition and Relation Extraction. For Subtask 6.1 (NER), we fine-tuned the GLiNER framework on expert-annotated GutBrain datasets, achieving robust entity recognition between the predefined categories. For the RE Subtasks (6.2.1-6.2.3), we injected entity markers into text and employed fine-tuned BiomedBERT and pubmed-bert classifiers to predict relations between entities. By exploring Precision-oriented, Recall-oriented, and balanced configurations, we identified the best setups for maximizing Precision, Recall, and F1 for each task. Finally, we show our results with scatter plots and discuss the trade-of each run ofers.
In the biomedical field, large volumes of textual data are generated daily, such as electronic health
records and biomedical literature. Extracting and structuring this information in an eficient way is
crucial for improving healthcare quality, supporting clinical decision-making, and advancing medical
research [
A key subtask of information extraction is Named Entity Recognition (NER), which involves
identifying and categorizing spans of text into predefined categories such as diseases, treatments, and
anatomical entities [
Based on NER, Relation Extraction (RE) identifies and extracts relationships between named entities
from the underlying content [
The CLEF 2025 conference is the 16th edition of the Conference and Labs of the Evaluation Forum
(CLEF), continuing the popular CLEF campaigns that have been running since 2000 and contributing to
the systematic evaluation of information access systems through experimentation with shared tasks1.
In particular, BioASQ 2025 Lab Task 6, known as GutBrainIE, promotes the development of Information
Extraction systems by extracting named entities and the relations between them2 [
The challenge is divided into 4 Subtasks:
dietary supplement human Does ⏞Fibre⏟-fix provided to ⏞peo⏟ple, with i⏞rritable bow⏟el syndrome, who are consuming a low target microbiome
FODMAP diet, improve their gut health, ⏞gut micr⏟obiome sleep and mental health? • Subtask 6.1, the participants are provided with PubMed abstracts about the gut-brain axis focusing on the Parkinson’s disease and mental health. Their task is to classify specific entity mentions into one of the 13 predefined categories: anatomical location, animal, biomedical technique, bacteria, chemical, dietary supplement, disease disorder or finding (DDF), drug, food, gene, human, microbiome, and statistical technique. An example is shown in Table 1. • The next challenges concerns RE over the entities identified by the NER systems: – Subtask 6.2.1 is the Binary Tag-Based RE, where participants are asked to identify which entities are in relation within a document. An example is shown in Table 2. – Subtask 6.2.2 is the Ternary Tag-Based RE: participants are required to identify the actual entities involved in a relation and predict the type of relation. An example is shown in Table 3. – Finally, Subtask 6.2.3, the Ternary Mention-Based RE, where participants are required to identify the actual entities involved in a relation and predict the type of relation. An example is shown in Table 4.
For more detailed information about the GutBrainIE task and the proposed subtasks, please refer to the
overview paper [
Early approaches to NER and RE relied on hand-crafted rules, which were later replaced by probabilistic
models such as Hidden Markov Models (HMMs) [
RE first appeared prominently in SemEval-2010 Task 8 [ 14], which focused on the “Relation Classification Subtask" and assigned a single label to a marked entity pair. The deep learning era began with convolutional neural networks (CNNs), which enabled mapping entire sentences to relation labels without manual feature engineering [15]. Subsequently, the advent of pre-trained language models (PLMs) marked a major advancement in RE, as highlighted by Li et al. [16]. These fine-tuned models, such as BERT and RoBERTa, trained on diverse datasets, have become standard for many modern NLP tasks.
NER models are widely used in data mining, textual analysis, and text processing. However, they often lack flexibility, and training them can be a challenging task. To address these limitations, Zaratiana et al. [17] proposed GLiNER, a recent and efective alternative to traditional NER models, which are typically restricted to predefined entity types and rely on expensive Large Language Models (LLMs). GLiNER is a NER model capable of recognizing a wide range of entity types using a Bidirectional Transformer architecture.
The model employs a Bidirectional Language Model (BiLM) and takes as input a set of entity type prompts and a sentence or text, with each entity separated by a learned token [ENT]. The BiLM outputs representations for each token. Entity embeddings are passed into a FeedForward Network, where input word representations are passed into a span representation layer to compute embeddings for each span. Finally, it computes a matching score between entity representations and span representations (using dot product and sigmoid activation) [17]. Figure 2 shows the overall architecture.
In our implementation to perform NER on the GutBrain datasets, we used NuNER Zero, a zero-shot NER model3. NuNER Zero is based on the GLiNER architecture and expects input as a concatenation of entity types and the target text. It was trained on the NuNER-v2.0 dataset4, which combines annotated subsets of the Pile5 and C46 corpora. Annotations were generated using large language models following the NuNER procedure, which employs GPT-3.5-turbo to label entity mentions in a large-scale English corpus (C4) [18] with semantically meaningful concepts. The LLM was prompted to extract as many relevant entities as possible from each sentence and assign them to one of approximately 200k unique concepts (e.g., “wellness”). This annotation process resulted in over 4.3 million labeled entities, providing high conceptual diversity but also showing class imbalance and ambiguities, which were addressed through filtering and training procedures [19].
Our implementation pipeline follows these steps: • Data analysis: we begin by examining the composition of the GutBrain corpus (titles, abstracts, and entity annotations), dropping any annotations whose location is not "title" or "abstract". • Data loading: load the JSON files containing metadata and character-level entity labels.
• Tokenizer initialization: initialize the microsoft/deberta-v3-large tokenizer7, which splits the text into tokens and provides ofset mappings for model input. • Preprocessing: combine the title and abstract for each document; convert character-level entity spans into token-level spans using the tokenizer’s ofset mappings; store the tokenized text and corresponding entities. • Hyperparameter configuration : define the number of training steps based on dataset size, along with the maximum number of epochs, batch size, and maximum token length. • Model setup: initialize the token classification model numind/NuNER_Zero based on GLiNER with pre-trained weights, and configure sampling parameters to manage training complexity. • Training loop: iterate over batches, perform forward passes, compute loss, and apply backpropagation. Update model weights using the AdamW optimizer8. • Evaluation: periodically evaluate model performance on dev.json, reporting Micro and Macro
Precision, Recall, and F1 scores.
To predict NER entities with our trained models, we structured our implementation pipeline in this way: for each article, we ran the model separately on its title and abstract, requesting predictions for all thirteen entity types (e.g., animal, anatomical location, DDF). The raw output consisted of a list of spans with start and end ofsets, labels, and confidence scores. We then merged any adjacent spans with the same label and converted the character-level ofsets into the required format. Finally, we saved the predictions in a JSON file, including the (start_idx) and (end_idx) fields.
For the other RE Subtasks of the GutBrainIE challenge, we employed two diferent pretrained models: NeuML/pubmedbert-base-embeddings9, a fine-tuned model based on sentencetransformers trained on a dataset of randomly sampled PubMed10 title–abstract pairs and similar titles; and microsoft/BiomedNLP-BiomedBERT-base-uncased-abstract-fulltext11, a biomedical 7https://huggingface.co/microsoft/deberta-v3-large 8https://pytorch.org/docs/stable/generated/torch.optim.AdamW.html 9https://huggingface.co/NeuML/pubmedbert-base-embeddings 10https://pubmed.ncbi.nlm.nih.gov/ 11https://huggingface.co/microsoft/BiomedCLIP-PubMedBERT_256-vit_base_patch16_224 model pre-trained from scratch using PubMed abstracts and full-text articles from PubMed Central12 [20]. Our pipeline for these Subtasks is as follows: • Data loading: we first load the JSON files containing the GutBrain datasets (platinum, gold, and development collections) and extract article texts, entity annotations, and relation annotations. We then initialize the microsoft/BiomedNLP-BiomedBERT-base-uncased-abstract-fu lltext tokenizer to prepare the input for the neural network, inserting special entity markers—[E1]. . . [/E1] for subjects and [E2]. . . [/E2] for objects—into the raw text. • Preprocessing: split the entities between the title and the abstract, and filter for relations involving entities from both. • Model configuration : extend the BERT embedding layer to recognize the newly introduced entity markers. Instantiate the BiomedBERT model for sequence classification, where each input sequence (text with entity markers) is processed to determine the presence of a binary relation. • Training setup: configure the AdamW optimizer and learning rate scheduler, and define the hyperparameters used to train the model, including number of epochs, batch size, learning rate, and sequence length. • Model training: in each training step, perform a forward pass, compute the loss, backpropagate gradients, and update model weights via the optimizer and scheduler. • Validation: after each epoch, evaluate model performance on the test split, reporting Micro and
Macro Precision, Recall, and F1-score.
To predict relations between entities in the test dataset, we implemented the following pipeline. Starting from the named entities identified in Subtask 6.1, we generate all ordered pairs of distinct entities (1, 2). We insert special markers ([E1], [E2]) into the raw text to highlight the subject and object, respectively. The marked text is then tokenized into a fixed-length sequence and converted into BERT input tensors. We feed the encoded sequence into a fine-tuned BertSequenceClassification model, whose output logits corresponds to the relation labels defined in the label2id.json file. Finally, we apply the softmax function, select the top-scoring label with a confidence above 0.7, and save the predictions.
• The project source code is available in the ataupd2425-gainer repository on GitHub: https://github.com/Vezzero/ataupd2425-gainer. • The dataset collection was provided by the CLEF BioASQ 2025 organizers and is available at: https://hereditary.dei.unipd.it/challenges/gutbrainie/2025/. • The evaluation script used is evaluation.py, made available by the organizers via their oficial GitHub repository: https://github.com/MMartinelli-hub/GutBrainIE_2025_Baseline/blob/main/Eval/evaluate.py.
Full details about provided data, baselines, and evaluation can be found in the overview paper [
A README file is provided in the GitHub repository with complete reproducibility instructions.
In this section, we report and describe the results obtained from the evaluation on the test data.
The runs submitted have been evaluated by the organizers on a held-out test set of 40 expert-annotated articles. These annotations are used as ground truth to compute the performance metrics.
For the NER Subtask, we trained three variants of the numind/NuNER_Zero model: PironA, PironD, and PironS. All models share the same architecture but difer in the training dataset, number of epochs, and training steps as reported in Table 5.
The NER test results are reported in Table 6. The PironA model with the ma run achieves the highest Micro-F1 score with high Micro-Precision and Micro-Recall. In particular, its Micro-Precision indicates that almost all entities are labeled correctly, resulting in the fewest overall errors. PironD with the md run obtains the highest Macro-Recall with lower Precision, suggesting it captures more true instances while generating more false positives. In contrast, PironS with the last run, ms, performs worst on both Micro and Macro scores, likely a consequence of the noisier annotations in the silver dataset.
For the RE Subtasks, we trained four diferent models, finetuning the BiomedNLP-BiomedBERT-base -uncased-abstract-fulltex13 and NeuML/pubmedbert-base-embeddings14. BiomedBERT was fine-tuned under three diferent configurations, each varying only in training data and hyperparameters. Table 7 reports the models setup.
As explained in Section 3, we base our RE models on the entity spans produced by our three NER variants (ma, md, and ms). For each test document, we first apply one of these variants to identify and save entities and then feed them into the following RE model to generate the final relation labels. 13https://huggingface.co/microsoft/BiomedNLP-BiomedBERT-base-uncased-abstract-fulltext 14https://huggingface.co/NeuML/pubmedbert-base-embeddings 5.2.1. Binary Tag-Based RE Results Table 8 reports both macro- and micro-averaged metrics for all submitted runs. The bp run, trained on the combined platinum, gold, and dev relation sets, with the BinaryTGDNeuml classifier and the NER predictions of the ma run, achieves the highest Macro F1 and Micro F1. We attribute this strong performance to its underlying BiomedBERT encoder: initially pre-trained on millions of PubMed abstracts and then fine-tuned in a sentence-transformers framework, it produces 768 dimensional embeddings that encode biomedical semantics. These more accurate representations help the model distinguish valid relations and invalid ones better and assign higher confidence scores to its predictions. The bd runs group, based on the NER prediction in the md run, shows a high Macro and Micro Recall but low Macro and Micro Precision, suggesting it is a Recall-oriented model. The ba group shows the opposite: higher Macro and Micro Precision but lower Macro and Micro Recall, suggesting it is a Precision-oriented model. The bs runs group had the lower Micro and Macro Precision (even if the Macro and Micro Recall are similar to the bd group). This shows again that the silver dataset’s noise negatively influences the Precision value.
In the scatter plot, Figure 3, each point represents a run, with Micro Precision on the x-axis and Micro Recall on the y-axis. The Recall-oriented bd/bs runs group cluster in the top-left with high Recall and low Precision. The Precision-oriented ba group is located in the mid-left. The balanced ones, bp group, appears toward the bottom-right, corresponding to their highest Micro F1. 5.2.2. Ternary Tag-Based RE Results For the Subtask 6.2.2, we leveraged our PironBinary models, updating the inference pipeline to predict and return relations between entities. Table 9 reports the performance of each run.
The td run, obtained using the PironBinaryTGD model and the ma run, achieves the highest Micro Precision, resulting in the highest Micro F1. This suggests that the run correctly predicted nearly all relations between entities. In contrast, the ts run shows the best score in Macro Precision, as both rare and common relations contribute equally to the final average, leading to the best Macro F1. This comparison highlights that PironBinaryTGD provides the highest overall accuracy in the td configuration, while PironBinaryTGS, in the ts setup with fewer positive examples, excels in achieving balanced performance across all relation classes. 5.2.3. Ternary Mention Based RE Results Finally, Table 10 reports the performance of all 6.2.3 runs. The ts configuration, using the PironBinaryTGS model with the entities of the ma run, achieves the highest Micro and Macro F1, making it the best run overall.
To explore how each run balances false positives and false negatives, Figure 4 plots Micro Precision against Micro Recall. In this scatter plot, the tma runs cluster in the upper left, with high Precision but low Recall, indicating they efectively avoid false positives but miss many true relations. The tmd group shows a well-balanced trade-of between Precision and Recall. The tms runs overlap with the tmd seeds, but the aggregated tms shifts into the top right quadrant, achieving the best combination of Micro Precision and Micro Recall—and consequently, the highest Micro F1.
In this paper, we presented the implementation pipelines for biomedical information extraction in the GutBrain corpus, covering both NER and RE. For Subtask 6.1, we fine-tuned the NuNer_Zero model under three hyperparameter configurations, achieving an almost 80% Micro F1 score. The chosen datasets used to train the model impacted the run performance, enabling us to see that the silver collections contained noisy annotations and leading to the worst results. In Subtasks 6.2.1, 6.2.2, and 6.2.3, we carried out RE as a marker-based sequence classification problem, using PironBinary models trained using the
BiomedBERT base model. We were able to produce Precision-oriented, Recall-oriented, and balanced variants, each performing the best in diferent Subtasks, but without a dominating configuration.
Looking ahead, we plan to investigate and test new LLMs’ models and more noise-robust training techniques, such as confidence-aware loss functions, to improve both NER and RE performance. In addition, we aim to integrate a semantic perspective grounded in linguistic analysis to enrich the linguistic and conceptual interpretation of extracted terms and relations. Specifically, we would like to apply semic analysis, which decomposes terms into minimal semantic units, as a structured approach to uncovering the internal organization of meaning in medical terminology [21, 22]. Incorporating this technique may enhance our ability to align terminological outputs with underlying conceptual structures, improving not only model interpretability but also the precision of information retrieval in domain-specific biomedical contexts.
This work is partially supported by the HEREDITARY Project, as a part of the European Union’s Horizon Europe research and innovation programme under grant agreement No GA 101137074.
[12] T. Mikolov, I. Sutskever, K. Chen, G. Corrado, J. Dean, Distributed representations of words and phrases and their compositionality, 2013. URL: https://arxiv.org/abs/1310.4546. arXiv:1310.4546. [13] G. Lample, M. Ballesteros, S. Subramanian, K. Kawakami, C. Dyer, Neural architectures for named entity recognition, 2016. URL: https://arxiv.org/abs/1603.01360. arXiv:1603.01360. [14] I. Hendrickx, S. N. Kim, Z. Kozareva, P. Nakov, D. Ó Séaghdha, S. Padó, M. Pennacchiotti, L. Romano, S. Szpakowicz, SemEval-2010 task 8: Multi-way classification of semantic relations between pairs of nominals, in: K. Erk, C. Strapparava (Eds.), Proceedings of the 5th International Workshop on Semantic Evaluation, Association for Computational Linguistics, Uppsala, Sweden, 2010, pp. 33–38. URL: https://aclanthology.org/S10-1006/. [15] D. Zeng, K. Liu, S. Lai, G. Zhou, J. Zhao, Relation classification via convolutional deep neural network, in: J. Tsujii, J. Hajic (Eds.), Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers, Dublin City University and Association for Computational Linguistics, Dublin, Ireland, 2014, pp. 2335–2344. URL: https://aclanthology.org/ C14-1220/. [16] J. Li, T. Tang, W. X. Zhao, J.-Y. Nie, J.-R. Wen, Pretrained language models for text generation: A survey, 2022. URL: https://arxiv.org/abs/2201.05273. arXiv:2201.05273. [17] U. Zaratiana, N. Tomeh, P. Holat, T. Charnois, Gliner: Generalist model for named entity recognition using bidirectional transformer, 2023. URL: https://arxiv.org/abs/2311.08526. arXiv:2311.08526. [18] C. Rafel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, P. J. Liu, Exploring the limits of transfer learning with a unified text-to-text transformer, Journal of Machine Learning Research 21 (2020) 1–67. URL: http://jmlr.org/papers/v21/20-074.html. [19] S. Bogdanov, A. Constantin, T. Bernard, B. Crabbé, E. Bernard, Nuner: Entity recognition encoder pre-training via llm-annotated data, 2024. arXiv:2402.15343. [20] Y. Gu, R. Tinn, H. Cheng, M. Lucas, N. Usuyama, X. Liu, T. Naumann, J. Gao, H. Poon, Domain-specific language model pretraining for biomedical natural language processing, 2020. arXiv:arXiv:2007.15779. [21] V. Bonato, G. M. Di Nunzio, F. Vezzani, A Novel Approach to Semic Analysis: Extraction of Atoms of Meaning to Study Polysemy and Polyreferentiality, Languages 9 (2024) 121. URL: https: //www.mdpi.com/2226-471X/9/4/121. doi:10.3390/languages9040121, number: 4 Publisher: Multidisciplinary Digital Publishing Institute. [22] V. Bonato, G. M. Di Nunzio, F. Vezzani, Preliminary Considerations on a Systematic Approach to Semic Analysis: The Case Study of Medical Terminology, Umanistica Digitale (2021) 211–234. URL: https://umanisticadigitale.unibo.it/article/view/12621. doi:10.6092/issn.2532-8816/12621, number: 10.