In recent years, we have witnessed the rise of powerful Large Language Models (LLMs) and their flexibility in accomplishing a wide range of NLP tasks, often achieving state-of-the-art (SOTA) accuracy. However for Named Entity Recognition (NER), there is a specific need for token-level class assignments and bidirectional context, both preceding and following a token, to understand its role. As a result, bidirectional encoder-style transformer models (BERT-like models) have been the standard approach. However, fine-tuning these models on available datasets often faces the bottleneck of limited training data. In this paper, we propose an alternative approach that leverages the extensive knowledge base of decoder-style Transformer models. These modern LLMs are typically trained on vast amounts of text, which enables them to overcome the challenge of limited labeled data. Instead of training from scratch, we focus on aligning the responses of these LLMs to suit NER. To this end, we use two methods: (i) few-shot prompting, and (ii) fine-tuning on available examples. Our findings indicate that ifne-tuning significantly outperforms prompting for biomedical NER, efectively aligning LLM outputs to the desired task outputs. Additionally, we propose an algorithm to parse the output of the LLM to extract relevant entities, their labels, and their start and end indices.
The GutBrainIE2025 challenge [
A competitive baseline model was provided to participants. This was a fine-tuned version of the
NuNER Zero model [
Recent advances in large language models (LLMs) have demonstrated significant promise for named
entity recognition (NER) tasks across diverse domains. Several studies have explored the application
of LLMs to NER, with notable successes in both general and specialized contexts. Xie et al. [
Chen et al. [13] provided a comprehensive systematic evaluation of four large language models (GPT-3.5, GPT-4, LLaMA 2, and PMC LLaMA) across 12 biomedical NLP datasets, including named entity recognition tasks under various settings. Additionally, domain-specific applications have shown promising results, with Jung et al. [14] demonstrating high performance using LLM-based biological named entity recognition on scientific literature related to p53 protein research, and Bian et al. [ 15] achieving strong one-shot biomedical named entity recognition performance through a two-step approach incorporating domain-specific knowledge.
When using Large Language Models (LLMs) for the NER task, it is crucial to adapt the task format in a way that aligns with the natural strengths of LLMs. Since LLMs are fundamentally next-token predictors, they are not inherently well-suited for pinpointing the exact start and end indices of entity spans, even if they can correctly identify and classify the entities themselves. Therefore, it becomes necessary to reformulate NER as a generative task, a paradigm where LLMs excel.
GPT-NER [16] introduced a novel annotation strategy that adapts NER to suit LLMs. It transforms
the task into a token prediction problem by marking the beginning and end of an entity span with
special tokens: @@ for the start and ## for the end. The entity’s class label immediately follows the
span-end marker ##, resulting in the format: @@<text_span>##<class_label>. This structured
output makes it straightforward to parse the model’s predictions and extract entities along with their
labels. While GPT-NER focuses solely on entity recognition and classification, without requiring start
and end character indices, the GutBrainIE challenge [
The organizers provided four datasets with varying annotation quality: Platinum, Gold, Silver, and Bronze, in descending order of reliability. The Bronze annotations were automatically generated using the fine-tuned baseline model. Since the Platinum and Gold datasets together included a suficient number of annotated articles (111 + 208 = 319), we used their combination as our training set. We excluded the Silver and Bronze datasets to maintain high data quality.
There are currently many strong LLMs available, both open-source and proprietary, accessible via APIs. For this challenge, we chose the GPT-4.1-mini from OpenAI, for several reasons: the GPT-4.1 series excels at instruction following, code generation, and handling long-context tasks; and importantly, GPT-4.1-mini is also significantly more cost-efective at approximately 84% cheaper than GPT-4o, while achieving nearly equivalent scores on benchmarks such as MMLU [17].
We adopted the annotation strategy introduced by GPT-NER [16]. Each input to the LLM consisted of a text segment (either from the title or abstract of a PubMed article) along with the list of 13 entity classes. The LLM was instructed to reproduce the original input text exactly, with the only modification being the insertion of entity annotations in the following format: @@<text_span>##<class_label>. The instructions emphasized that all characters—letters, spaces, and special Unicode characters—must be preserved exactly, apart from the added annotation tokens. This consistency is critical for accurately calculating the start and end character indices of each entity.
To provide the model with relevant in-context examples, we moved beyond a fixed set of few-shot examples and instead implemented a dynamic Retrieval-Augmented Generation (RAG) strategy. This approach ensures that the examples included in the prompt are contextually similar to the input text, thereby ofering more targeted guidance to the LLM. The RAG process involved three main steps:
First, we processed the entire training set (319 articles) to create a searchable knowledge base. For each title and abstract in the training data, we generated a dense vector embedding using OpenAI’s text-embedding-3-large model. These embeddings, along with their corresponding text and metadata (PubMed ID, example ID), were then indexed using the FAISS (Facebook AI Similarity Search) [18] library, an eficient similarity search library developed by Meta. We specifically utilized an IndexFlatL2 index, which performs an exhaustive search by calculating the L2 (Euclidean) distance between the query vector and all vectors in the index. While computationally intensive for massive datasets, this index guarantees the retrieval of the exact nearest neighbors, making it ideal for our moderately-sized corpus where accuracy is paramount.
When a new input text (from the validation or test set) required annotation, we first generated its embedding using the same text-embedding-3-large model. This query embedding was then used to search the FAISS index to retrieve the top-k most similar examples from our training set.
The retrieved examples were then used to construct the final prompt for the LLM. For each retrieved example, we generated the expected annotated output based on the ground-truth labels. These inputoutput pairs were then prepended to the prompt that contained the new text to be annotated. The number of examples included was based on the prompting configuration: • 0-shot: No examples were retrieved or included. • 1-shot: The single most similar abstract example was included. • 3-shot: The top two most similar abstract examples and the single most similar title example were included. • 5-shot: The top three most similar abstract examples and the top two most similar title examples were included.
This balance ensured the model was exposed to both abstract and title texts, which difer in length and structure along with this RAG-based strategy ensured that the model was always primed with highly relevant examples that mirrored the structure and content of the target text. However, as detailed in the Results section, this dynamic few-shot approach still yielded significantly worse performance than the baseline model, which led us to explore fine-tuning.
As gpt-4.1-mini is a proprietary model, we don’t have access to its weights that we can manipulate ourselves using standard finetuning process, thus we have to use OpenAI’s fine-tuning framework, which requires a .jsonl file containing input-output pairs. For each training example, we used the entity spans and labels to generate the expected annotated output in the same format used during prompting. Both training and validation sets were prepared and uploaded to OpenAI’s API. Once fine-tuning was complete, the resulting model could be accessed via a dedicated API endpoint, just like OpenAI’s standard models. Post fine-tuning, we re-evaluated the model using the same prompting configurations (0, 1, 3, and 5-shot) to measure the improvements in entity recognition accuracy.
OpenAI’s fine-tuning framework allows users to configure a limited set of hyperparameters. The
three adjustable hyperparameters are: Batch size, Number of epochs, Learning rate multiplier. These
can either be manually specified or set to "auto", allowing OpenAI to select optimal values automatically.
Through experimentation, we identified a configuration that produced the best results for our use case:
The training prompt for each example included only the base instruction (i.e., 0-shot setup) and excluded
any in-context examples. Including 1-shot, 3-shot, or 5-shot examples during fine-tuning negatively
impacted recall and also increased token usage, leading to higher training costs. The train set included
examples only from Gold and Platinum sets from the dataset provided by GutBrainIE2025 [
The output generated by the LLM follows the annotation format described earlier. For each identified entity, the challenge requires extracting five key pieces of information: Location (title or abstract), Class label, Text span (the exact string constituting the entity), Start index (in the original text), End index (in the original text).
To achieve this, we developed a parsing algorithm that converts the annotated text into structured entity objects with these five attributes. Although the LLM is explicitly instructed to only insert the special tokens @@, ##, and the corresponding class label—while keeping all other characters unchanged—hallucinations can still occur, particularly with non-fine-tuned models. We observed such deviations only in outputs from the base GPT-4.1-mini, not in the fine-tuned version. Nevertheless, our parsing approach is designed to be robust against such inconsistencies and to recover valid entities wherever possible.
We first remove any malformed or extraneous occurrences of @@ or ## that do not conform to the valid pattern: @@<text_span>##<class_label>. This is done using regular expressions.
W again use regular expressions to extract all valid entity annotations from the output text. From each match, we extract: Group 1: the entity’s text span, and Group 2: the associated class label. To calculate the character indices, we maintain a cumulative ofset variable that tracks the total number of extra characters (@@, ##, and class labels) that have been inserted into the original text. Subtracting this ofset from the indices in the annotated text, we can extract the text span.
In cases where hallucinations occur, the calculated indices might not correctly align with the original text—typically because the entity string has been slightly altered. To recover from this, we apply a sliding window search strategy: First, we attempt to match the extracted text span in the original text by sliding it rightward up to 10 characters. If no match is found, we slide it leftward up to 10 characters. If a match is found during this process, we use that position as the corrected start index. If no match is found at all, we adjust the ofset to account for the mismatch and proceed to the next entity. This combination of pattern matching, ofset correction, and sliding window search ensures that we can reliably extract entity information even in the presence of minor hallucinations or deviations from the expected output format.
Our training set consisted of 319 articles, each containing both a title and an abstract. This yielded a total of 638 training examples (319 × 2). The total token count—including both the base prompt and annotated output—was approximately 500,000 tokens. OpenAI’s pricing for fine-tuning the GPT-4.1-mini model is approximately $5 per 1 million tokens. Therefore, training for one epoch (500K tokens) cost around $2.50. Inference with both the base and fine-tuned versions of GPT-4.1-mini is highly cost-efective, averaging about $1 per 1 million tokens, making this approach scalable for real-world applications.
All reported results in Table 1 are based on the validation set. Table 2 below shows the oficial evaluation results on the test set for our best-performing model: the fine-tuned GPT-4.1-mini (0-shot), as provided by the organizers. Our findings demonstrate a clear performance gap between the base and fine-tuned versions of GPT-4.1-mini.1 1All code for this work is available on GitHub at github.com/hpgupt/GutBrainIE-CLEF25.
The base model’s performance improves progressively with the number of few-shot examples: 5-shot prompting achieved the best results, followed by 3-shot, 1-shot, and 0-shot, which performed the worst. This trend suggests that the base model, having no task-specific tuning, benefits from additional in-context examples that help it better understand the task format.
Interestingly, the fine-tuned model shows the reverse trend: 0-shot prompting consistently outperforms 1-shot and 5-shot methods. 5-shot prompting, in particular, performs the worst among all configurations post-fine-tuning. This reversal likely occurs because the fine-tuning process itself exposes the model to hundreds of training examples. As a result, additional in-context examples during inference do not improve performance—and may even degrade it due to token budget constraints or overfitting to redundant patterns. Consequently, the 0-shot fine-tuned GPT-4.1-mini emerges as the best-performing model overall. It also ofers significant inference cost savings, since fewer tokens are consumed compared to multi-shot prompting strategies.
Even though the base model performs worse across the board, qualitative analysis reveals that its errors are often not due to a lack of understanding, but rather ambiguities in entity boundary definitions. For example, consider the training annotation: The span "secretory IgA (SIgA)" is labeled in the dataset as two separate entities: "secretory IgA" and "SIgA", both tagged as Gene. However, the base model often predicts the entire phrase "secretory IgA (SIgA)" as a single entity. While this is not incorrect from a semantic perspective, it fails to match the exact annotated boundaries, leading to penalization in standard evaluation metrics like precision and recall. The fine-tuning process helps the model learn these annotation-specific conventions, allowing it to mimic the labeling patterns found in the training set more closely. This alignment with annotator intent significantly boosts its performance metrics.
Finally, we can conclusively state that our best model—the fine-tuned GPT-4.1-mini (0shot)—outperforms the baseline provided by the challenge organizers. However, it is important to note that the improvement in evaluation metrics is marginal. This raises an important consideration regarding eficiency vs. performance. The baseline model (a fine-tuned NuNER/GLiNER encoder) consists of millions of parameters, while GPT-4.1-mini is a much larger decoder-style model with billions of parameters. As a result, the computational cost for inference with GPT-4.1-mini is significantly higher. In summary, if maximum accuracy is the primary goal, then LLMs like GPT-4.1-mini – especially when fine-tuned – ofer a promising path forward for biomedical NER. However, if eficiency and resource constraints are critical, then state-of-the-art encoder-based models like GLiNER remain highly competitive and more practical for deployment at scale.
While fine-tuned LLMs such as GPT-4.1-mini demonstrate strong potential for biomedical NER, several directions remain open for future research and optimization. These include exploring smaller, opensource LLMs by applying the same fine-tuning and prompting techniques used with larger models; this would assess whether comparable performance can be achieved with significantly lower computational cost, reduce reliance on proprietary systems, and clarify if data quality rather than model size is the primary bottleneck.
Furthermore, developing improved and more LLM-friendly annotation strategies beyond the GPTNER style is critical, as the design of annotation schemes profoundly impacts model performance for tasks like entity recognition and span extraction. Alternatively, investigating how LLMs perform under the traditional BIO (Beginning, Inside, Outside) tagging scheme could enable structured token-level classification and potentially better align LLM outputs with existing NER pipelines.
This work was supported in part by a seed award from the AI Innovation Institute (AI3) at Stony Brook University (State University of New York at Stony Brook).
During the preparation of this work, the authors used GPT-4o for grammar and spelling checks. After using this tool, the authors reviewed and edited the content as needed and takes full responsibility for the publication’s content. [9] E. F. Tjong Kim Sang, F. De Meulder, Introduction to the CoNLL-2003 shared task: Languageindependent named entity recognition, in: Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003, 2003, pp. 142–147. [10] S. Karamcheti, L. Orr, J. Bolton, T. Zhang, K. Goel, A. Narayan, R. Bommasani, D. Narayanan, T. Hashimoto, D. Jurafsky, C. D. Manning, C. Potts, C. Ré, P. Liang, Mistral - A Journey towards Reproducible Language Model Training, 2021. URL: https://github.com/stanford-crfm/mistral. [11] H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, et al., Llama: Open and eficient foundation language models, arXiv preprint arXiv:2302.13971 (2023). [12] M. S. Obeidat, M. S. Al Nahian, R. Kavuluru, Do llms surpass encoders for biomedical ner?, arXiv preprint arXiv:2504.00664 (2025). arXiv:2504.00664. [13] Q. Chen, Y. Hu, X. Peng, Q. Xie, Q. Jin, A. Gilson, M. B. Singer, X. Ai, P.-T. Lai, Z. Wang, et al., Benchmarking large language models for biomedical natural language processing applications and recommendations, Nature communications 16 (2025) 3280. [14] S. J. Jung, H. Kim, K. S. Jang, Llm based biological named entity recognition from scientific literature, in: 2024 IEEE International Conference on Big Data and Smart Computing (BigComp), 2024, pp. 433–435. doi:10.1109/BigComp60711.2024.00095. [15] J. Bian, J. Zheng, Y. Zhang, H. Zhou, S. Zhu, One-shot Biomedical Named Entity Recognition via Knowledge-Inspired Large Language Model, in: Proceedings of the 15th ACM International Conference on Bioinformatics, Computational Biology and Health Informatics, BCB ’24, Association for Computing Machinery, New York, NY, USA, 2024. doi:10.1145/3698587.3701356. [16] S. Wang, X. Sun, X. Li, R. Ouyang, F. Wu, T. Zhang, J. Li, G. Wang, Gpt-ner: Named entity recognition via large language models, arXiv preprint arXiv:2304.10428 (2023). arXiv:2304.10428. [17] OpenAI, Gpt-4 turbo and the future of ai assistants, 2024. URL: https://openai.com/index/gpt-4-1/, accessed: 2025-05-19. [18] Facebook AI Research, FAISS: Facebook AI Similarity Search, 2025. URL: https://github.com/ facebookresearch/faiss, accessed: 2025-05-19.
This section provides the detailed prompt used for instructing the LLM and a concrete example of a 1-shot prompt constructed using our Retrieval-Augmented Generation (RAG) approach. We are using a "title" example as input here as it is much shorter than "abstract".
Perform Named Entity Recognition on the biomedical text provided in the "Text" section using a specific inline annotation style.
Task Description: You are an expert Named Entity Recognition (NER) system specializing in biomedical texts related to the gut-brain axis. Your task is to identify and extract entities from the provided text. Identify all mentions of entities belonging to the predefined categories listed below. An entity can occur multiple times; treat each occurrence as a separate entity and mark them directly within the text. Mark the beginning of an entity's span with "@@" and the end of the span with "##" followed immediately by the entity's category label. Note - the output text should be exactly identical to the input text i.e all spaces , special characters/ unicode characters,html/markdown tags,etc (any character) also should be exactly the same, except for the added "@@, ## and entity label" annotations for each entity detected.
Text span of an entity means the actual words/characters that form the entity in the text. An entity's span can contain single or multiple words but never partial words. The format to follow while marking an entity with its label is @@<entity_text_span>##<label> Predefined Entity Categories (use these exact labels after ##): [ "anatomical_location", "animal", "biomedical_technique", "bacteria", "chemical", "dietary_supplement", "DDF", "drug", "food", "gene", "human", "microbiome", "statistical_technique" Note - DDF stands for Disease, Disorder, or Finding. The remaining categories refer to their conventional or scientific meaning.
Also, If the first word or first set of words in output belong to an entity then ensure to start the output with @@ and follow rest of instructions.
Follow the format shown in the detailed examples below precisely. ### Examples Input: The brain-gut axis has gained increasing attention due to its contribution to the etiology of various central nervous system disorders. This study aims to elucidate the hypothesis that schizophrenia is associated with disturbances in intestinal microflora and imbalance in intestinal metabolites. By exploring the intricate relationship between the gut and the brain, with the goal of offering fresh perspectives and valuable insights into the potential contribution of intestinal microbial and metabolites dysbiosis to the etiology of schizophrenia. In this study, we used a 16S ribosomal RNA (16S rRNA) gene sequence-based approach and an untargeted liquid chromatography-mass spectrometry-based metabolic profiling approach to measure the gut microbiome and microbial metabolites from 44 healthy controls, 41 acute patients, and 39 remission patients, to evaluate whether microbial dysbiosis and microbial metabolite biomarkers were linked with the severity of schizophrenic symptoms.
Here, we identified 20 dominant disturbances in the gut microbial composition of patients compared with healthy controls, with 3 orders, 4 families, 9 genera, and 4 species. Several unique bacterial taxa associated with schizophrenia severity. Compared with healthy controls, 145 unusual microflora metabolites were detected in the acute and remission groups, which were mainly involved in environmental information processing,metabolism, organismalsystems, and human diseases in the Kyoto encyclopedia of genes and genomes pathway. The Sankey diagram showed that 4 abnormal intestinal and 4 anomalous intestinal microbial metabolites were associated with psychiatric clinical symptoms. These findings suggest a possible interactive influence of the gut microbiota and their metabolites on the pathophysiology of schizophrenia.
Output: The brain-gut axis has gained increasing attention due to its contribution to the etiology of various @@central nervous system disorders##DDF. This study aims to elucidate the hypothesis that @@schizophrenia##DDF is associated with disturbances in @@intestinal microflora##microbiome and imbalance in @@intestinal metabolites##chemical. By exploring the intricate relationship between the @@gut##anatomical_location and the @@brain##anatomical_location, with the goal of offering fresh perspectives and valuable insights into the potential contribution of @@intestinal microbial and metabolites dysbiosis##DDF to the etiology of @@schizophrenia##DDF. In this study, we used a @@16S ribosomal RNA (16S rRNA) gene sequence-based approach##biomedical_technique and an @@untargeted liquid chromatography-mass spectrometry-based metabolic profiling approach##biomedical_technique to measure the @@gut microbiome##microbiome and @@microbial metabolites##chemical from 44 @@healthy controls##human, 41 @@acute patients##human, and 39 @@remission patients##human, to evaluate whether @@microbial dysbiosis##DDF and microbial metabolite biomarkers were linked with the severity of schizophrenic symptoms. Here, we identified 20 dominant disturbances in the @@gut microbial composition of patients##human compared with @@healthy controls##human, with 3 orders, 4 families, 9 genera, and 4 species. Several unique bacterial taxa associated with @@schizophrenia##DDF severity. Compared with @@healthy controls##human, 145 unusual @@microflora metabolites##chemical were detected in the acute and remission groups, which were mainly involved in environmental information processing, metabolism, organismal systems, and human diseases in the Kyoto encyclopedia of genes and genomes pathway. The Sankey diagram showed that 4 abnormal intestinal and 4 anomalous intestinal @@microbial metabolites##chemical were associated with @@psychiatric clinical symptoms##DDF. These findings suggest a possible interactive influence of the @@gut microbiota##microbiome and their metabolites on the pathophysiology of @@schizophrenia##DDF. ### Text Input: Alteration of Gut Microbiome in Patients With Schizophrenia Indicates Links Between Bacterial Tyrosine Biosynthesis and Cognitive Dysfunction.
Output:
The fine-tuning dataset was structured as a ‘.jsonl‘ file, where each line is a JSON object representing a single training example. Each object contains a list of messages defining the conversation flow: a system message with basic instructions, a user message with the input text and task instructions, and an assistant message with the correctly annotated output. Below is one example from our training set, using a title for brevity. {"messages": [{"role": "system", "content": "You are an expert Named Entity Recognition (NER) system specializing in biomedical texts related to the gut-brain axis."}, {"role": "user", "content": "\nTask Description:\nYour task is to identify and extract entities from the provided text. Perform Named Entity Recognition on the biomedical text provided in the \"Text\" section using a specific inline annotation style.\nIdentify all mentions of entities belonging to the predefined categories listed below. An entity can occur multiple times; treat each occurrence as a separate entity and mark them directly within the text.\nMark the beginning of an entity's span with \"@@\" and the end of the span with \"##\" followed immediately by the entity's category label. Note - the output text should be exactly identical to the input text i.e all spaces , special characters/ unicode characters,html/markdown tags,etc (or any other character) also should be exactly the same, except for the added \"@@, ## and entity label\" annotations for each entity detected.\nText span of an entity means the actual words/characters that form the entity in the text. An entity's span can contain single or multiple words but never partial words.\nThe format to follow while marking an entity with its label is @@<entity_text_span>##<label>\n\nPredefined Entity Categories (use these exact labels after ##):\n[\n \"anatomical_location\", \"animal\", \"biomedical_technique\", \"bacteria\",\n \"chemical\", \"dietary_supplement\", \"DDF\", \"drug\", \"food\", \"gene\",\n \"human\", \"microbiome\", \"statistical_technique\"\n]\n\nNote - DDF stands for Disease, Disorder, or Finding. The remaining categories refer to their conventional or scientific meaning.\nAlso, If the first word or first set of words in output belong to an entity then ensure to start the output with @@ and follow rest of instructions.\n\n### Text\nInput: Analysis of the Efficacy of Diet and Short-Term Probiotic Intervention on Depressive Symptoms in Patients after Bariatric Surgery: A Randomized Double-Blind Placebo Controlled Pilot Study.\nOutput:"}, {"role": "assistant", "content": "Analysis of the Efficacy of Diet and Short-Term Probiotic Intervention on @@Depressive Symptoms##DDF in @@Patients##human after Bariatric Surgery: A Randomized Double-Blind Placebo Controlled Pilot Study."}]}
While most hyperparameters were set to "auto" by the OpenAI API, the configuration for our bestperforming model is summarized in Table 3.