Open-weight versions of large language models (LLMs) are rapidly advancing, with state-of-the-art models like DeepSeek-V3 now performing comparably to proprietary LLMs. This progression raises the question of whether small open-weight LLMs are capable of efectively replacing larger closed-source models. We are particularly interested in the context of biomedical question-answering, a domain we explored by participating in Task 13B Phase B of the BioASQ challenge. In this work, we compare several open-weight models against top-performing systems such as GPT-4o, GPT-4.1, Claude 3.5 Sonnet, and Claude 3.7 Sonnet. To enhance question answering capabilities, we use various techniques including retrieving the most relevant snippets based on embedding distance, in-context learning, and structured outputs. For certain submissions, we utilize ensemble approaches to leverage the diverse outputs generated by diferent models for exact-answer questions. Our results demonstrate that open-weight LLMs are comparable to proprietary ones. In some instances, open-weight LLMs even surpassed their closed counterparts, particularly when ensembling strategies were applied. All code is publicly available at https://github.com/evidenceprime/BioASQ-13b.
In question answering tasks, access to domain-specific knowledge can significantly enhance response quality, particularly when answers need to be grounded in provided supplementary materials.
The BioASQ Challenge [
Over the years, various approaches have been employed for question answering tasks in the BioASQ
Challenge. In its earliest editions, classic methods were applied, such as BM25, which ranked retrieved
documents based on their relevance to a question. Additionally, researchers applied similarity algorithms
using vector embeddings, algorithms based on linguistic annotations of texts, and early deep learning
models. For an extended period, BERT-based solutions, notably BioBERT [
However, since the global emergence of ChatGPT [
A significant shift began in 2024 and 2025, with a growing number of organizations publicly releasing
models featuring open weights and permissive licenses. Today, large open-weight models like Llama
3-405B [
This year marks the 13th edition of the BioASQ Challenge, and we participated in Task 13B, Phase B
[
We experimented with numerous techniques to optimize performance in biomedical question answering. The successful approaches implemented in our solutions are outlined below.
For our submissions, we selected the 10 best-matching snippets from the provided PubMedQA articles.
Our team experimented with varying snippet counts, validating them against datasets from previous
BioASQ challenge versions. This process led us to select 10 snippets as the optimal number, as it
consistently produced the most robust results. We utilized the sentence-transformers library [
Research has demonstrated that in-context learning enhances the performance of language models in
diverse applications [
For all question types, we utilized hand-crafted prompts. As noted previously, a zero-shot prompting
approach was employed for yes/no and summary questions, given empirical observations that few-shot
prompting detrimentally afected performance for these specific categories. Conversely, for factoid
and list questions, few-shot prompting demonstrated clear benefits. Accordingly, we implemented
a 3-shot prompting strategy for these questions, based on insights gained through comprehensive
experimentation. The system prompts are detailed in Table 1, and the actual prompts guiding the
models to generate answers for all question types are presented in Table 2. We also briefly experimented
with DSPy [
You are an expert in the medical texts summarization.
Answer the given question with a single paragraph text and your answer should be based on the provided context snippets. You should generate your response in at most 2-3 sentences (30-50 words).
Given only the following SNIPPETS and QUESTION, answer the QUESTION only with ’Yes’ or ’No’.
Extract key biomedical entities **strictly using the provided SNIPPETS** to answer the QUESTION. List **1 to 5** of the most relevant entities, ranked by confidence. **Never exceed 5 entities.** If more exist, return only the top 5. Prefer concise entities and **remove redundant or longer variants** of the same term. If no relevant entities exist, return ‘None.‘.
Extract key biomedical entities **strictly using the provided SNIPPETS** to answer the QUESTION. List **1 to 5** of the most relevant entities. Prefer concise entities and **remove redundant or longer variants** of the same term. If no relevant entities exist, return ‘None‘.
Answer the QUESTION by returning a single paragraph sized text (use max 50 words) ideally summarizing only the most relevant information in the SNIPPETS. hand-crafted prompts. In the future, we plan to investigate whether automatic prompt optimization can help by creating model-specific prompts.
We opted to use structured outputs to facilitate the extraction of LLM results in a predefined format.
We defined a JSON schema for response formatting, and subsequently followed a context-free grammar
(CFG) approach for it. We used CFG implementations introduced by model providers or accessed via
external libraries like Outlines [
In our study, we utilized various LLMs, drawing from both open-weight and closed options. Our primary
focus was on relatively smaller open-weight models, specifically those with up to 14 billion parameters,
such as Phi-4 [
Ensembling methods are widely recognized as beneficial when combining responses from multiple weaker models to achieve a single, stronger result. Various techniques exist for this purpose, including majority voting, confidence scoring, and aggregation.
For yes/no questions, we applied a straightforward majority voting approach, where the final answer was determined by the option chosen by the most models. For factoid and list questions, we developed a more sophisticated aggregation method. This involved collecting responses from all models and calculating the frequency of each distinct response. The most frequent response was then selected, provided its number of appearances exceeded a predefined threshold. The process is visualized on Figure 1. For factoid questions, we limited the output list to a maximum of five best responses, as specified in the rules for this question type.
For later batches, we incorporated diferent classes of LLMs into our ensembling strategy. This approach leveraged the observed benefits of integrating the diverse characteristics of various LLM families such as Phi, Qwen, Mistral, and Gemma. As emphasized by Jiang et al. [31], diferent LLMs, trained on varied data and architectures, inherently exhibit unique strengths that can be synergistic in an ensemble.
We participated solely in Task 13B, Phase B, of the BioASQ challenge [
For each submission, the rules detailed in 2 were applied. The models used to generate responses in each submission, grouped by system name from 1 to 5, are presented below. The results for these systems in all batches are summarized in Table 3, Table 4, Table 5, and Table 6. A detailed specification of each submitted system follows:
• EP-1: Phi-4 • EP-2: HuatuoGPT-o1 8B • EP-3: Qwen2.5-14B • EP-4: GPT-4o • EP-5: Claude 3.5 Sonnet
• EP-1: Ensemble - Gemma-3-12B, Qwen2.5-14B, Phi-4, GPT-4o, Claude 3.5 Sonnet • EP-2: Claude 3.5 Sonnet • EP-3: Phi-4 • EP-4: Phi-4 + DSPy prompt (only for factoid questions, without prompt optimization) • EP-5: Qwen2.5-14B + DSPy prompt (only for factoid questions, without prompt optimization)
Batch 4: • EP-1: Ensemble - Mistral-Small-3.1-24B, Gemma-3-12b, Gemma-3-27b, Qwen2.5-14B, Phi-4 • EP-2: Ensemble - GPT-4o, GPT-4.1, Claude 3.5 Sonnet • EP-3: GPT-4.1 • EP-4: Phi-4 • EP-1: Ensemble - GPT-4.1, GPT-4o, Claude 3.5 Sonnet, Claude 3.7 Sonnet • EP-2: Ensemble - Gemma-3-12B, Qwen2.5-14B, Meditron3-Phi4-14B, Phi-4 • EP-3: Ensemble - Qwen2.5-14B, Meditron3-Phi4-14B, Phi-4, GPT-4.1, GPT-4o, Claude 3.5 Sonnet,
Claude 3.7 Sonnet • EP-4: Ensemble - Qwen2.5-14B, Phi-4, GPT-4.1, GPT-4o, Claude 3.7 Sonnet • EP-5: GPT-4.1
We used distinct answering strategies for each batch, as detailed in 3.1. For batch 1, our primary focus was on assessing the performance of individual models within each system. In batches 2 and 3, we also incorporated ensembling techniques, specifically involving combinations of open-weight and selected closed models. It is important to note that for these batches, we only used quantized versions of open models. Finally, for the last batch, we conducted a comparative analysis between full open-weight models and proprietary models.
The most advantageous approach for yes/no questions was dificult to determine. Proprietary models achieved the best results in batches 1 and 4, while open-weight models dominated in the remaining. It is shown in Table 3. The quality of the provided context appears to be a critical factor, with both model types demonstrating suficient capability to extract key information pertinent to the question.
List-based questions demonstrably pose a greater challenge for LLMs. Despite this, open-weight models performed competitively to proprietary ones. Furthermore, we found that ensembling more diverse models leads to improved scores.
In more detail, ensembling a mixture of open and closed models proved to be beneficial for factoid questions. In batch 2, single proprietary models were outperformed by such ensembling mixture. This solution also represented the best approach in batch 4, surpassing individual closed models and ensembles composed solely of open-weight or closed models. In batch 3, multiple solution types exhibited competitive performance. Table 4 provides a summary of the results for factoid questions across all batches.
These insights are further corroborated by the findings from list questions, presented in Table 5. Ensembling solely open-weight models or a mixture of both model types consistently yielded the best approaches in batches 2, 3, and 4. For batch 1, the results between both model types were notably similar. These observations strongly suggest that ensembles of open-weight models can address more challenging tasks at a comparable, or even superior, level to closed models.
For the summary questions, we directly generated each summary using a chosen LLM and the prompt
detailed in Table 2, without employing ensembling techniques. For systems involving multiple LLMs,
we generated candidate summaries from all participating models for each question. The best summary
was then selected using a cross-encoder reranking approach. This method involved calculating the
similarity score between each generated summary and its corresponding question. The summary with
the highest score was subsequently selected. For this purpose, we used the BiomedBERT model [
The recall scores for ROUGE metrics [32] achieved by our method for batches 2 and 4 were comparable to those of the top-performing solutions. However, F1 scores for ROUGE metrics in these batches were significantly lower. The results for summary questions can be seen in Table 6.
As shown in Table 6, Phi-4 exhibited the strongest performance among the evaluated LLMs, suggesting that open-weight models can also be competitive for summary-based questions. However, a full analysis of the responses to these questions requires manual scores that have not yet been published.
Our primary goal was to evaluate the competitive performance of open-weight LLMs against stateof-the-art proprietary LLMs for biomedical question answering. To this end, we rigorously tested numerous configurations, including both smaller open-weight models and various closed models. Our results consistently demonstrated the competitiveness of ensembles of open models for BioASQ 13B Phase B questions.
For yes/no questions, this thesis is supported by results from batches 2 and 3 (Table 3), where openweight models outperformed closed-weight models. This trend also holds for list-based questions, with batches 2 through 4 demonstrating strong performance by open models on both factoid and list-type questions (Tables 4 and 5, respectively). For summary questions, the open-weight model Phi-4 exhibited promising performance in terms of ROUGE metrics in Batches 1 through 3, as detailed in Table 6.
This conclusion holds significant implications. The ability to use open-weight models negates the need for proprietary solutions in every application. This is particularly relevant for applications involving highly restricted data that require on-premise deployment, a common scenario with medical data. In such contexts, smaller self-deployable models ofer a compelling and practical alternative.
This research was co-funded by the European Union – European Regional Development Fund (Programme: European Funds for a Modern Economy 2021-2027, grant no. FENG.01.01-IP.02-4479/23).
During the preparation of this work, the author(s) used Gemini in order to: Grammar and spelling check, Paraphrase and reword. After using this tool/service, the author(s) reviewed and edited the content as needed and take(s) full responsibility for the publication’s content. [30] Anthropic, Claude 3.7 sonnet system card, 2025. URL: https://assets.anthropic.com/m/ 785e231869ea8b3b/original/claude-3-7-sonnet-system-card.pdf. [31] D. Jiang, X. Ren, B. Y. Lin, Llm-blender: Ensembling large language models with pairwise ranking and generative fusion, 2023. URL: https://arxiv.org/abs/2306.02561. arXiv:2306.02561. [32] C.-Y. Lin, ROUGE: A package for automatic evaluation of summaries, in: Text Summarization Branches Out, Association for Computational Linguistics, Barcelona, Spain, 2004, pp. 74–81. URL: https://aclanthology.org/W04-1013/.