Commercial large language models (LLMs), like OpenAI's GPT-4 powering ChatGPT and Anthropic's Claude 3 Opus, have dominated natural language processing (NLP) benchmarks across diferent domains. New competing Open-Source alternatives like Mixtral 8x7B or Llama 3 have emerged and seem to be closing the gap while often ofering higher throughput and being less costly to use. Open-Source LLMs can also be self-hosted, which makes them interesting for enterprise and clinical use cases where sensitive data should not be processed by third parties. We participated in the 12th BioASQ challenge, which is a retrieval augmented generation (RAG) setting, and explored the performance of current GPT models Claude 3 Opus, GPT-3.5-turbo and Mixtral 8x7b with in-context learning (zero-shot, few-shot) and QLoRa fine-tuning. We also explored how additional relevant knowledge from Wikipedia added to the context-window of the LLM might improve their performance. Mixtral 8x7b was competitive in the 10-shot setting, both with and without fine-tuning, but failed to produce usable results in the zero-shot setting. QLoRa fine-tuning and Wikipedia context did not lead to measurable performance gains. Our results indicate that the performance gap between commercial and open-source models in RAG setups exists mainly in the zero-shot setting and can be closed by simply collecting few-shot examples for domain-specific use cases. The code needed to rerun these experiments is available through GitHub*.
the LLMs during pre-training, retrieval augmented generation (RAG) [
The BioASQ challenge is a great example of a RAG setup in a specialized domain, as the participating systems first have to find relevant biomedical papers from PubMed and extract snippets that are later used to generate answers for biomedical questions.
We set out to explore the usefulness and competitiveness of open-source models to the current SOTA commercial oferings in a typical domain-specific RAG setup represented by the BioASQ challenge. Compared to our last year’s approach where we only looked at the zero-shot performance of commercial models, we now explored few-shot learning because we saw that this enables open-source models to better follow instructions while also improving overall performance.
Another aspect that we explored was how additional relevant context retrieved from Wikipedia might aid the models in generating useful answers or relevant queries, as they might be limited in their biomedical knowledge about entities and their synonyms.
BioASQ is "a competition on large-scale biomedical semantic indexing and question answering"[
The 12th BioASQ challenge comprises several tasks:
• BioASQ Task Synergy On Biomedical Semantic QA For Developing Issues [
Content [
We participated in Task B and Synergy[
In the 12th installment, another Phase A+ was introduced, where the systems were supposed to provide answers to the questions before the gold snippets and documents were provided, relying solely on their own retrieved documents and snippets.
For the Synergy task, the systems receive a similar list of questions, for which they also have to retrieve useful papers and extract snippets and as soon as a question is marked as ready to answer, they also need to submit answers in the same format as for Task B. The diference between Synergy and Task B is that initially in the first round no gold set of documents and snippets is provided, instead the submitted documents and snippets by the systems are evaluated by biomedical experts and selected as gold reference items for subsequent rounds. This also means that the same questions might be reintroduced 5https://cohere.com/blog/five-reasons-enterprises-are-choosing-rag 6https://www.pinecone.io/learn/retrieval-augmented-generation/ 7https://gretel.ai/blog/what-is-retrieval-augmented-generation 8https://clef2024.clef-initiative.eu/ 9http://www.bioasq.org/ in subsequent rounds, possibly with additional questions and positive and negative feedback on the previously submitted documents.
Following this introduction, we will highlight some related work in Section 2, describe our methodology in Section 3, report our results in Section 4 and discuss them in Section 5. Section 6 will present some ethical considerations, and Section 7 ofers our conclusions.
We will briefly introduce the related work that led to the creation of the evaluated models, as well as the approaches that inspired our methodology.
Nearly all the popular SOTA LLMs that are used across various NLP tasks and use cases today are based
on the transformer architecture [
OpenAI was the first company that released such a fine-tuned model to the public in November 2022 10 which sparked massive interest in generative artificial intelligence research and products. Their latest model at the time of writing that is powering their ChatGPT product was GPT-4 [11]. One interesting competitor model that we also used during this competition is Claude 3 Opus11 by Anthropic, which reached GPT-4 level performance (GPT-4-0125-preview) at the time of the BioASQ competition12. The exact architecture of GPT-4 and Claude Opus 3 and other commercial models is unknown.
GPT-4 is the most expensive and slowest model that OpenAI is ofering via their API service. A more afordable alternative that they are ofering is GPT-3.5-turbo. We compared both these models’ performance in last year’s BioASQ competition and were able to show that GPT-3.5-turbo was sometimes performing better than GPT-4 in some question formats and subtasks of the competition [12].
For this year’s BioASQ competition we also used Mixtral 8x7B [13] a downloadable open-source model (Apache 2.0 license) which uses a type of Mixture-of-Experts Architecture [14][15]. This architecture ofers higher computational eficiency by routing requests to expert subnetworks to generate a response. Since only some specialized experts are active during the generation, less computation and memory is needed to serve the request.
Few-Shot learning is the ability of LLMs to learn how to solve a new problem that they were not specifically fine-tuned for by only showing them a few examples. When GPT-3 was first introduced, its impressive few-shot learning abilities made the concept popular [16] because it greatly reduces the need for expensive training data.
Zero-shot learning [17] takes this concept a step further by only requiring an abstract task description or direct question, which leads the model to ideally generate a useful completion that solves the task at hand [18]. In last year’s BioASQ competition, we were able to win some batches while only using zero-shot learning with SOTA commercial models [12].
Current LLMs have billions of parameters and require specialized hardware with enough GPUs and VRAM to hold all the model weights in memory. For example, "16-bit finetuning of a LLaMA 65B 10https://web.archive.org/web/20240502090536/https://openai.com/index/chatgpt/ 11https://web.archive.org/web/20240516173322/https://www-cdn.anthropic.com/de8ba9b01c9ab7cbabf5c33b80b7bbc618857627/
Model_Card_Claude_3.pdf 12https://chat.lmsys.org/?leaderboard parameter model requires more than 780 GB of GPU memory"[19]. Since this makes fine-tuning these models prohibitively expensive for many researchers and users, several clever techniques have been invented to reduce these hardware requirements. We wanted to fine-tune Mixtral 8x7B, which roughly takes up the memory of a 47B model13.
One popular approach is QLora by Dettmers et al. [19] where the model weights are quantized to 4 bits and frozen and only some low rank adapters (LoRa)[20] are fine-tuned. This would enable ifne-tuning Mixtral 8x7B, for example, on only two RTX A6000 GPUs with 2x 48 GB of VRAM.
Retrieval augmented generation (RAG) is a technique [
RAG has been shown to improve the factual accuracy of generated text compared to standalone
language models [21]. It allows the model to access a vast amount of external knowledge and incorporate
it into the generated output. RAG is particularly useful for tasks that require domain-specific knowledge
or up-to-date information [
Professional search is conducted in a professional context, often to aid in work-related research tasks [22]. In some professional search settings, highly trained specialists are needed to create documented and reproducible search strategies, this sets professional search apart from everyday web search [23]. The BioASQ challenge exemplifies one possible professional search setting where biomedical experts aim to find answers to domain-specific questions with suficient evidence.
Other examples of professional search might be systematic reviews [24], patent-search or search conducted by recruitment professionals [25]. All of these settings might require complex search strategies, where the search expert makes use of a query syntax involving boolean operators on specific search fields. Systematic reviews, for example, also require the search to be explainable and reproducible, which makes it dificult to use advanced vector-based retrieval techniques. Formulating traditional queries but with large language models that might be able to expand synonyms and related terms based on their semantic representations is therefore an interesting approach that might aid in professional search settings. We set out to explore this approach in the BioASQ challenge.
3.1. Model In this year’s BioASQ competition, we looked at the commercial oferings GTP-3.5-turbo and GPT-4 from OpenAI and also used Antrophics Claude 3 Opus, which was at the time of the run submissions the only other model that was on a level with GPT-4 according to the LMSYS Chatbot Arena Leaderboard [26].
Since last year’s BioASQ competition, some competitive Open-Source models were published. The most notable ones being the Llama series models by Meta [27], with the latest being Llama 3 [28]. Llama comes with its own custom License which is quite permissive, but even though commercial use is allowed under this license, as long as the monthly user base does not exceed 700 million users, 13https://mistral.ai/news/mixtral-of-experts/
Listing 1: Query Expansion Prompt {"role": "user", "content": f"""Turn the following biomedical question into an efective elasticsearch query using the query_string query type by incorporating synonyms and additional terms that closely relate to the main topic and help reduce ambiguity. Focus on maintaining the query’s precision and relevance to the original question, the index contains the fields ’title’ and ’abstract’, return valid json: ’{question}’ """} the license might not be straightforward to adopt for enterprise use cases when licenses have to be pre-approved by a legal team.
When we prepared our runs for the competition, the best-performing model with a permissive open-source license (Apache 2.0) on the LMSYS leaderboard was Mixtral 8x7B [13]. The model also has a large context length of 32k tokens, which makes it especially interesting for RAG use cases or few-shot learning. We therefore choose Mixtral 8x7B as our open-source competitor model for this competition. During the competition, the newer Mixtral 8x22B model was also published, and we used it in some batches of Task B.
We used the commercial hosting service fireworks.ai 14 to access and fine-tune Mixtral 8x7B, as the provided speed was very high and costs for their API usage were low.
We downloaded and indexed the annual PubMed baseline from the oficial website 15. We indexed both the title and the abstract of all papers in separate fields of our index using the built-in English analyzer of Elasticsearch16. For every round of synergy, the most recent snapshots of 2024 up to the date considered in that round were downloaded and indexed in another similar index, which was then also searched during the runs.
For synergy, we used both gpt-4-0125-preview and gpt-3.5-turbo-0125, the newest available versions of OpenAI’s GPT-4 and GPT-3.5-turbo at the time of the competition. We used 2-shot learning to generate queries for our Elasticsearch PubMed index and zero-shot learning for extracting and reranking snippets as well as answering questions. We also wanted to use Mixtral 8x7B in this task, but the model was unable to follow instructions well enough to produce usable runs, especially in the zero-shot setting.
Given a question, we prepended the prompt in Listing 1 with two examples where the same prompt contained other questions, for example "Is CircRNA produced by back splicing of exon, intron or both, forming exon or intron circRNA?" and an ideal completion in form of an Elasticsearch query for the query_string endpoint of Elasticsearch for which an example can be seen in Listing 2. We sent both the examples and the prompt with our actual question to the model and received back a JSON object that could be used to directly query our index.
We ran the generated query to retrieve the top 50 relevant documents from Elasticsearch. We filtered out documents that were marked as irrelevant in the feedback file for the synergy round. We then sent each remaining article alongside the question to the model and used a zero-shot prompt to extract a list of relevant snippets from the article. We then used string matching to insure the returned snippets were actually present in the article title or abstract and to calculate the ofsets.
We collected all relevant snippets from all potentially 50 articles and filtered out articles as irrelevant where the model did not extract any snippets from. We then prompted the model to select the top 10 snippets ranked by helpfulness from the set of snippets. We finally reranked the retrieved articles according to the snippet order returned by this step. 14https://fireworks.ai/ 15https://pubmed.ncbi.nlm.nih.gov/download/ 16https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-lang-analyzer.html#english-analyzer
Listing 2: Query Expansion Competion Example {"role": "assistant", "content": """ { "query": { "query_string": { "query": "(CircRNA OR \"circular RNA\") \"back splicing\" exon OR intron", "fields": [ "title^10", "abstract" ], "default_operator": "and" } """},
In the question answering step, we used the identical zero-shot prompts from our last year’s participation in Task B for this year’s Synergy task [12]. And we also merged the already deemed relevant snippets from the feedback files into our list of snippets that we passed on to the modal alongside the question prompt.
We also sent the same initial system prompt that we used last year [12] to the models. For the parameters, we set the temperature parameter to 0 to reduce randomness in the completion and supplied a seed parameter which is a new feature ofered by OpenAI that can help maximize reproducibility for the model output, but determinism is still not guaranteed17. We also used the new response_format parameter to insure the model produced valid JSON responses for the prompts where we needed it18. The exact python notebooks with all the implementation details and prompts used are available in our GitHub repository. 3.3. Task 12 B For Task B we reused the indexed PubMed annual snapshot that we created from the synergy task. We also switched the models, we added Mixtral 8x7B Instruct v0.1 as an open-source model, and we used Claude 3 Opus instead of GPT-4 because it became available shortly before the Phase started, and we had access to a free beta evaluation account.
In batches 1 & 2 we also explored the fine-tuning service of OpenAI and created 6 fine-tuned versions of GPT-3.5-turbo for each sub problem that our system had to solve. We also created 6 QLoRa fine-tuned versions of Mixtral 8x7B Instruct v0.1 using the fine-tuning service of fireworks.ai. The training sets were created from the supplied training data. We also adjusted our code to be able to use these training ifles as sources for few-shot examples.
The sub problems that we sampled training sets for were: • Snippet Extraction • Snippet Reranking & Selection • Summary Question Answering • Exact yes/no Question Answering • Exact factoid Question Answering • Exact list Question Answering 17https://platform.openai.com/docs/api-reference/chat/create#chat-create-seed 18https://platform.openai.com/docs/api-reference/chat/create#chat-create-response_format
Listing 3: Wikipedia Retrieval Prompt prompt = f"""
Given the question "{question}", identify existing Wikipedia articles that ofer helpful background information to answer this question.
Ensure that the titles listed are of real articles on Wikipedia as of your last training cut−of. Wrap the confirmed article titles in hashtags (e.g., #Article Title#).
Provide a step−by−step reasoning for your selections, ensuring relevance to the main components of the question.
In batches 3&4 we explored how we could use the models to retrieve relevant additional context from Wikipedia. We hypothesized that supplying these models with knowledge about relevant entities in the questions might improve their ability to generate correct answers. We wanted to explore the approach of retrieving such additional information about entities from a Wiki, because such a wiki like could also be generated in an enterprise setting, potentially closing the knowledge gap for entities that the models didn’t encounter during pre-training. We chose Wikipedia as a knowledge base even though the concepts described there might not be novel to the models because it was easy to use, and we hoped that we could observe an efect even with known concepts. We suspected that these models might just "know" which Wikipedia articles might be relevant to a question because they might have been highly trained on Wikipedia and links to Wikipedia articles. The exact zero-shot prompt that was used for ifnding relevant Wikipedia articles can be seen in Listing 3.
The titles of the Wikipedia articles returned by this prompt were extracted and, if the Wikipedia articles actually existed, their content was retrieved and concatenated. Finally, the concatenated articles were again sent to the model, and it was prompted to make a concise summary to help answer the question. This summary was then added as additional context to all subsequent prompts such as query generation or snippet extraction, reranking and question answering. 3.3.1. Phase A In Phase A, we changed the way we prompted the models for queries compared to the Synergy task. Instead of expecting the whole valid JSON query object back, we only prompted the models to create the query string in the valid query_string query syntax to pass on to the Elasticsearch endpoint and manually controlled weighting of the fields and the document return size.
We then used the questions from batch 1 from last year’s task, that were provided in the development set, to create a set of Elasticsearch queries with Claude 3 Opus. Finally, we ran these queries and evaluated the returned documents against the gold set to select 10 queries with the highest f1 score as few-shot examples for all models. We used 10 examples for most of our few-shot tasks because they fit in most models context lengths, with GPT-3.5-turbo having the smallest context length of 16k tokens.
The rest of our approach to retrieving relevant documents and snippets was mostly in line with our approach from the Synergy task, except that we re-added the step that we also used last year when we prompted the models for an improved query when a generated query did not return any results.
Compared to our system from last year, we changed the query expansion/generation prompt, added the possibility to prepend few-shot examples to the prompts and used diferent models where some were also fine-tuned. We also reranked snippets instead of titles, and filtered and ranked the articles according to their extracted snippets. We also used our own Elasticsearch index of the PubMed annual baseline instead of the online PubMed search endpoint. 3.3.2. Phase A+ and Phase B Our approach for Phase A+ and Phase B were mostly identical. We used the same prompts and few-shot examples taken from the sampled training sets for model fine-tuning. The only diference was the relevant snippets provided as context to the models. For Phase A+ we used the same snippets as input for all models, these were taken from the run in Phase A where the most snippets were found. We opted to use the same snippets, as we wanted to be able to compare the performance of the diferent models in Phase A+ isolated from their performance in Phase A. For Phase B we took the gold snippets provided in the run file as input.
The prompts used for Phase A+ and Phase B were identical to the question answering prompts from the Synergy task and our last year’s approach. We only added the option to add additional context before the snippets and supply few-shot examples. In batches 1 & 2 we compared the performance of ifne-tuned models with their non-fine-tuned counterparts, and in batches 3 & 4 we compared systems with additional relevant context taken from Wikipedia with systems that did not have this context.
We participated with our systems in two tasks of the BioASQ challenge: Synergy and Task B On Biomedical Semantic QA. For the synergy task, we report the results only for batch 3 and 4 as we were unable to participate in earlier batches. For Task B we competed in all batches and report the results of sub-tasks A (Retrieval), A+ (Q&A with own retrieved documents) and B (Q&A with gold documents). The results presented in this section are only preliminary, as the manual assessment of the system responses by the BioASQ team of biomedical experts is still ongoing. The final results will be available on the BioASQ homepage once the manual assessment is finished 19.
We participated with 2 systems in batch 3 and 4 of the Synergy task. The full result table is accessible on the BioASQ website20. The two systems both used the same 2-shot query expansion and zero-shot Q&A approach but with diferent commercial models. The system names and corresponding models are listed below: • UR-IW-2: gpt-4-0125-preview.
• UR-IW-3: gpt-3.5-turbo-0125
We tried to also use Mixtral 8x7B Instruct v0.1 as an open-source alternative in this task, but the model was unable to follow the zero-shot prompts for query expansion and extracting the snippets consistently enough to produce a submittable run file. Our retrieval approach for query expansion and ifltering and reranking via extracted snippets was not competitive in the document retrieval stage of the task, with both models performing similarly poorly compared to the top competitors, as indicated in Table 1. The oficial metrics to rank systems in each subtask are highlighted in bold in the following tables21. 19http://participants-area.bioasq.org/results/ 20http://participants-area.bioasq.org/results/synergy_v2024/ 21"Top Competitor" are the systems that took the first position in a round or batch that are not ours. They are added as a reference point for the reported metrics. When "Top Competitor" is missing in a reported batch, one of our systems was the best-performing one.
For snippet extraction, the performance of our approach was also poor, except for batch 4 where gpt-3.5-turbo-0125 was able to achieve second place as can be seen in Table 2. Gpt-4-0125-preview was unable to extract any snippets in the same batch, which was unusual and could have been due to issues OpenAI had with serving the preview model via their API.
For the question-answering stage, both models achieved perfect scores in the exact yes/no answer format, see Table 3. For factoid answers, gpt-4-0125-preview was able to take first place in batch 4 and achieved higher placements than gpt-3.5-turbo-0125 over both batches, as can be seen in Table 4. A similar diference is also observable in Table 5 for the list answer results.
Completing one run with gpt-4-0125-preview cost around $12 in API fees, while the same run with gpt-3.5-turbo-0125 was around 10 times cheaper at $1.2. Gpt-4-0125-preview was also quite slow, taking around 180 minutes to complete one run, while gpt-3.5-turbo-0125 took only a few minutes. Cost significantly decreased compared to our last year’s participation, while speed increased. This enabled us to actually use GPT-4 on the snippet extraction task, while last year we were not able to complete runs with snippet extraction for GPT-4 due to time and cost constraints. We also encountered less to none API errors during our runs, except for the empty responses in batch 4 for GPT-4, while last year we often had to rerun questions because the API timed-out or returned other errors. 4.2. Task 12 B Phase A We participated with 5 systems in all 4 batches of Task 12 B Phase A. The systems either used 1- or 10-shot learning with the plain or a fine-tuned model in batches 1+2 or additional context retrieved from Wikipedia in batches 3-4. The system names and configurations are listed below.
Batches 1-2: • UR-IW-1: Claude 3 Opus + 1-shot • UR-IW-2: Mixtral 8x7B Instruct v0.1, QloRa Fine-Tuned + 10-Shot • UR-IW-3: gpt-3.5-turbo-0125 fine-tuned + 1-shot • UR-IW-4: Mixtral 8x7B Instruct v0.1 + 10-shot • UR-IW-5: gpt-3.5-turbo-0125 + 10-shot • UR-IW-1: Claude 3 Opus 1-shot + wiki • UR-IW-2: Claude 3 Opus 1-shot • UR-IW-3: Mixtral 8x7B Instruct v0.1 10-Shot + wiki • UR-IW-4: Mixtral 8x7B Instruct v0.1 10-Shot • UR-IW-5: Mixtral 8x22B Instruct v0.1 10-Shot + wiki
The following Tables 6 and 7 show the results of our systems participating in the 4 batches. MAP was the oficial metric to compare the systems.
In batches 1 & 2 where we compared fine-tuned versions of GTP-3.5 and Mixtral 8x7b with 10-shot learning and Claude 3 Opus, no clear trend was observable over batches in the document retrieval stage of Phase A as can be seen in Table 6. Only Mixtral 8x7B with 10-shot learning was consistently performing worse than all our other models, While Claude 3 Opus with 1-shot learning was our best model in batch 2 and second best in batch 1.
We used 1-shot learning instead of 10-shot learning for Claude 3 Opus due to time constraints because the model was slow, and for the fine-tuned gpt-3.5-turbo due to cost constraints. Sending 10 examples of abstracts for snippet extractions per 50 highest-ranked search results would have amounted to quite some input tokens per run for these models.
For batch 3 & 4 where we explored if giving the systems additional Wikipedia context while creating queries for Elasticsearch could improve their performance, we also could not observe a consistent efect over batches. While in batch 3 the systems with additional Wikipedia context performed better, this efect was reversed in batch 4.
In the snippet extraction stage of Phase A, our QloRa fine-tuned version of Mixtral 8x7B was our worst-performing system in both batch 1 & 2, followed by the 10-shot Mixtral 8x7B version as can be seen in Table 7. The fine-tuned version of GPT-3.5-turbo was consistently ahead of its 10-shot counterpart in both batches, while Claude 3 Opus was worse than the GPT-3.4-turbo systems in batch 1 and better in batch 2.
Additional Wikipedia context did not lead to consistent results across batches. While the systems with Wikipedia context (UR-IW-1, UR-IW-3) performed better than their counterparts (UR-IW-2, UR-IW-4) in batch 3, this efect was again reversed in batch 4.
One run with Claude 3 Opus with 1-shot learning and additional Wikipedia context took around 140 minutes to complete, we did not have to pay for the tokens used because we had an early beta evaluation account. For Mixtral 8x7B with 10-shot learning and additional Wikipedia context, the runs took around 14 minutes to complete via the fireworks.ai API and cost around $ 11. Fireworks.ai charged $0.50 /1M tokens for both input and output tokens as of writing, while Anthropic would have charged $ 15 /1M tokens for input and $ 75 /1M tokens for output. So the cost for doing 10-shot learning with Claude 3 Opus would have been at least 30 times as high while being 10 times slower. 4.3. Task 12B Phase A+ We participated with 5 systems in nearly all 4 batches of Task 12 B Phase A+22. The systems either used 10-shot learning with the plain or a fine-tuned model in batches 1+2 or additional context retrieved from Wikipedia in batches 3-4. Per batch, we used the same input snippet file for all systems to base their answers on to ensure that their performance is comparable.
Batches 1-2: • UR-IW-1: Claude 3 Opus + 10-shot • UR-IW-2: Mixtral 8x7B Instruct v0.1, QLoRa Fine-Tuned + 10-Shot • UR-IW-3: gpt-3.5-turbo-0125 fine-tuned + 10-shot • UR-IW-4: Mixtral 8x7B Instruct v0.1 + 10-shot • UR-IW-5: gpt-3.5-turbo-0125 + 10-shot • UR-IW-1: Claude 3 Opus 10-shot + wiki • UR-IW-2: Mixtral 8x22B Instruct v0.1 10-Shot • UR-IW-3: Mixtral 8x7B Instruct v0.1 10-Shot + wiki • UR-IW-4: Mixtral 8x7B Instruct v0.1 10-Shot • UR-IW-5: Mixtral 8x22B Instruct v0.1 10-Shot + wiki 22We failed to submit one system run for system number 5 in batch 3.
For the yes/no exact answer format, Claude 3 Opus with 10-shot learning was our worst-performing system, while the fine-tuned version of GPT-3.5-turbo was our top-performing system in batch 1 and only beaten by its 10-shot counter-part in batch 2, as can be seen in Table 8. It was interesting to see that the open-source models could perform better than the presumably most advanced commercial model, Claude 3 Opus, in this task.
For batches 3 & 4, we could show that additional Wikipedia context led to inconsistent results. While this context improved performance in batch 3 for the Wikipedia enhanced systems (UR-IW-5, UR-IW-3) over their normal 10-shot counterparts (UR-IW-2, UR-IW-4) it again led to worse performance in batch 4. This result is in line with the results from Phase A where these systems performed similarly for document retrieval and snippet extraction. We speculate that the models are sensitive to the Wikipedia context, and the usefulness of the context is highly influenced by both the entities present in the questions, and its relationship to the relevant snippets.
In the exact answer factoid format, our worst-performing system in batches 1 & 2 was consistently Mixtral 8x7B with 10-shot learning while its fine-tuned counterpart performed better as can be seen in Table 9. This order was reversed for GPT-3.5-turbo, where the fine-tuned version performed worse than its counterpart.
The additional Wikipedia context again led to inconsistent results across batches, but this time the systems with Wikipedia context performed better in batch 4 compared to batch 3, which is contrary to the observed behavior in the document retrieval and snippet extraction in Phase A as well as the yes/no answer format in Phase A+.
For the list exact answer format, Mixtral 8x7B was again our worst-performing system in batch 1 & 2, while its fine-tuned counterpart was competing with the fine-tuned version of gpt-3.5-turbo for the top positions as can be seen in Table 10.
In batches 3 & 4 Claude 3 Opus with 10-shot learning and additional Wikipedia context was the bestperforming system in both batches while Mixtral 8x7B with 10-shot learning and additional Wikipedia context was the worst-performing one. Overall no consistent efect of the Wikipedia context was observable across models and batches.
We participated with 5 systems in all 4 batches of Task 12B Phase B. The systems used 10-shot learning with the plain or a fine-tuned model in batches 1 & 2 or additional context retrieved from Wikipedia in batches 3 & 4.
Batches 1-2: • UR-IW-1: Claude 3 Opus + 10-shot • UR-IW-2: Mixtral 8x7B Instruct v0.1, QLoRa Fine-Tuned + 10-Shot • UR-IW-3: gpt-3.5-turbo-0125 fine-tuned + 10-shot • UR-IW-4: Mixtral 8x7B Instruct v0.1 + 10-shot • UR-IW-5: gpt-3.5-turbo-0125 + 10-shot Batch 3 Batch 4 • UR-IW-1: Claude 3 Opus 10-shot + wiki • UR-IW-2: Mixtral 8x22B Instruct v0.1 10-Shot • UR-IW-3: Mixtral 8x7B Instruct v0.1 10-Shot + wiki • UR-IW-4: Mixtral 8x7B Instruct v0.1 10-Shot • UR-IW-5: Mixtral 8x22B Instruct v0.1 10-Shot + wiki • UR-IW-1: Claude 3 Opus 10-shot + wiki • UR-IW-2: Claude 3 Opus 10-shot • UR-IW-3: Mixtral 8x7B Instruct v0.1 10-Shot + wiki • UR-IW-4: Mixtral 8x7B Instruct v0.1 10-Shot
Batch Test Batch 1 Test Batch 2
In the exact yes/no answer settings of Phase B, Claude 3 Opus with 10-shot learning and gpt-3.5 turbo with 10-shot learning were sharing first place in batch 1 while the fine-tuned version of gpt-3.5-turbo was the best-performing system in batch 2 as can be seen in Table 11. The fine-tuned version of Mixtral 8x7B was also competitive, taking the 5th position in batch 1 and 8th position in batch 2.
In batch 3 & 4 with additional wikipedia context the systems performed better than their counterparts in batch 3 while this result was mixed in batch 4, leading again to inconsistent results.
For the exact factoid answer format in Phase B, Claude 3 Opus and our fine-tuned Mixtral 8x7B model where sharing first place in batch 2 while in batch 3, Claude 3 Opus and gpt-3.5-turbo where the on the same level as can be seen in Table 12.
For Mixtral 8x7B, additional Wikipedia context improved the outcome in batch 3 but led to worse results in batch 4, while a similar efect was observable for Claude 3 opus in batch 4 23.
For the exact list answer format in Phase B, Mixtral 8x7B with 10-shot learning took first place in batch 2 while being on place 22 of 39 in batch 1 where gpt-3-5-turbo with 10-shot learning was our best-performing one as can be seen in Table 13.
For batches 3 & 4 both Claude 3 Opus and Mixtral 8x7B performed worse with additional Wikipedia context across batches. In batch 3 our best-performing system was Mixtral 8x7B with 10-shot learning and in batch 4 it was Claude 3 Opus with 10-shot learning.
The costs for completing runs in Phase B were lower than in Phase A and the runs were faster because we did not have to do snippet extraction for 50 documents per question, times the number of few-shot examples, we therefore were able to also do 10-shot learning with Claude 3 Opus and the ifne-tuned version of GPT-3.5-turbo. The processing with Mixtral 8x7B via the Fireworks.ai API only took around 30 seconds for plain 10-shot examples and around 2 minutes for 10-shot examples and additional Wikipedia context.
We also submitted ideal answers for Task B and A+, but do not report on the preliminary results here, as the oficial judging metric for this answer type is based on the manual judgements that are not available yet.
While testing both commercial and open-source models, we could observe that there was no clear dominating model across batches or sub-tasks. Even our presumably weakest model, Mixtral 8x7B Instruct v0.1 with 10-shot learning was able to secure some leading spots in some batches of the competition, beating all other competing systems, (see batch 3 in Table 8 and batch 2 in Table 13). We 23The run of Mixtral 8x22B with Wikipedia context was not successfully submitted to batch 3, we might have overlooked to upload them. speculate that both the RAG setting and 10-shot learning might level the playing field a bit between commercial and open-source models, and it indicates that there is clear potential for creating state-ofthe-art systems even with cheaper, faster and presumably smaller open-source models, if they are used in the right way.
While the Mixtral model weights are publicly available, their training data is not published, which makes these models not ideal candidates for scientific research. A truly open-source LLM alternative is OLMo [29] published by the Allen Institute for AI. We choose Mixtral nevertheless because we wanted to study models that might be used by commercial practitioners in clinical or enterprise use cases, and we think the permissive license combined with the seemingly competitive performance on public benchmarks and its large context length makes it an ideal candidate for these use cases.
From our results with our experiments with additional Wikipedia context in BioASQ we could see that it had an impact on performance, but it was inconsistent across question batches. For some questions in some subtasks it led to improvements, while for others the performance declined. We speculate that this might be dependent on the relevant entities in the questions and the quality of the retrieved Wikipedia context. Further experiments are needed to analyze the impact of this additional context.
We also speculate that Wikipedia might not be a good proxy knowledge base for doing domainspecific RAG for these models because they are probably already highly trained on Wikipedia data and therefore the additional knowledge from this source might not tell the models much that they not already know.
Another reason for the inconsistent results with Wikipedia context could be that we only prepended the context for the last prompt, and the preceding n-shot examples were not generated with taking this context into account.
Regarding fine-tuning, we had the impression that the commercial ofering from OpenAI was not worth the cost. Even though it led to top results in some batches (batch 3 in Table 11) it also produced models with worse results than their significantly cheaper non fine-tuned counterpart in others. Maybe with the right training set and the right training run you might get a consistently superior model, but then you also have to add the engineering cost compared to simple 10-shot learning to the already more expensive usage and fine-tuning costs.
We had a similar impression regarding the QLoRa fine-tuning that we explored for Mixtral 8x7B. For example, the Mixtral model fine-tuned for list question answering was performing better than all other systems in batch 1 of Phase A+ (see Table 10) but worse than its non fine-tuned counterpart in batch 2 of Phase B (see Table 13). Overall, adapter fine-tuning appears not to be straightforward and requires more time for dataset creation, training and testing than simple few-shot learning. A more promising research direction might be selecting optimal few-shot examples for a given task.
It is important to note that most of the results we presented here are preliminary and might change when the manual assessment of the system responses is completed by the BioASQ experts. But we expect that yes/no answers will stay the same and factoid and list answers might just change slightly.
The preliminary performance of our systems in the document retrieval stage (Phase A) was quite poor compared to the other systems. We speculate that our approach of relying on TF_IDF-based retrieval and adding a richer semantic representation to the keyword query instead of using embeddings and vector search to add such information is not in line with the baseline system used to create the preliminary gold set. If that is true, it might be possible that our retrieval performance is actually better than we expect and the performance improves when the final results are out. But it could also just be that the approach is inferior. The good performance of our systems in Phase A+ where the questions had to be answered without gold snippets, might indicate that our retrieved snippets are not as useless as the preliminary results from Phase A suggest.
For future work, we would like to further explore optimal few-shot example selection [30][31][32], as few-shot learning seems to ofer the best flexibility and requires less engineering efort compared to ifne-tuning while being transferable between models. We also would like to revisit our knowledge base context augmentation approach, beyond the BioASQ challenge. We think that on a more technical test set where the relevant knowledge is highly unlikely to be present in the pre-training of these models, this approach could have a bigger impact. We also would need to use a diferent knowledge source as the English Wikipedia, which is part of most pre-training datasets [33].
The current generation of LLMs still exhibit the phenomenon of so-called hallucinations [34] that is they sometimes make up factually incorrect statements and even harmful misinformation. LLMs might also reproduce myths or misinformation that they encountered during their open-domain training or that might have been added to their input context. A recent prominent case was Google’s new AI overview feature suggesting users should add glue to their pizza24.
The hallucination and misinformation problems seem fundamental and dificult to solve as they have been known for quite some time now and even Google, one of the most experienced AI research companies, was unable to save itself from repeated embarrassment.
Even though RAG has been shown to reduce hallucinations in some settings [21] occasional hallucinations might still happen, which could be especially problematic in biomedical use cases and might warrant additional manual fact checking [35] before using the output of LLM-based systems in downstream tasks [36].
Another issue to consider is data privacy. These models might repeat their training data if prompted in a specific way [ 37]. That means training data has to be carefully anonymized before training or ifne-tuning these models. The same problem arises for the few-shot examples, personal data should be removed from all context that these models might repeat. They also might just make up facts about people, which could put vendors and service providers at legal risk for defamation25.
Another big ethical issue is job replacement and automation. Klarna, a financial service provider and one of the early enterprise customers of OpenAI published a report in February 2024 stating that their AI-powered customer support assistant handled one third of their customer support requests "doing the equivalent work of 700 full-time agents"26. This automation trend could not only lead to societal issues if more and more people are made redundant with LLM-powered systems, but also to quality issues when humans are taken out of the loop and more and more users and companies are trusting LLM-generated content without double-checking it2728.
We showed that a downloadable open-source model (Mixtral 8x7B Instruct V0.1) was competitive with some of the best available commercial models in a domain-specific biomedical RAG setting when used with 10-shot learning. This opens up the possibility to have state-of-the-art performance in use cases where using third-party APIs is not feasible because of the confidentiality of the data. The model used via a commercial hosting service was also significantly faster than Claude 3 Opus while being at least 30x cheaper.
We also observed that the zero-shot performance of this model was still lagging behind its commercial competitors, making it even unusable in some settings where highly specific structured output is required.
We were unable to achieve consistent performance improvements from QLoRa fine-tuning Mixtral or ifne-tuning gpt-3.5-turbo via the proprietary fine-tuning service of OpenAI. This might be an indication that successfully fine-tuning these LLMs requires more engineering efort and costs and might not be worth the efort in some use cases compared to few-shot learning.
We tried to augment the context of these models with additional relevant knowledge from a knowledge base (Wikipedia), but again could not see consistent performance improvements. We speculate that this might be due to the knowledge in this setup being not novel enough for the models, or because of the way we combined it with few-shot learning.
For future work we want to verify our results with diferent, more domain-specific tasks where less knowledge might have been present during the pre-training of these LLMs, and we also want to further explore optimal selection of few-shot examples, as this seems to get the best performance out of these models while being straightforward methodically.
We want to thank the organizers of the BioASQ challenge for setting up this challenge and supporting us during our participation. We are also grateful for the feedback and recommendations of the anonymous reviewers.