A large number of performant, open Large Language Models (LLMs) are continuously appearing. Here we deploy a selection of these for embedding and retrieval of documents and snippets as well as retrieval-augmented generators to answer biomedical questions within the BioASQ competition. Dense retrieval based on distances between dense representations obtained by LLM embeddings of the corpus and the question and hybrid sparse/dense methods result in higher mean average precisions compared to traditional sparse retrieval methods. In the exact answer category, which is processed using open LLMs in a zero-shot approach, our submission shares one first place in the last batch of the BioASQ 12b competition.
Doc Corpus BM25
Open Embedding models Query BM25 + RM3 Frequency Matrices Vectors Candidate Documents GANBERT Answers
Open Embedding models Cosine / Euclidean Distance Sparse Index Dense Index Documents
N = 10 Snippets
Cosine / Euclidean Distance
Open Embedding models Candidate Snippets Phase A
Snippets abstracts. For the systems Fleming-2 and Fleming-3 in batch 2 we adopted a methodology where the top 20 retrieved documents were processed with a prompt asking to assess the relevance of each document for the given question using several quantized open LLMs and finally MIXTRAL 1 via GPT4All and Ollama, a software designed to streamline access to open models locally, eliminating the need for manual downloading and scripting. This additional selection step did not improve retrieval performance.
For batch 3 and 4, we evaluated dense retrieval methods that compare the embeddings of the
corpus with the embedding of the question obtained with diferent open LLMs that have embedding
dimensions <= 1024, selected from the Massive Text Embedding Benchmark (MTEB) Leaderboard
at https://huggingface.co/spaces/mteb/leaderboard. We utilized both Euclidean distance and cosine
similarity metrics to evaluate the proximity between the documents and the query. This facilitated
the identification of the ten most closely related documents, as determined by smaller distances.
1https://mistral.ai/news/mixtral-of-experts
Subsequently, to combine the advantage of sparse retrieval methods finding documents with less
frequent words with the higher sensitivity of dense methods for semantic similarities, we use a threshold
for the distances returned from the dense search to replace documents exceeding this threshold with the
top documents returned from the sparse search. The details and performances of the sparse, dense and
hybrid retrieval methods are shown in table 1. Hybrid sparse and dense retrieval methods have been
suggested e.g. in [
A standard approach [
Phase A+
Predicted Phase B
Golden
Query smaug dolphinstarling mistral llama3 llama2 phi
In Phase A+, participants will submit exact and/or ideal answers before the expert selected (gold) documents and snippets (released in Phase B) are known. Thus, each participant has to use their predictions for documents and snippets for further processing. Participants will have 24 hours to provide exact answers for various question types ("yes/no," "factoid," "list") and ideal answers in the form of paragraph-sized summaries. In figure 2 processing for both Phase A+ and Phase B is illustrated.
We experimented with diferent prompts using various models to generate the answers using the information given in the snippets in a zero-shot approach. Ultimately, we chose the MIXTRAL model for List, Factoid, and Summary type of questions. The prompts used for each question type are as follows:
Yes/No Prompt Given only the following INFORMATION and QUESTION, answer the QUESTION only with ’Yes’ or ’No’ INFORMATION: %s QUESTION: %s
List Prompt Answer the QUESTION using only the TEXT by only returning a list of entity names, numbers, or similar short expressions that are an answer to the question and are separated by commas. Only the list should be returned. If you do not know any answer return the word EMPTY. TEXT: %s QUESTION: %s
Factoid Prompt Answer the QUESTION using only the TEXT by only returning a list of entity names, numbers, or similar short expressions that are an answer to the question and are separated by commas,ordered by decreasing confidence. Only the list should be returned. If you do not know any answer return the word EMPTY. TEXT: %s QUESTION: %s
Summary Prompt ##ABSTRACT: %s ##QUESTION: %s ##TASK: Answer the QUESTION by returning a single paragraph sized text ideally summarizing only the most relevant information in the ABSTRACT.
In all these prompts, the %s after QUESTION is replaced by the actual question, and the %s after INFORMATION, TEXT or ABSTRACT is replaced with the collection of the related snippets, concatenated and separated by a single blank. The answers by the LLMs are processed by custom awk scripts that eliminate doublettes in the case of list and factoid questions and extract the diference of ’Yes’ and ’No’ for Yes/No type questions.
The performances obtained for the Yes/No questions of the training set using diferent open LLMs are listed in table 3. As also observed in other applications, a larger number of parameters typically also leads to a higher prediction accuracy. Llama3-70B is a noteworthy exception in our tests, performing slightly worse than Llama3-8B.
In table 4 the performances of our document retrieval submissions for the BioASQ12 competition are listed.
In table 5 the performances of our snippets predictions for the BioASQ12 competition are listed.
In tables 6 and 7 the performances of our submissions for Phase A+ and Phase B of the BioASQ12 competition are listed. In batch 4 of Phase B, our submissions share the first place in average rank with rank
F-Meas. rank
batch
batch
System
At the time of writing, manual scores to assess the free text in the ideal answers were not ready and we cannot evaluate our submissions in this category. The higher performance of our hybrid sparse and dense retrieval system are promising and might be further improved by an adaptive combination of the two results and by using an optimized subset of the embedding for distance measurement. The open LLM ’farming’ approach employing a collection of (complementary) LLMs and used for the Yes/No questions can be transferred to the other question categories. With the observed rapid progress in the development of open LLMs, novel systems are easily incorporated into our pipelines.
We thank the anonymous reviewers for their comments. GPU computations for GANBERT were ofered by HYPATIA, the Cloud infrastructure of the Greek ELIXIR node. Other GPU computations were performed on two servers obtained within the project ID 16624, "Creation - Expansion - Upgrading of the Infrastructures of research centers supervised by the General Secretariat for Research and Innovation (GSRI)" with code MIS 5161770. This project was funded under the "National Recovery and Resilience Plan Greece 2.0".