This paper reports the work done by the CUHK-AIH team in the 12th BioASQ Challenge task 12b, which involves Phases A, A+, and B. In Phase A, we build BM25 indexes for all documents from PubMed Central (PMC). When an input question is received, the system uses the question as a keyword to retrieve relevant documents from PMC via the BM25 retriever, obtaining a list of targeted documents. For Phase A+, we construct a hierarchical Retrieval-Augmented Generation (RAG) pipeline based on the Llama2-chat-7B model. The model is fine-tuned on the BioASQ training set using a Parameter-eficient Fine-Tuning (PEFT) method called Low-Rank Adaptation (LoRA). The system further refines the search results from Phase A by employing an ensemble retriever that combines sparse and dense retrievers to identify the most relevant chunks. Finally, the system feeds the question and the most relevant chunks into the base model to generate the answer using appropriate prompts. In Phase B, the answer generation pipeline is similar to Phase A+, with the main diference being that we directly build indexes for the questions and their relevant snippets, treating snippets as the basic retrieval unit. We conducted detailed ablation studies and analyses on the model types and retrieval techniques, which indicate that PEFT and RAG can significantly improve the performance in biomedical Question Answering (QA) tasks.
In recent years, Large Language Models (LLM) have undergone significant development and have gained
widespread adoption [
In the field of medical information retrieval, a critical issue lies in constructing a robust information
retrieval system to efectively manage a massive corpus of medical literature [
Here we utilize the BM25 retriever to search documents in Phase A, and the Parameter-Eficient
Fine-Tuning on the BioASQ corpus [
Since this paper can be divided into pipeline-related research and retrieval-related research, this section will outline two parts of related work in Section 2.1 and Section 2.2. Specifically, we will begin by sorting out the related work of the main pipeline, which consists of LLMs in medical informatics, parametereficient model fine-tuning, and retrieval augmented generation. After that, we will introduce the structure of the retrieved text in RAG, followed by a work that converts the retrieved text composition.
In recent years, the advancement of Large Language Models (LLMs), exemplified by GPT[
Parameter-Eficient Fine-Tuning (PEFT) techniques encompass a series of fast, efective, and low GPU
computing and memory demand model fine-tuning methods. They are widely applied in the research of
Large Language Models, especially in specific domain applications [
To supplement the knowledge required by large language models when handling knowledge-intensive
tasks, the system can retrieve information from large corpora like Wikipedia to generate prompt
information [
In recent years, RAG also gains much attention in the biomedical field. MedCPT [
Recently, numerous studies have focused on tasks within the medical domain, enhancing the capacity of
Large Language Models (LLMs) to tackle Question Answering (QA) tasks related to medical knowledge
through the application of Retrieval Augmented Generation (RAG) on biomedical databases. In these
studies, due to the substantial size of their databases and the absence of structured organization
or sentence extraction methods for Question Answering inquiries, the entirety of text or chunks is
employed as the basic unit of retrieval. GeneGPT[
Besides the exemplary medical knowledge Question Answering Large Language Model (LLM) pipelines, which stand out in the research domain, we also examined other systems featured in the BioASQ task 11b’s publication (CLEF-WN 2023 [38]). These systems primarily utilize the provided snippets directly, but there are some slight variations in specific applications. UR-gpt [ 39], which performs very well in the rankings, inserts snippets directly into prompts for in-context learning. The Sentence-based Ranking [40] system’s method is close to the previous system, but they used the top 5 most relevant sentences due to the input length limit of its model. The IISR [41] firstly selects the top 5 most relevant snippets and uses ChatGPT to truncate or summarize them according to a fixed length. The context is input as the ChatGPT conversation history instead of fusing it together with the question in IISR. 2https://www.chatpdf.com 3https://www.langchain.com/ 4https://www.trychroma.com/ 5https://www.pinecone.io/
This section details the pipelines we employed in each phase. Our system is called Corpus PEFT Searching (CPS), for our main contribution is to integrate Parameter-Eficient Fine-Tuning (PEFT) on the PMC corpus with a hierarchical retrieval-based searching method in the BioASQ 12b task. Since the inputs and targeted outputs for submission are diferent in the three phases, we built three pipelines named CPS-A, CPS-B, and CPS-A+ for Phase A, B, and A+ submissions respectively. Table 1 shows the systems used in diferent phases and the sections and figures that describe them.
In Phase A (Section 3.1) we leveraged the BM25 retriever to search relevant documents and return the document list. Then in Phase B (Section 3.2), we used the BioASQ training set as the corpus to ifne-tune a Large Language Model (LLM) based on a Parameter-Eficient Fine-Tuning (PEFT) method to generate more reasonable answers. Finally, in Phase A+ (Section 3.3), we proposed a hierarchical Retrieval-Augmented Generation (RAG) method combined with the methods mentioned above to construct a general biomedical Q&A system, which could serve as a chatbot for any other biomedical questions. 3.1. Phase A
As illustrated in Figure 1, in Phase A, we first downloaded all the abstracts available from PubMed
Central (PMC)6 and employed the package Pyserini[42] to build BM25 indexes of these abstracts, treating
each abstract as an individual document. Subsequently, we applied the BM25 retriever to search related
documents using the query questions as the keywords. After being filtered by an appropriate similarity
score threshold, the final list of relevant articles is obtained.
3.2. Phase B
Our work in Phase B can be divided into two parts: Pre-processing and Retrieval & Generation, as
shown in Figure 2. In the Pre-processing stage, we built BM25 indexes of the golden enriched relevant
snippets using Pyserini[42] for later further retrieval. To help generate more accurate answers, we
used a Parameter-Eficient Fine-Tuning (PEFT) method, LoRA[ 43] to fine-tune the Llama2-chat-7B
model[
Relevant Snippets (Provided in Phase B) set (consisting of 5049 Q&A pairs). Due to the computing resource limit, we were unable to directly ifne-tune the whole Large Language Model on the corpus. The Low-Rank Adaptation (LoRA)[ 43] helps reduce the fine-tuning time and resources significantly while ensuring the model keeps a satisfying performance. The work in this stage was done before the test sets were released and wouldn’t be repeated later.
The Retrieval & Generation stage is the process to generate answers for the query questions in Phase B. Our system leverages an ensemble retriever combining sparse (BM25) and dense (vector similarity) retrievers for searching the most relevant snippets and provides them as references to the LLM. Bge7 was used to embed the chunks into vectors for similarity searching. The reason we constructed the retriever is that we found the performance of directly inputting all snippets into the model are not good enough, which we will discuss in Section 4.3.3.
Given appropriate prompts with the retrieved snippets and the query question, the fine-tuned model can generate a response for the query question to be submitted in Phase B. The prompt template is shown below, which we modified from Alpaca 8 by adding a role definition at the beginning.
You are an expert in the field of biomedical science.
Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request. ### Instruction: {question} ### Input: {context} ### Response: In this template, the position of ‘context’ will be replaced by the relevant snippets found in the retrieval procedure, while ‘question’ will be replaced by the query question body. 3.3. Phase A+ Phase A+ is more challenging than Phase B since there are no golden-enriched snippets. Therefore, our system needs to first search the relevant documents and snippets from the whole PMC database, and then generate answers. Here, we implement a hierarchical Retrieval Augmented Generation (RAG) pipeline combining the insights from Phase A and B (shown in Figure 3). 7https://huggingface.co/BAAI/bge-large-en 8https://github.com/tatsu-lab/stanford_alpaca
Pre-processing
Similarly to Phase B, the framework consists of two stages: Pre-processing and Retrieval & Generation. The fine-tuned model in the Pre-processing stage is identical to that in Phase B, while the indexes are built from the whole PMC abstracts rather than golden-enriched snippets.
In the Retrieval & Generation stage, we propose a hierarchical strategy to search question-relevant texts. Specifically, considering the quicker nature of sparse retrieval compared to dense retrieval, we still employed the BM25 sparse retrieval indexed on PMC abstracts as the first-level retrieval. This process selects the most relevant abstracts as the preliminary information screening. These documents are then segmented into multiple chunks by the TextSplitter from LangChain Package9 for the next search. Subsequently, an ensemble retriever which is the same as the retriever in phase B is employed as a more precise second-level retrieval. This ensemble retriever will identify multiple chunks most relevant to the question as the second level of retrieval. Finally, these most relevant chunks and the questions are combined by the prompt template mentioned in Section 3.2 to form the model input.
In the oficial submissions this year, we utilize the training set of BioASQ Task 12b (consisting of 5049 question-answer pairs) as the LoRA fine-tuning (mentioned in Section 3.2 and Section 3.3) dataset for CPS-B and CPS-A+ submissions. In the ablation study experiments in Section 4.3, we employ the training set of BioASQ Task 11b (consisting of 4719 question-answer pairs) as the LoRA fine-tuning dataset and the test set of BioASQ Task 11b batch 1 (consisting of 90 question-answer pairs) as the test set to do the evaluations.
Considering factors such as deployment complexity, open-source availability, and parameter scale, we chose the Llama2-7b-chat mode as the basic model in this work. For not focusing on the model 9LangChain:langchain_text_splitters.RecursiveCharacterTextSplitter
Batch 1 Batch 2 Batch 3 Batch 4
System
CPS-A Top Competitor
CPS-A Top Competitor
CPS-A Top Competitor
CPS-A Top Competitor itself, this study adheres to default settings for model parameters and fine-tuning parameters. In all our experiments and submissions, we use fp16 precision for both model fine-tuning and inference. We performed our experiments on a Linux server with two NVIDIA GeForce RTX 3090 GPUs.
In this section, we present the oficial evaluation results of our system in the four batches of BioASQ Task 12b. For comparison, we have included the best-performing systems in each batch. As there are multiple metrics, we selected the system that ranked in the top 5 among all metrics in each batch. If no system met this criterion, we expanded the selection to include systems ranked in the top 10. The selected system is referred to as the "Top Competitor" in the tables in Section 4.2. 4.2.1. Phase A We utilized the BM25 retrieval-based CPS-A system to retrieve relevant documents in phase A (introduced in Section 3.1). Here we show the submission results of the retrieved document in Table 2. As for selecting the relevant document list, we filter the retrieval score by an appropriate threshold optimized on the training set and choose the top 10 documents if the filtered documents are more than 10. In this section, we demonstrate the ‘ideal answer’ submission results in BioASQ task 12b phase B using CPS-B in Table 3. Since there is randomness in answer generation, we submitted two or three results in the challenge and we recorded the best one in the table. We used CPS-A+ to complete the submission of Phase A+, and the ‘ideal answer’ results are illustrated in Table 4. We did not submit the result of batch 1 due to the time limit.
The capability of the answer generation model has a significant influence on the performance in Phase B. Here we compare the performance of diferent generation models when taking questions and relevant contexts as input (shown in Table 5).
The experiments in the first three rows of Table 5 were carried out by ourselves. In particular, the Llama2-7B-chat + LoRA system is identical to the mentioned system CPS-B, but its dataset has been changed as 4.1.1 stated. The diference between Llama2-7B-chat and CPS-B is that the fine-tuned model is replaced by the original Llama2-7B-chat model. The GPT-3.5-turbo system is based on the API of GPT-3.5-turbo, where the input consists of the query question and the context obtained from the retrieval process of the CPS-B system. In this way, the input in the case of GPT-3.5-turbo is consistent with that of CPS-B.
Apart from these systems, we also select baselines and outstanding results in BioASQ task 11b phase B10. BioASQ Baseline FS is an oficial baseline obtained from BioGPT [ 44], and is detailed described in [38]. This oficial baseline system uses the concatenation of the question body and the relevant snippets until the input length is exceeded. UR-gpt4 [39] and UR-gpt3.5 [40] are systems using ChatGPT4 and ChatGPT3.5 with corresponding snippets. IISR [41] also used ChatGPT as its main model, but used a diferent snippet input method as desctibed in 2.2.2.
We can observe that the GPT-4 model is quite powerful and can obtain impressive results without ifne-tuning (the highest R-2 Recall and the second highest R-SU4 Recall). Besides, our system won the highest place among the three metrics including R-2 F1-score, R-SU4 Recall, and R-SU4 F1-score. Therefore, fine-tuning a relatively small LLM can achieve amazing results and beat a very powerful LLM in specific domains.
Beyond the generation model itself, the retrieved context can improve the accuracy of the generated answers. Here we evaluated the performance while there is or isn’t context given to the models, which is shown in Table 6. We can observe that all models didn’t perform very well without relevant context. Llama2-7B-chat + LoRA and BioASQ Baseline ZS are better since they were fine-tuned on biomedical corpora. However, when the relevant context is given, all models have a significant improvement even more than 100% on some metrics. Therefore, Retrieval-Augmented Generation is useful and promising when dealing with domain-specific Qustion-Answering tasks. 10http://participants-area.bioasq.org/results/11b/phaseB/
Besides, we explored the influence of the retrieved context quality on the performance from two aspects: Retrieval Unit and Retrieval Source.
• Retrieval Unit The golden-enriched snippets provided in Phase B test sets are of course the best retrieval unit for the answer generation since they are validated by biomedical experts. Therefore, we build a baseline treating chunks split from the golden-enriched documents as retrieval units. The ensemble retriever needs to search the most relevant snippets or chunks to provide references for the answer generation model (Llama2-7B-chat + LoRA). The results are illustrated in Table 7, which indicates retrieving from the golden-enriched snippets brings better performance. • Retrieval Source Then we experiment with adding noise to the retrieval process by changing the retrieval sources. The retrieval source means where the retrieval context comes from. We conducted experiments in three diferent settings: ‘Test Set’, ‘Training Set’, and ‘Training + Test Set’. The ‘Test Set’ setting is identical to the setting in Phase B. The ‘Training + Test Set’ setting means the relevant snippets should be first searched from snippets in the training and test set, which will bring some noise to the given snippets. The ‘Training Set’ setting means the relevant snippets should be searched from snippets only in the training set. Notice that snippets in the training set could be irrelevant in most cases, so it will bring a huge amount of noise. The results in Table 8 also support the conclusion that higher quality of the retrieved contexts will lead to better performance.
When the generation model and the context were the same, we further explored diferent context input methods. We constructed a Baseline called ‘Stuf Documents’ here by replacing the ensemble retriever of the system CPS-B as the ‘create stuf chain’ 11 from the LangChain Package. This chain means stufing 11Langchain API: langchain.chains.combine_documents.stuf.create_stuf_documents_chain all the relevant snippets of query questions as the context for the model input. The results in Table 9 show that introducing the retrieval process in CPS-B is more efective than just inputting all the relevant snippets as the context.
We constructed the Corpus PEFT Searching (CPS) system step by step following diferent phases in the BioASQ challenge. Firstly, in phase A the system acquires access to the PubMed Central database with a BM25 retriever. Secondly, in phase B the system integrates the Llama2-7b-chat model fine-tuned by LoRA with an ensemble retriever. Finally, in phase A+, we used an optimized two-stage hierarchical retrieval structure to connect the corpus and answer generation LLM. The first stage employs BM25 for a rough retrieval of documents, while the second stage combines BM25 with vector similarity for a ifne-grained retrieval of document chunks. The CPS system can search documents related to query questions and generate appropriate answers. It achieved a mid to slightly higher position on the BioASQ 12b leaderboard rankings. The reason may be that the scale and the context window length of our model are relatively small.
However, we made great eforts to improve its performance by utilizing diferent techniques including sparse and dense ensemble retrievers, Parameter-Eficient Fine-Tuning (PEFT), and hierarchical RetrievalAugmented Generation (RAG). The experiments in 4.3 demonstrate our exploration of the model ifne-tuning and retrieval techniques. We could conclude that the techniques we employed could significantly enhance the model performance without altering its architecture, leveraging the full synthesis capabilities of a 7B-parameter scale LLM. The experiment and analysis elaborate the idea that if the retrieved context contains all the necessary information to complete the task, a small-scale LLM fine-tuned by a parameter-eficient method is fully capable of generating ideal answers. Besides, improving the context input strategy like using ensemble retrievers also has the potential to enhance system eficiency.
Experiment results illustrated that our method exceeded the best level of last year’s ranking. However,
there is a gap between the system performance in last year and this year’s challenges. In this work, we
only applied the BM25 method to retrieve documents or snippets, since BM25 is still a strong retriever
in the biomedical RAG domain, according to the comparisons in Xiong et al. [
This work was supported by the IdeaBooster Fund IDBF23ENG05 of The Chinese University of Hong Kong. arXiv:2107.03374 (2021). [36] J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V. Le, D. Zhou, et al., Chain-of-thought prompting elicits reasoning in large language models, Advances in neural information processing systems 35 (2022) 24824–24837. [37] W. Shi, Y. Zhuang, Y. Zhu, H. Iwinski, M. Wattenbarger, M. D. Wang, Retrieval-augmented large language models for adolescent idiopathic scoliosis patients in shared decision-making, in: Proceedings of the 14th ACM International Conference on Bioinformatics, Computational Biology, and Health Informatics, 2023, pp. 1–10. [38] A. Nentidis, G. Katsimpras, A. Krithara, S. Lima López, E. Farré-Maduell, L. Gasco, M. Krallinger, G. Paliouras, Overview of bioasq 2023: The eleventh bioasq challenge on large-scale biomedical semantic indexing and question answering, in: International Conference of the Cross-Language Evaluation Forum for European Languages, Springer, 2023, pp. 227–250. [39] S. Ateia, U. Kruschwitz, Is chatgpt a biomedical expert, Exploring the Zero-Shot Performance of
Current GPT Models in Biomedical Tasks (2023). [40] A. Aksenova, T. Asamov, P. Ivanov, S. Boytcheva, Improving biomedical question answering with sentence-based ranking at bioasq-11b, in: Conference and Labs of the Evaluation Forum, 2023.
URL: https://api.semanticscholar.org/CorpusID:264441330. [41] C.-Y. Hsueh, Y. Zhang, Y.-W. Lu, J.-C. Han, W. Meesawad, R. T.-H. Tsai, Ncu-iisr: Prompt engineering on gpt-4 to stove biological problems in bioasq 11b phase b, in: 11th BioASQ Workshop at the 14th Conference and Labs of the Evaluation Forum (CLEF), 2023. [42] J. Lin, X. Ma, S.-C. Lin, J.-H. Yang, R. Pradeep, R. Nogueira, Pyserini: A Python toolkit for reproducible information retrieval research with sparse and dense representations, in: Proceedings of the 44th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2021), 2021, pp. 2356–2362. [43] E. J. Hu, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, W. Chen, et al., Lora: Low-rank adaptation of large language models, in: International Conference on Learning Representations, 2021. [44] R. Luo, L. Sun, Y. Xia, T. Qin, S. Zhang, H. Poon, T.-Y. Liu, Biogpt: generative pre-trained transformer for biomedical text generation and mining, Briefings in bioinformatics 23 (2022) bbac409.