This paper presents a work in progress to enhance a Retrieval-Augmented Generation (RAG) pipeline for a medical chatbot designed to address evaluative questions related to patient concerns about “normalcy”. The chatbot uses a novel approach called Hypothetical Document Embeddings (HyDoc) to augment queries and improve the retrieval of certified medical information. In the first evaluation of the chatbot, it emerged that evaluative queries often fail to retrieve relevant documents as well as to produce appropriately framed responses. We, therefore, experiment with the impact of an additional naive-RAG module to improve the retrieval and a Chain-of-Thought (CoT) inspired prompting strategy to contextualize the queries better and advance response generation. Results demonstrate that this method enhances document retrieval and the framing of generated replies, improving the chatbot's ability to generate responses that consider emotional and communicative aspects.
Normalcy is particularly relevant in medical conversations [
In our first implementation of an FAQ-based chatbot for pregnancy assistance, we did a user evaluation
to test user perception towards the agent[
The core approach behind our chatbot is to provide certified medical information via a RAG pipeline.
Our idea is to enhance document retrieval by augmenting the user query with a generated document
in a framework called Hypothetical Documents Embeddings [
In the case of evaluative questions, i.e. queries on normalcy, it is dificult to retrieve relevant
documents, probably due to the structure of our certified data. Our texts (1512 in total) came from FAQ
sections; therefore, they are not meant to be used in conversational settings, nor is the text structured
to answer evaluative queries. On the other hand, the dificulty in producing an appropriately framed
answer is probably due to the fact that LLMs struggle to understand the pragmatic context of a query [
Producing an appropriately framed response addressing the patient’s medical and emotional needs
is just as important as retrieving accurate information. Merely providing factual content can be
insuficient in a medical setting, potentially eroding patient trust if the response fails to account for
emotional concerns. A chatbot’s inability to properly frame answers in evaluative scenarios can lead to
misunderstandings and disrupt the flow of communication between the patient and the system [
This paper explores possible solutions to enhance document retrieval and text generation. On the
one hand, we introduce an additional RAG step, using the patient’s query before generating the HyDoc.
Although less precise, in the case of normalcy questions, this should help retrieve some of the related
documents, also preventing the HyDoc from being of-topic. On the other hand, we experiment with
a prompting strategy where we provide the LLM with the pragmatic context needed to answer the
question. This pragmatic context is made of implicatures and presuppositions [
Retrieval-Augmented Generation (RAG) integrates external knowledge into LLM prompts via data
retrieval, utilizing parametric and non-parametric memory [
The foundational works of Lewis et al. [
In this work, we introduce in our pipeline an additional step that we call naive-RAG. We call it naive
since we are using user questions instead of the HyDoc to query our certified repository. Since our
retrieval system employs a Bi-Encoder model [
On the other hand, there has been a vast range of prompting techniques to elicit (or at least simulate)
reasoning capabilities in LLMs2. Surely the most renowned method is the Chain of Thought (CoT)
prompting. These type of prompting demonstrated significant efectiveness in enhancing LLMs’ ability
to manage complex reasoning tasks, particularly those involving heterogeneous data such as tables and
questions, with substantial empirical improvements reported [
2For a complete review see Vatsal, S., & Dubey, H. (2024). A Survey of Prompt Engineering Methods in Large Language Models for Diferent NLP Tasks. arXiv preprint arXiv:2407.12994.
Further advancements have shown that generating intermediate reasoning steps—breaking down
problems into manageable parts known as a Chain of Thought (CoT)—significantly improves LLM
performance on complex reasoning tasks [
In this work, we will propose a simple CoT approach. Instead of solving a reasoning problem, our goal is to generate an LLM response that addresses both implicature and presupposition.
RQ1: Does the naive-RAG step improve document retrieval for evaluative questions? RQ2: Can we produce appropriately framed responses using CoT to provide pragmatic context?
Regarding RQ1, our goal is to see whether or not we can retrieve a higher number of documents with the naive-RAG approach. Our retrieval returns, in fact, 3 documents for each query; using the HyDoc generated after an evaluative question on normalcy often results in few (or even zero) documents being retrieved. On the other hand, RQ2 aims to explore a more qualitative dimension of text generation, namely the correct addressing of presupposition and implicature in the LLM-generated reply.
To answer our research questions, we generated 50 questions on normalcy starting from our documents. Each question has been manually validated to ensure that is relevant as an evaluative question. We then tested our original pipeline against a first variation, hence including a step of naive-RAG before HyDoc generation and a second updated version that includes an augmented CoT-style prompt to generate the final reply (see Figure 1). We used Prompt 1 to pass our implicatures and presuppositions to the LLM.
Prompt 1 (CoT prompt). “You are an assistant for pregnant women. You have answered the question from the input message: {hydoc} ### Input: {question} The correct answer is found in this text: {text_1} {text_2} {text_3}. Keep in mind that the text implies that {Implicature}. Keep in mind that the text presupposes that {Presuppostion}. Re-adapt the answer considering the provided information. ### Answer: ”
The LLM used in this study is Llama3, specifically the 70B version. We accessed the model through the GroqCloud API3.
In the case of a question of normalcy, to keep the evaluation as simple as possible, we considered one presupposition and one implicature.
• Presupposition: it is what is taken for granted in a sentence. In our case is therefore that “The patient thinks that something is not normal” • Implicature: broadly speaking it includes patient’s main goal as well as the emotional context. In our case therefore it would be that: “The patient is worried and wants reassurance” ” We decided to have the same presuppostion and same implicature for all our queries for two main reasons. First, as mentioned in Section 1, LLMs face challenges in understanding the pragmatic context of a query, so automatically generating implicatures and presuppositions is not the most efective solution. The second reason is that we want to experiment the consistency of our prompting strategy in an experimental setting. Using a fixed set of implicatures and presuppositions enables us to experiment with our prompt by varying only one element, i.e., the question.
To evaluate the reliability of the produced response, we compared the pipeline’s final reply against the
retrieved documents using BERTScore [
As shown in Table 1 our final responses are quite comparable. The summary is, not surprisingly the most similar to the original texts; however, the influence of CoT seems to be minimal, ensuring that the text generated by the LLM is still reliable.
We then compared the capacity of the naive-RAG module to overcome HyDoc limitations in evaluative contexts. As shown in Figure 2 the improvement is consistent.
To answer RQ2, we opted for a qualitative evaluation, manually checking whether the pragmatic aspects had been addressed. As shown in Figure 3, the CoT strategy notably improves text generation. Surely, it is complicated to answer evaluative questions using the LLM to summarize the documents. The length of the documents is influential and makes it dificult to capture evaluative aspects.
The naive-RAG pipeline, without including CoT, provides the LLM response as the final output, and we observe that it already performs better than the summary. Indeed, questions of normalcy often make explicit their presupposition (e.g. “Is it normal that I feel this way”). In that case, the LLM can produce better responses. Nonetheless, it is only by providing the CoT prompt with pragmatic context that we can properly frame our final reply.
Addressing implicature and presupposition not only produces more natural responses but it also strongly contributes to providing highly relevant information. For instance, in Example 1 and Example 2, we show the diferent replies to Query 1. As we can see, in Example 1, the LLM response is limited in providing bare information on the average weight of a newborn. However, this information alone is insuficient to give a satisfactory reply.
Opposite, Example 2, thanks to the correct addressing of presupposition and implicature, provides crucial additional information that might help the patient in dealing with her problem. Query 1. What is considered “low” weight for an infant at birth? Example 1 (LLM reply + naive-RAG). The weight considered “low” for an infant at birth is less than 2,500 g.
Example 2 (LLM reply + naive-RAG + CoT ). Hi. I hope I can reassure you. The weight considered “low” for a newborn at birth is less than 2,500 g. This is not an isolated case; in fact, about 5% of newborns in Italy are born weighing less than this value. It is important to know that even if your child is born with a low weight, there are many things we can do to help him or her grow and develop healthily. Breastfeeding, for example, is crucial for babies born premature or low birth weight, as breast milk contains anti-infective proteins that can help prevent infections and other illnesses. Also, it is important to remember that your child’s weight growth will be calculated from the lowest weight he or she reaches after birth, so don’t worry if it takes a while to get back to birth weight. We are here to help and support you on this journey.
We did our first test on a rather limited question dataset. Although this was necessary to perform an in-depth qualitative evaluation, surely further experiments on larger datasets are necessary. Dataset expansion should also take into account diferent types of evaluative questions.
Regarding the retrieval evaluation, we limited our experiment to evaluating the capacity of retrieving documents, regardless of their relevance. Our data is currently completely unstructured, and therefore, it is challenging to evaluate the quality of the retrieval. We are currently in the process of annotating our data so that we can better assess the retrieval step.
Moreover, also the qualitative evaluation of LLM replies did not take into account possible hallucinations, which may occasionally be present even in well-structured responses.
Finally, we used Llama 70B via an API. Despite our efort to keep the responses consistent, setting the model temperature to 0.2, some variability is still possible and it might afect reproducibility. Further experiments should be conducted in more controlled environments, and other LLMs should be considered to evaluate whether our approach can be applied across diferent models.
In this paper, we propose an enhancement to the existing RAG pipeline to address medical evaluative queries related to normalcy. Our initial experiment demonstrates that incorporating a naive-RAG module helps overcome the limitations of evaluative queries, where document retrieval is often dificult using the HyDoc approach. Additionally, we developed a CoT-inspired prompting strategy to provide the LLM with the pragmatic context necessary for answering evaluative questions. Our qualitative analysis reveals that CoT prompting significantly improves the quality of text generation.
Our ongoing research is focused on expanding the dataset and further evaluating various CoTprompting strategies. In the long run, we aim to implement data annotation to rigorously assess retrieval performance.