Large Language Models (LLMs) such as GPT-4 [2] and LLaMA 3 [3] have significantly advanced the field of natural language processing (NLP) by demonstrating impressive performance across various tasks and exhibiting emergent abilities that push the boundaries of artificial intelligence [4]. However, these models face challenges such as generating unreliable outputs due to issues like hallucination and outdated parametric memories [5].
Retrieval-Augmented Generation (RAG) models have shown promise in addressing these issues by integrating externally retrieved information to support more effective performance on complex, knowledge-intensive tasks [6]. Despite these advancements, the deployment of RAG systems within broader AI frameworks continues to face significant challenges, particularly in handling noise and irrelevance in retrieved data [7].
A key limitation of existing RAG systems is their inability to adapt outputs to users' specific informational and contextual needs. Personalized techniques in information retrieval, such as adaptive retrieval based on user interaction data and context-aware strategies, are increasingly recognized as essential for enhancing user interaction and satisfaction [8,9]. These methods aim to refine the retrieval process dynamically, tailoring it more closely to individual user profiles and situational contexts [10].
The integration of agent-based systems with personalized RAG architectures presents a compelling avenue for research. Such systems utilize a multi-agent framework to simulate complex, adaptive interactions tailored to userspecific requirements [11]. By embedding intelligent, useroriented agents within the RAG framework, these systems can evolve into more sophisticated tools that not only retrieve relevant information but also align it closely with the user's specific preferences and contexts in real-time. Importantly, the personalization strategy employed in these systems is fully transparent to the user, ensuring that the user is aware of how their information is being used to tailor the results. Information Retrieval's Role in RAG Systems (IR-RAG) workshop at SIGIR, 2024, Washington D.C., USA Envelope saber.zerhoudi@uni-passau.de (S. Zerhoudi); michael.granitzer@uni-passau.de (M. Granitzer) Orcid 0000-0003-2259-0462 (S. Zerhoudi); 0000-0003-3566-5507 (M. Granitzer) In this study, we present PersonaRAG, an innovative methodology that extends traditional RAG frameworks by incorporating user-centric agents into the retrieval process. This approach addresses the previously mentioned limitations by promoting active engagement with retrieved content and utilizing dynamic, real-time user data to continuously refine and personalize interactions. PersonaRAG aims to enhance the precision and relevance of LLM outputs, adapting dynamically to user-specific needs while maintaining full transparency regarding the personalization process.
Our experiments, conducted using GPT-3.5, develop the PersonaRAG model and evaluate its performance across various question answering datasets. The results indicate that PersonaRAG achieves an improvement of over 5% in accuracy compared to baseline models. Furthermore, Per-sonaRAG demonstrates an ability to adapt responses based on user profiles and information needs, enhancing the personalization of results. Additional analysis shows that the principles underlying PersonaRAG can be generalized to different LLM architectures, such as Llama 3 70b and Mixture of Experts (MoE) 8x7b [12]. These architectures benefit from the integration of external knowledge facilitated by PersonaRAG, with improvements exceeding 10% in some cases. This evidence indicates that PersonaRAG not only contributes to the progress of RAG systems but also provides notable advantages for various LLM applications, signifying a meaningful step forward in the development of more intelligent and user-adapted information retrieval systems. [13] addressed this issue by employing natural language inference models to select pertinent sentences, thereby enhancing the RAG's robustness. Additionally, advancements have been made in adaptively retrieving information, with systems like those proposed by Jiang et al. [14] dynamically fetching passages that are most likely to improve generation accuracy.
Despite these improvements, RAG systems still face limitations, particularly in adapting their output to the user's specific profile, such as their information needs or intellectual knowledge. This limitation stems from the current design of most RAG systems, which do not typically incorporate user context or personalized information retrieval strategies [15]. Consequently, there exists a gap between the general effectiveness of RAG systems and their applicability in personalized user experiences, where context and individual user preferences play a crucial role.
Personalization in information retrieval is increasingly recognized as essential for enhancing user interaction and satisfaction [16]. Techniques such as user profiling, contextaware retrieval, and adaptive feedback mechanisms are commonly employed to tailor search results to individual users' needs. For instance, Jeong et al. [17] proposed adaptive retrieval strategies that dynamically adjust the retrieval process based on the complexity of the query and the user's historical interaction data. These personalized approaches not only improve user satisfaction but also increase the efficiency of information retrieval by reducing the time users spend sifting through irrelevant information.
The integration of personalized techniques with agentbased systems provides a promising pathway to augment the capabilities of RAG systems. Agent-based systems, particularly in the form of LLM-Based Multi-Agent Frameworks [18], enable the simulation of complex interactions that can lead to more nuanced and contextually appropriate outputs. By incorporating multi-agent systems into RAG frameworks, there is potential for developing more robust and adaptive retrieval mechanisms that can handle a broader range of queries and generate more accurate responses, closely tailored to the specific needs and contexts of individual users.
In conclusion, while significant progress has been made in enhancing the effectiveness and personalization of RAG systems, ongoing research is crucial to address their existing limitations and expand their applications. The integration of personalized information retrieval and agent-based enhancements represents a promising avenue for further enhancing the adaptability and accuracy of RAG systems, potentially leading to intelligent information retrieval tailored to the specific needs of users.
In this section, we present the methodology underlying our PersonaRAG approach, which aims to enhance the ability of Language Large Models (LLMs) to actively engage with, understand, and leverage user profile information for personalized content generation. We begin by discussing the fundamental concepts of Retrieval-Augmented Generation (RAG) models (Section 3.1) and then introduce our Per-sonaRAG technique, which encourages LLMs to actively assimilate knowledge from live search sessions (Section 3.2).
State-of-the-art RAG models, as described in previous studies [19,20,21], employ retrieval systems to identify a set of passages 𝐷 = {𝑑 1 , … , 𝑑 𝑛 } when given a query q. These passages are intended to enhance the generative capabilities of LLMs by providing them with contextually relevant information.
Early versions of RAG models typically employ a traditional retrieval-generation framework, in which the retrieved data set 𝐷 = {𝑑 1 , … , 𝑑 𝑛 } is directly fed into LLMs to generate responses to the query 𝑞. However, these passages often contain irrelevant information, and the direct utilization approach in RAG has been shown to restrict the potential benefits of the RAG framework [22]. This limitation has sparked further discussion on how to improve LLMs by integrating retrieval results and outputs generated by the models themselves [23].
Drawing from the principles of adaptive learning and usercentered design, we develop a new PersonaRAG architecture to enable IR systems to dynamically learn from and adapt to user behavior in real-time. As shown in Figure 2, Per-sonaRAG introduces a three-step pipeline: retrieval, user interaction analysis, and cognitive dynamic adaptation. Unlike traditional IR models that statically respond to queries, PersonaRAG focuses on leveraging live user data to continually refine its understanding and responses without the need for manual retraining.
To understand user behavior from live interactions, Person-aRAG treats the IR system as a cognitive structure capable of receiving, interpreting, and acting upon user feedback [24]. Mimicking human learning behaviors, we establish four distinct agents within the system dedicated to analyzing user interactions from different perspectives: engagement tracking, preference analysis, context understanding, and feedback integration. These agents' roles are detailed in Section 3.2.2.
Following adaptive learning principles, we employ a dynamic adaptation mechanism to assist the IR system in utilizing real-time user data for continuous improvement. This mechanism facilitates the integration of insights gained from User Interaction Analysis into the system's retrieval processes. Specifically, we prompt the system to adjust its query responses based on an initial understanding of the user's needs and refine these responses as more user data becomes available. This approach not only personalizes the search results but also helps in correcting any misalignments or errors in real-time. PersonaRAG employs a highly specialized agent architecture, with each agent focusing on a specific aspect of the information retrieval process. All agents utilize in-context learning, i.e., prompting, to perform their designated tasks. This role specialization allows for the efficient decomposition of complex user queries into manageable tasks [25].
To foster this, we engage the IR system as five specialized agents to analyze user interactions based on retrieved data. At present, the focus is on the functionality and interaction of these agents rather than their individual performance metrics.
User Profile Agent This component manages and updates user profile data, incorporating historical user interactions and preferences [26,27]. It monitors how users interact with search results, such as click-through rates and navigation paths. The User Profile Agent helps the system understand what captures user interest and leads to deeper engagement, enabling personalized search experiences.
This agent is responsible for the initial retrieval of documents based on the user's current query. It accesses both a traditional search index and a more dynamic context-aware system that can consider broader aspects of the query environment. It utilizes user profile data to modify and refine search queries or to prioritize search results. For instance, if a user consistently engages more with certain types of documents or topics, the retrieval agent can boost those document types in the search results, ensuring that the most relevant information is presented to the user.
This agent analyzes the current session in real-time, observing user actions such as clicks, time spent on documents, modifications to the query, and any feedback provided. It creates a session-specific context model that captures the user's immediate needs and interests. The real-time data collected by this agent is used to adjust the ongoing session, potentially re-ranking search results or suggesting new queries based on the user's behavior and preferences. Additionally, the Live Session Agent updates the user profile with new insights gleaned from the session, allowing for a more personalized and efficient search experience in future interactions.
This agent is responsible for re-ranking the documents retrieved by the Contextual Retrieval Agent. It integrates insights from both the User Profile Agent and the Live Session Agent to score and order the documents more effectively. By considering the user's historical preferences and their current session behavior, the Document Ranking Agent ensures that the most relevant and valuable documents are presented to the user in a prioritized manner. This agent continuously adapts its ranking algorithms based on the feedback received from the user and the insights provided by the other agents in the system.
Feedback Agent This agent gathers implicit and explicit feedback during and after user interactions. Implicit feedback includes behavioral data like time spent on documents, click counts, and navigation patterns. Explicit feedback involves direct user input on document relevance and quality, collected through ratings, surveys, or comments. The agent uses this information to train and refine models for other agents, particularly the Document Ranking Agent. This process enhances the system's ability to anticipate user needs and deliver relevant documents based on accumulated feedback and insights.
By dynamically integrating insights from the User Profile Agent, Contextual Retrieval Agent, Live Session Agent, Document Ranking Agent, and Feedback Agent into the IR processes, PersonaRAG not only adapts to immediate user needs but also evolves over time to better anticipate and meet user expectations. This multi-agent approach enables PersonaRAG to embody a truly adaptive and user-focused information retrieval system, leveraging specialized agents to analyze user interactions from different behavioral perspectives and deliver highly personalized and contextually relevant search experiences. The inclusion of the Document Ranking Agent ensures that the most pertinent documents are identified and presented to users, further enhancing the system's ability to effectively satisfy user information needs.
The PersonaRAG framework employs a structured workflow that allows for sequential and parallel processing of tasks, ensuring clarity and consistency in communication between agents through well-defined data structures and protocols [28]. The process involves the User Profile Agent, Contextual Retrieval Agent, Live Session Agent, Document Ranking Agent, and Feedback Agent working together to refine search queries, prioritize relevant results, and improve document scoring and re-ranking based on user profile, session-specific contexts, and feedback.
PersonaRAG's modular design allows for flexibility in the system setup, enabling researchers to focus on the most relevant aspects of the user's profile, session, and feedback data. Agents work collaboratively by utilizing content from the Global Message Pool, which serves as a central hub for interagent communication [28], eliminating inefficiencies and enabling agents to access or update information as required.
The Feedback Agent collects and analyzes implicit and explicit user feedback to generate insights into the effectiveness of retrieval strategies and document relevance. This feedback is used to make dynamic adjustments to the system, refining retrieval methods and altering the weighting of user profile factors. Through this iterative process, Person-aRAG continuously adapts and improves its performance, enhancing the accuracy and user satisfaction of the retrieval results [29].
In this section, we present the experimental setup employed in our study, including the datasets, baseline models, evaluation metrics, and implementation details. We also provide an overview of the prompts used in our experiments.
Our experiments are conducted on three widely used singlehop benchmark datasets in the field of Information Retrieval (IR): NaturalQuestions (NQ) [30], TriviaQA [31], and We-bQuestions (WebQ) [32]. NQ is a well-known dataset in Natural Language Understanding (NLU), consisting of structured questions and corresponding Wikipedia pages annotated with long and short answers. TriviaQA comprises question-answer pairs collected from trivia and quiz-league websites, while WebQ consists of questions selected using the Google Suggest API, with answers being entities in Freebase.
Table 1 summarizes the datasets used in our initial study. Due to the high cost of using language models and the large number of API calls required, we randomly sampled 500 questions from each raw dataset to create more manageable subsets for our experiments. While this sampling approach limits the scope of our study, it allows us to conduct an initial investigation into the performance of different RAG systems on these datasets. We acknowledge that future work with larger sample sizes and more comprehensive experiments will be necessary to draw definitive conclusions. Nonetheless, we believe this preliminary study provides valuable insights into the relative strengths and weaknesses of the tested RAG approaches.
We compare PersonaRAG with several baseline models, including prompt learning and RAG models. The prompt templates used in user interaction analysis and dynamic adaptation are presented in Section 4. 4 the vanilla answer generation model. Following the work of Wei et al. [33], the Chain-of-Thought model is implemented, which generates question rationale results to produce the final results. Additionally, the Guideline model serves as a baseline, generating problem-solving steps and guiding Language Models (LLMs) to generate the answer.
For the RAG-based baselines, two models are implemented: vanilla RAG and Chain-of-Thought, which include utilizing raw retrieved passages (CoT with Passage) and refining the passages as notes (CoT with Note). The vanilla RAG model directly feeds the top-ranked passages to the LLM. The Chain-of-Note model [1] is also implemented, which refines and summarizes the retrieved passages for generation. Inspired by Self-RAG Asai et al. [34], the Self-Rerank model is conducted, which filters out unrelated contents without fine-tuning LLMs.
When evaluating adaptive models, it is crucial to consider both task performance and user-centric adaptability simultaneously, along with their trade-offs. Therefore, the results are reported using different metrics, some of which measure effectiveness and others measure efficiency.
For effectiveness, accuracy is used, following the standard evaluation protocol in the field of Information Retrieval (IR) [35,36,34]. Accuracy assesses whether the predicted answer contains the ground-truth answer. Both the outputs of the Language Learning Model (LLM) and golden answers are converted to lowercase, and string matching (StringEM) is performed between each golden answer and the model prediction to calculate accuracy.
To evaluate user-centric adaptability, the BLEU-2 score is measured to assess the text similarity between different RAG and baseline setups and how well the generated answers resemble each other. This metric provides insights into the system's ability to generate consistent and coherent responses across various configurations. Additionally, the average sentence length and the average number of syllables of the answers from different RAG setups are reported as a post-hoc analysis. These measures validate whether the RAG system effectively adjusts its responses based on user knowledge levels, ensuring that the generated answers are tailored to the user's understanding and expertise.
Combining these evaluation strategies provides a comprehensive view of both the effectiveness and user-centric adaptability of the RAG system. The accuracy metric ensures that the system generates correct answers, while the BLEU-2 score and post-hoc analysis of sentence length and syllable count confirm the system's ability to adapt to user knowledge levels. As the understanding of user needs and system capabilities evolves, it is essential to continuously refine these metrics to maintain the RAG system's effectiveness in delivering personalized, context-aware responses that cater to the diverse requirements of users in the field of IR.
For a fair comparison and following the work of Mallen et al. [35] and Trivedi et al. [37], the same retriever, a term-based sparse retrieval model known as BM25 [38], is used across all different models. The retrieval model is implemented using the OpenMatch toolkit [39]. For the external document corpus, the KILT-Wikipedia corpus preprocessed by Petroni et al. [40] is used, and the top-k relevant documents are retrieved.
Regarding the LLMs used to generate answers, the Llama 3 model instruct (ref) with 70b parameters, Mixture of Experts (MoE) 8x7b (ref), and the GPT-3.5 model (gpt-3.5turbo-0125) are employed. For the retrieval-augmented LLM design, the implementation details from Trivedi et al. [37] are followed, which include input prompts, instructions, and the number of test samples for evaluation (e.g., 500 samples per dataset).
This subsection presents the prompt templates employed in the construction of the PersonaRAG model. The prompts utilized in the User Interaction Analysis and Cognitive Dynamic Adaptation components are detailed below. The prompt templates used by the baseline models are available in the project repository 1 . In the templates, {question} represents the input question, {global_memory} the Global Message Pool, while {passages} denotes the retrieved passages. Additionally, {cot_answer} is populated with the output generated by the Chain-of-Thought model.
The placeholder {user_profile_answer} is filled with the response produced by the User Profile agent model.
Respectively, {contextual_answer} corresponds to the Contextual Retrieval agent model, {live_session_answer} to the Live Session agent model, {document_ranking_answer} to the Document Ranking agent model, and {feedback_answer} to the Feedback agent model.
Your task is to help the User Profile Agent improve its understanding of user preferences based on ranked document lists and the shared global memory pool.
Task Description: From the provided passages and global memory pool, analyze clues about the user's search preferences. Look for themes, types of documents, and navigation behaviors that reveal user interest. Use these insights to recommend how the User Profile Agent can refine and expand the user profile to deliver better-personalized results.
You are responsible for maintaining and enriching the Global Message Pool, serving as a central hub for inter-agent communication.
Task Description: Using the responses from individual agents and the existing global memory, consolidate key insights into a shared repository. Your goal is to organize a comprehensive message pool that includes agent-specific findings, historical user preferences, sessionspecific behaviors, search queries, and user feedback. This structure should provide all agents with meaningful data points and strategic recommendations, reducing redundant communication and improving the system's overall efficiency.
To solve the problem, Please think and reason step by step, then answer.
Question: {question} Passages: {passages} Reasoning process: 1. Read the given question and passages to gather relevant information. 2. Write reading notes summarizing the key points from these passages.
3. Discuss the relevance of the given question and passages. 4. If some passages are relevant to the given question, provide a brief answer based on the passages. 5. If no passage is relevant, directly provide the answer without considering the passages.
Answer:
Your task is to help the Cognitive Agent enhance its understanding of user insights to continuously improve the system's responses.
In this section, we show the overall experimental results and offer in-depth analyses of our method.
Table 2 summarizes the primary findings for PersonaRAG across various single-hop question answering datasets. The approach was evaluated against multiple baseline models, including large language models (LLMs) without retrievalaugmented generation (RAG), the conventional RAG model, and self-refined variants, such as utilizing raw retrieved passages (CoT with Passage) or refining passages into notes (CoT with Note). PersonaRAG demonstrated superior performance compared to most of the baseline models, achieving significant improvements over the conventional RAG (i.e., vanillaRAG) of over 10%, particularly on the WebQ dataset. It also consistently outperformed the ChatGPT-3.5 model, except on TriviaQA, which we suspect is part of the model's training dataset. These results suggest PersonaRAG's capability to guide LLMs in extracting relevant information through active learning techniques.
Specifically, the performance of RAG models was assessed using the top 3 and 5 ranked passages. While other RAG models generally benefited from more passages, Person-aRAG maintained consistent performance with either 3 or 5 passages, suggesting that 3 passages were adequate for generating accurate answers. PersonaRAG agents played a crucial role in efficiently extracting the necessary information regarding the user's information need to achieve these improvements.
Furthermore, on the WebQ dataset, PersonaRAG achieved accuracy scores of 63.46% and 67.50% using Top-3 and Top-5 passages, respectively, surpassing the vanillaRAG model by 25% and 17.36%, and nearly all other baseline models (except for Chain-of-Thought using Top-5, which performed equally). On the NQ dataset, PersonaRAG maintained similarly robust performance with scores of 49.02% and 48.78%, outperforming all baselines (except for Chain-of-Thought and Self-Rerank (SR) using Top-5). This pattern was further validated by experiments on other datasets, with results showing that PersonaRAG consistently outperforms conventional RAG models with the capability of providing an answer tailored to the user's interaction and information need. The comprehensive understanding it provides contributes to the generation of accurate and user-centric answers across various question complexities.
Further experiments explored PersonaRAG's adaptive capabilities (Figure 3). BLEU-2 scores compared outputs from Chain-of-Note (consistently best outside PersonaRAG) with other methods. PersonaRAG showed higher similarity scores, indicating its ability to generate responses that address user needs rather than just summarizing input. Additionally, PersonaRAG provides personalized answers tailored to user profiles, extending beyond mere information provision.
The Chain-of-Note approach demonstrated comparable performance to the Chain-of-Thought approach, implying that both techniques effectively extract pertinent information from the retrieved passages and adapt it to align with the user's information need.
In contrast, vanillaGPT and vanillaRAG outputs differed significantly from the Chain-of-Note approach, indicating that counterfactual cognition often leads to diverse outcomes rather than focusing solely on query-relevant content. This suggests LLMs can construct knowledge from multiple perspectives and customize responses based on user understanding.
Post-hoc analyses of average sentence length and syllable count across RAG configurations provided insights into the system's ability to adapt responses to user comprehension levels. These observations highlight PersonaRAG's capacity to synthesize knowledge from various perspectives and tailor responses to different levels of user expertise.
This experiment evaluates the quality of knowledge construction using different large language models (LLMs). As illustrated in Table 3, the PersonaRAG outcomes are used to prompt open-source LLMs, specifically LLaMA3-70B and MoE-8x7b, to generate accurate answers. Compared to LLMs without retrieval-augmented generation (w/o RAG), vanilla RAG and Chain-of-Note often exhibit lower performance. This result suggests that retrieved passages can act as noise, adversely affecting model performance even after refinement through note generation. One primary reason for this behavior is that both LLaMA3-70B and MoE-8x7b struggle to efficiently analyze and identify relevant knowledge due to limitations in their processing capacities.
In contrast, the PersonaRAG method provides notable performance improvements: over 8% for LLaMA3-70B and more than 10% for MoE-8x7b across all datasets, underscoring its effectiveness. The PersonaRAG methodology distinguishes itself from the Chain-of-Note approach by offering a cognitive framework that connects retrieved passages with prior knowledge. This framework models the instructor's (GPT-3.5) reasoning process, guiding the student models (LLaMA3-70B and MoE-8x7b) to better understand knowledge retrieved from passages. The results demonstrate that the LLMs are capable of selecting appropriate passages to build more accurate responses, highlighting the benefits of the PersonaRAG approach for improving generalization.
Finally, we randomly sample one case in Table to demonstrate the effectiveness of PersonaRAG.
The user interaction analysis mechanism effectively generates comprehensive results by integrating foundational and advanced insights from user data. Retrieved passages provide critical clues for answering questions, while agent analyses summarize and illustrate the applicability of external information to user queries. The cognitive dynamic adaptation module refines initial chain-of-thought responses using these insights, generating accurate answers. For example, including knowledge about the "theft of the Mona Lisa in 1911," "Vincenzo Peruggia," and "Florence" enhances the reasoning process's precision and detail. This demonstrates PersonaRAG's effectiveness in helping IR agents combine external knowledge with intrinsic user data to produce well-informed responses.
This paper proposes PersonaRAG, which constructs the retrieval-augmentation architecture incorporating user interaction analysis and cognitive dynamic adaptation. Per-sonaRAG builds the user interaction agents and dynamic cognitive mechanisms to facilitate the understanding of user needs and interests and enhance the system capabilities to deliver personalized, context-aware responses with the intrinsic cognition of LLMs.
Furthermore, PersonaRAG demonstrates effectiveness in leveraging external knowledge and adapting responses based on user profiles, knowledge levels, and information needs to support LLMs in generation tasks without finetuning. However, this approach requires multiple calls to the LLM's API, which can introduce additional time latency and increase API calling costs when addressing questions. The process involves constructing the initial Chain-of-Thought, processing the User Interaction Agents results, and executing the Cognitive Dynamic Adaptation to generate the final answer. Furthermore, the inputs to LLMs in this approach tend to be lengthy due to the inclusion of extensive retrieved passages and the incorporation of user needs, interests, and profile construction results. These factors can impact the efficiency and cost-effectiveness of the PersonaRAG approach in practical applications of Information Retrieval (IR) systems.
Future research will aim to optimize the process by reducing API calls and developing concise representations of user profiles and retrieved information without compromising response quality. We also plan to explore more user-centric agents to better capture writing styles and characteristics of RAG users/searchers. This will enhance the system's ability to understand and adapt to individual preferences, improving personalization and relevance in IR tasks.
This work has received funding from the European Union's Horizon Europe research and innovation program under grant agreement No 101070014 (OpenWebSearch.EU, https: //doi.org/10.3030/101070014).