Large Language Models (LLMs) struggle with generating reliable outputs due to outdated knowledge and hallucinations. RetrievalAugmented Generation (RAG) models address this by enhancing LLMs with external knowledge, but often fail to personalize the retrieval process. This paper introduces PersonaRAG, a novel framework incorporating user-centric agents to adapt retrieval and generation based on real-time user data and interactions. Evaluated across various question answering datasets, PersonaRAG demonstrates superiority over baseline models, providing tailored answers to user needs. The results suggest promising directions for user-adapted information retrieval systems. Findings and resources are available at https://github.com/padas-lab-de/ir-rag-sigir24-persona-rag.
Large Language Models (LLMs) such as GPT-4 [
Retrieval-Augmented Generation (RAG) models have
shown promise in addressing these issues by integrating
externally retrieved information to support more efective
performance on complex, knowledge-intensive tasks [
A key limitation of existing RAG systems is their inability
to adapt outputs to users’ specific informational and
contextual needs. Personalized techniques in information retrieval,
such as adaptive retrieval based on user interaction data and
context-aware strategies, are increasingly recognized as
essential for enhancing user interaction and satisfaction [
The integration of agent-based systems with
personalized RAG architectures presents a compelling avenue for
research. Such systems utilize a multi-agent framework
to simulate complex, adaptive interactions tailored to
userspecific requirements [
Information Retrieval’s Role in RAG Systems (IR-RAG) workshop at SIGIR, (M. Granitzer)
Retrieval-Augmented Generation (RAG) systems have
emerged as a significant advancement in natural language
processing and machine learning, enhancing language
models by integrating external knowledge bases to improve
performance across various tasks, such as question answering,
dialog understanding, and code generation [
Recent developments in RAG systems have focused on reifning these models to better handle the noise and irrelevant information often retrieved during the process. Xu et al. CEUR
ceur-ws.org
[
Despite these improvements, RAG systems still face
limitations, particularly in adapting their output to the user’s
specific profile, such as their information needs or
intellectual knowledge. This limitation stems from the current
design of most RAG systems, which do not typically
incorporate user context or personalized information retrieval
strategies [
Personalization in information retrieval is increasingly
recognized as essential for enhancing user interaction and
satisfaction [
The integration of personalized techniques with
agentbased systems provides a promising pathway to augment
the capabilities of RAG systems. Agent-based systems,
particularly in the form of LLM-Based Multi-Agent
Frameworks [
In conclusion, while significant progress has been made in enhancing the efectiveness and personalization of RAG systems, ongoing research is crucial to address their existing limitations and expand their applications. The integration of personalized information retrieval and agent-based enhancements represents a promising avenue for further enhancing the adaptability and accuracy of RAG systems, potentially leading to intelligent information retrieval tailored to the specific needs of users.
In this section, we present the methodology underlying our PersonaRAG approach, which aims to enhance the ability of Language Large Models (LLMs) to actively engage with, understand, and leverage user profile information for personalized content generation. We begin by discussing the fundamental concepts of Retrieval-Augmented Generation (RAG) models (Section 3.1) and then introduce our PersonaRAG technique, which encourages LLMs to actively assimilate knowledge from live search sessions (Section 3.2).
State-of-the-art RAG models, as described in previous
studies [
Early versions of RAG models typically employ a
traditional retrieval-generation framework, in which the
retrieved data set = { 1, … , } is directly fed into LLMs
to generate responses to the query . However, these
passages often contain irrelevant information, and the direct
utilization approach in RAG has been shown to restrict the
potential benefits of the RAG framework [
Drawing from the principles of adaptive learning and usercentered design, we develop a new PersonaRAG architecture to enable IR systems to dynamically learn from and adapt to user behavior in real-time. As shown in Figure 2, PersonaRAG introduces a three-step pipeline: retrieval, user interaction analysis, and cognitive dynamic adaptation. Unlike traditional IR models that statically respond to queries, PersonaRAG focuses on leveraging live user data to continually refine its understanding and responses without the need for manual retraining.
To understand user behavior from live interactions,
PersonaRAG treats the IR system as a cognitive structure capable of
receiving, interpreting, and acting upon user feedback [
Following adaptive learning principles, we employ a dynamic adaptation mechanism to assist the IR system in utilizing real-time user data for continuous improvement. This mechanism facilitates the integration of insights gained from User Interaction Analysis into the system’s retrieval processes. Specifically, we prompt the system to adjust its query responses based on an initial understanding of the user’s needs and refine these responses as more user data becomes available. This approach not only personalizes the search results but also helps in correcting any misalignments or errors in real-time.
PersonaRAG employs a highly specialized agent
architecture, with each agent focusing on a specific aspect of the
information retrieval process. All agents utilize in-context
learning, i.e., prompting, to perform their designated tasks.
This role specialization allows for the eficient
decomposition of complex user queries into manageable tasks [
User Profile Agent This component manages and
updates user profile data, incorporating historical user
interactions and preferences [
Live Session Agent This agent analyzes the current session in real-time, observing user actions such as clicks, time spent on documents, modifications to the query, and any feedback provided. It creates a session-specific context model that captures the user’s immediate needs and interests. The real-time data collected by this agent is used to adjust the ongoing session, potentially re-ranking search results or suggesting new queries based on the user’s behavior and preferences. Additionally, the Live Session Agent updates the user profile with new insights gleaned from the session, allowing for a more personalized and eficient search experience in future interactions.
Document Ranking Agent This agent is responsible for re-ranking the documents retrieved by the Contextual Retrieval Agent. It integrates insights from both the User Profile Agent and the Live Session Agent to score and order the documents more efectively. By considering the user’s historical preferences and their current session behavior, the Document Ranking Agent ensures that the most relevant and valuable documents are presented to the user in a prioritized manner. This agent continuously adapts its ranking algorithms based on the feedback received from the user and the insights provided by the other agents in the system.
Feedback Agent This agent gathers implicit and explicit feedback during and after user interactions. Implicit feedback includes behavioral data like time spent on documents, click counts, and navigation patterns. Explicit feedback involves direct user input on document relevance and quality, collected through ratings, surveys, or comments. The agent uses this information to train and refine models for other agents, particularly the Document Ranking Agent. This process enhances the system’s ability to anticipate user needs and deliver relevant documents based on accumulated feedback and insights.
By dynamically integrating insights from the User Proifle Agent, Contextual Retrieval Agent, Live Session Agent, Document Ranking Agent, and Feedback Agent into the IR processes, PersonaRAG not only adapts to immediate user needs but also evolves over time to better anticipate and meet user expectations. This multi-agent approach enables PersonaRAG to embody a truly adaptive and user-focused information retrieval system, leveraging specialized agents to analyze user interactions from diferent behavioral perspectives and deliver highly personalized and contextually relevant search experiences. The inclusion of the Document Ranking Agent ensures that the most pertinent documents are identified and presented to users, further enhancing the system’s ability to efectively satisfy user information needs.
The PersonaRAG framework employs a structured
worklfow that allows for sequential and parallel processing of
tasks, ensuring clarity and consistency in communication
between agents through well-defined data structures and
protocols [
PersonaRAG’s modular design allows for flexibility in the
system setup, enabling researchers to focus on the most
relevant aspects of the user’s profile, session, and feedback data.
Agents work collaboratively by utilizing content from the
Global Message Pool, which serves as a central hub for
interagent communication [
The Feedback Agent collects and analyzes implicit and
explicit user feedback to generate insights into the
efectiveness of retrieval strategies and document relevance. This
feedback is used to make dynamic adjustments to the
system, refining retrieval methods and altering the weighting of
user profile factors. Through this iterative process,
PersonaRAG continuously adapts and improves its performance,
enhancing the accuracy and user satisfaction of the retrieval
results [
In this section, we present the experimental setup employed in our study, including the datasets, baseline models, evaluation metrics, and implementation details. We also provide an overview of the prompts used in our experiments.
Our experiments are conducted on three widely used
singlehop benchmark datasets in the field of Information Retrieval
(IR): NaturalQuestions (NQ) [
Table 1 summarizes the datasets used in our initial study. Due to the high cost of using language models and the large number of API calls required, we randomly sampled 500 questions from each raw dataset to create more manageable subsets for our experiments. While this sampling approach limits the scope of our study, it allows us to conduct an initial investigation into the performance of diferent RAG systems on these datasets. We acknowledge that future work with larger sample sizes and more comprehensive experiments will be necessary to draw definitive conclusions. Nonetheless, we believe this preliminary study provides valuable insights into the relative strengths and weaknesses of the tested RAG approaches.
We compare PersonaRAG with several baseline models,
including prompt learning and RAG models. The prompt
templates used in user interaction analysis and dynamic
adaptation are presented in Section 4.4. Initially, the
questionanswering (QA) instruction is fed to ChatGPT to conduct
8,757
8,837
2,032
the vanilla answer generation model. Following the work of
Wei et al. [
For the RAG-based baselines, two models are
implemented: vanilla RAG and Chain-of-Thought, which include
utilizing raw retrieved passages (CoT with Passage) and
refining the passages as notes (CoT with Note). The vanilla
RAG model directly feeds the top-ranked passages to the
LLM. The Chain-of-Note model [
When evaluating adaptive models, it is crucial to consider both task performance and user-centric adaptability simultaneously, along with their trade-ofs. Therefore, the results are reported using diferent metrics, some of which measure efectiveness and others measure eficiency.
For efectiveness, accuracy is used, following the standard
evaluation protocol in the field of Information Retrieval
(IR) [
To evaluate user-centric adaptability, the BLEU-2 score is measured to assess the text similarity between diferent RAG and baseline setups and how well the generated answers resemble each other. This metric provides insights into the system’s ability to generate consistent and coherent responses across various configurations. Additionally, the average sentence length and the average number of syllables of the answers from diferent RAG setups are reported as a post-hoc analysis. These measures validate whether the RAG system efectively adjusts its responses based on user knowledge levels, ensuring that the generated answers are tailored to the user’s understanding and expertise.
Combining these evaluation strategies provides a comprehensive view of both the efectiveness and user-centric adaptability of the RAG system. The accuracy metric ensures that the system generates correct answers, while the BLEU-2 score and post-hoc analysis of sentence length and syllable count confirm the system’s ability to adapt to user knowledge levels. As the understanding of user needs and system capabilities evolves, it is essential to continuously refine these metrics to maintain the RAG system’s efectiveness in delivering personalized, context-aware responses that cater to the diverse requirements of users in the field
For a fair comparison and following the work of Mallen et al.
[
Regarding the LLMs used to generate answers, the Llama
3 model instruct (ref) with 70b parameters, Mixture of
Experts (MoE) 8x7b (ref), and the GPT-3.5 model
(gpt-3.5turbo-0125) are employed. For the retrieval-augmented
LLM design, the implementation details from Trivedi et al.
[
This subsection presents the prompt templates employed in the construction of the PersonaRAG model. The prompts utilized in the User Interaction Analysis and Cognitive Dynamic Adaptation components are detailed below. The prompt templates used by the baseline models are available in the project repository 1. In the templates, {question} represents the input question, {global_memory} the Global Message Pool, while {passages} denotes the retrieved passages. Additionally, {cot_answer} is populated with the output generated by the Chain-of-Thought model.
The placeholder {user_profile_answer} is filled with the response produced by the User Profile agent model. Respectively, {contextual_answer} corresponds to the Contextual Retrieval agent model, {live_session_answer} to the Live Session agent model, {document_ranking_answer} to the Document Ranking agent model, and {feedback_answer} to the Feedback agent model.
User Profile Agent
Your task is to help the User Profile Agent improve its understanding of user preferences based on ranked document lists and the shared global memory pool. Contextual Retrieval Agent
You are a search technology expert guiding the Contextual Retrieval Agent to deliver contextaware document retrieval.
Live Session Agent
Your expertise in session analysis is required to assist the Live Session Agent in dynamically adjusting results.
Document Ranking Agent
Your task is to help the Document Ranking Agent prioritize documents for better ranking. Feedback Agent
You are an expert in feedback collection and analysis, guiding the Feedback Agent to gather and utilize user insights.
Question: {question} Passages: {passages} Global Memory: {global_memory} Task Description: Using the retrieved passages and global memory pool, identify methods for collecting implicit and explicit user feedback. Suggest ways to refine feedback mechanisms to align with user preferences, such as ratings, surveys, or behavioral data. Your recommendations should guide the Feedback Agent in updating other agents' models for more personalized and relevant results.
Global Message Pool
You are responsible for maintaining and enriching the Global Message Pool, serving as a central hub for inter-agent communication. Question: {question} Agent Responses: {agent_responses} Existing Global Memory: {global_memory} Task Description: Using the responses from individual agents and the existing global memory, consolidate key insights into a shared repository.
Your goal is to organize a comprehensive message pool that includes agent-specific findings, historical user preferences, sessionspecific behaviors, search queries, and user feedback. This structure should provide all agents with meaningful data points and strategic recommendations, reducing redundant communication and improving the system's overall efficiency.
Chain-of-Thought
To solve the problem, Please think and reason step by step, then answer.
Question: {question} Passages: {passages} Reasoning process: 1. Read the given question and passages to gather relevant information. 2. Write reading notes summarizing the key points from these passages. 3. Discuss the relevance of the given question and passages. 4. If some passages are relevant to the given question, provide a brief answer based on the passages. 5. If no passage is relevant, directly provide the answer without considering the passages.
Answer: Cognitive Agent
Your task is to help the Cognitive Agent enhance its understanding of user insights to continuously improve the system's responses. Question: {question} Initial Response: {cot_answer}
w/o RAG vanillaRAG Self-Refined PersonaRAG gpt-3.5-turbo-0125 Guideline Chain-of-Thought (CoT) Chain-of-Note (CoN) Self-Rerank (SR)
In this section, we show the overall experimental results and ofer in-depth analyses of our method.
crucial role in eficiently extracting the necessary information regarding the user’s information need to achieve these improvements.
Furthermore, on the WebQ dataset, PersonaRAG achieved accuracy scores of 63.46% and 67.50% using Top-3 and Top-5 passages, respectively, surpassing the vanillaRAG model by 25% and 17.36%, and nearly all other baseline models (except for Chain-of-Thought using Top-5, which performed equally). On the NQ dataset, PersonaRAG maintained similarly robust performance with scores of 49.02% and 48.78%, outperforming all baselines (except for Chain-of-Thought and Self-Rerank (SR) using Top-5). This pattern was further validated by experiments on other datasets, with results showing that PersonaRAG consistently outperforms conventional RAG models with the capability of providing an answer tailored to the user’s interaction and information need. The comprehensive understanding it provides contributes to the generation of accurate and user-centric answers across various question complexities.
Further experiments explored PersonaRAG’s adaptive capabilities (Figure 3). BLEU-2 scores compared outputs from Chain-of-Note (consistently best outside PersonaRAG) with other methods. PersonaRAG showed higher similarity scores, indicating its ability to generate responses that address user needs rather than just summarizing input. Additionally, PersonaRAG provides personalized answers tailored to user profiles, extending beyond mere information provision.
The Chain-of-Note approach demonstrated comparable performance to the Chain-of-Thought approach, implying that both techniques efectively extract pertinent information from the retrieved passages and adapt it to align with the user’s information need.
In contrast, vanillaGPT and vanillaRAG outputs difered significantly from the Chain-of-Note approach, indicating that counterfactual cognition often leads to diverse outcomes rather than focusing solely on query-relevant content. This suggests LLMs can construct knowledge from multiple perspectives and customize responses based on user understanding.
Post-hoc analyses of average sentence length and syllable count across RAG configurations provided insights into the system’s ability to adapt responses to user comprehension levels. These observations highlight PersonaRAG’s capacity to synthesize knowledge from various perspectives and tailor responses to diferent levels of user expertise. (a) Text Similarity for Top-3 Passages
This experiment evaluates the quality of knowledge construction using diferent large language models (LLMs). As illustrated in Table 3, the PersonaRAG outcomes are used to prompt open-source LLMs, specifically LLaMA3-70B and MoE-8x7b, to generate accurate answers.
Compared to LLMs without retrieval-augmented generation (w/o RAG), vanilla RAG and Chain-of-Note often exhibit lower performance. This result suggests that retrieved passages can act as noise, adversely afecting model performance even after refinement through note generation. One primary reason for this behavior is that both LLaMA3-70B and MoE-8x7b struggle to eficiently analyze and identify relevant knowledge due to limitations in their processing capacities.
In contrast, the PersonaRAG method provides notable performance improvements: over 8% for LLaMA3-70B and more than 10% for MoE-8x7b across all datasets, underscoring its efectiveness. The PersonaRAG methodology distinguishes itself from the Chain-of-Note approach by ofering a cognitive framework that connects retrieved passages with prior knowledge. This framework models the instructor’s (GPT-3.5) reasoning process, guiding the student models (LLaMA3-70B and MoE-8x7b) to better understand knowledge retrieved from passages. The results demonstrate that the LLMs are capable of selecting appropriate passages to build more accurate responses, highlighting the benefits of the PersonaRAG approach for improving generalization. Finally, we randomly sample one case in Table to demonstrate the efectiveness of PersonaRAG.
The user interaction analysis mechanism efectively generates comprehensive results by integrating foundational and advanced insights from user data. Retrieved passages provide critical clues for answering questions, while agent analyses summarize and illustrate the applicability of external information to user queries. The cognitive dynamic adaptation module refines initial chain-of-thought responses using these insights, generating accurate answers. For example, including knowledge about the ”theft of the Mona Lisa in 1911,” ”Vincenzo Peruggia,” and ”Florence” enhances the reasoning process’s precision and detail. This demonstrates PersonaRAG’s efectiveness in helping IR agents combine external knowledge with intrinsic user data to produce well-informed responses.
This paper proposes PersonaRAG, which constructs the retrieval-augmentation architecture incorporating user interaction analysis and cognitive dynamic adaptation. PersonaRAG builds the user interaction agents and dynamic cognitive mechanisms to facilitate the understanding of user needs and interests and enhance the system capabilities to deliver personalized, context-aware responses with the intrinsic cognition of LLMs.
Furthermore, PersonaRAG demonstrates efectiveness in leveraging external knowledge and adapting responses based on user profiles, knowledge levels, and information needs to support LLMs in generation tasks without finetuning. However, this approach requires multiple calls to the LLM’s API, which can introduce additional time latency and increase API calling costs when addressing questions. The process involves constructing the initial Chain-of-Thought, processing the User Interaction Agents results, and executing the Cognitive Dynamic Adaptation to generate the final answer. Furthermore, the inputs to LLMs in this approach tend to be lengthy due to the inclusion of extensive retrieved passages and the incorporation of user needs, interests, and profile construction results. These factors can impact the efifciency and cost-efectiveness of the PersonaRAG approach in practical applications of Information Retrieval (IR) systems.
Future research will aim to optimize the process by reducing API calls and developing concise representations of user profiles and retrieved information without compromising response quality. We also plan to explore more user-centric agents to better capture writing styles and characteristics of RAG users/searchers. This will enhance the system’s ability to understand and adapt to individual preferences, improving personalization and relevance in IR tasks.
This work has received funding from the European Union’s Horizon Europe research and innovation program under grant agreement No 101070014 (OpenWebSearch.EU, https: //doi.org/10.3030/101070014).