Recent advancements in Machine Learning (ML), Natural Language Processing (NLP), and the proliferation of smart devices have significantly enhanced conversational AI. This progress has led to a variety of commercial conversational services that enable natural spoken interactions, thereby increasing the demand for more human-centric approaches in information retrieval (IR) [1]. The objective of Conversational Information Retrieval (CIR) is to facilitate information seeking through multi-turn natural language dialogues between users and systems, a longstanding yet challenging area of research [2]. Figure 1 illustrates a comparison between a web search in a traditional IR system and a dialogue in CIR. Traditional IR systems typically focus solely on the user's current query, treating each query as an independent event. These systems do not account for the influence of previous queries on the current search.
UM-CIR 2024: The 1st Workshop on User Modelling in Conversational Information Retrieval, December 12, 2024, Tokyo, Japan Envelope xiao.fu.20@ucl.ac.uk (X. Fu); aldo.lipani@ucl.ac.uk (A. Lipani); Noriko.Kando@nii.ac.jp (N. Kando) Orcid 0000-0003-4676-8608 (X. Fu); 0000-0002-3643-6493 (A. Lipani); 0000-0002-2133-0215 (N. Kando) In contrast, CIR systems emphasize the importance of context in the retrieval process. Previous queries and responses influence the current response. The second part of Figure 1 shows a conversation between a user and a CIR system from the FaithDial dataset [3], where the interaction is more natural, and queries form sequences akin to conversations. For instance, the word they in the second turn of the conversation refers to the shops mentioned in the system's previous response.
This shift in communication style introduces several differences between traditional IR and CIR. The conversation-like query format in CIR allows for richer, more interactive responses. Firstly, CIR systems can provide more detailed responses compared to traditional IR, which typically returns documents directly. CIR systems can refine information needs and the search space through user or system revealment. Secondly, users have more strategic options in their interactions. If unsatisfied with a response, users can ask follow-up questions to refine their search or request additional information. This necessitates the introduction of ad-hoc search methods (such as session-based and task-based search) in CIR, enabling the system to refine the search space based on the conversation, querying related sections instead of the entire database. Thirdly, context is crucial in CIR for understanding user queries, unlike traditional IR where queries are treated independently. For example, the meaning of pronouns can change depending on their order in a conversation.
Currently, CIR systems are predominantly used for simple tasks, as they are not sufficiently effective for complex and exploratory information-seeking conversations. Nonetheless, advancements in key components, particularly in ML, are driving a trend within the IR community towards more conversational methodologies [4,5].
Despite the progress, several critical questions remain unresolved, presenting challenges to the development of CIR systems. Zamani et al. [5] identifies four key directions in the CIR field with potential for significant advancements:
This paper primarily focuses on the first and last directions, which are both challenging and interconnected. The first direction involves modelling and producing conversational interactions, addressing the uncertainty of information needs between multiple agents through a mixed-initiative approach. Additionally, understanding long-term conversational interactions and addressing associated privacy and transparency concerns are critical topics in this direction.
The final direction pertains to the measurement and evaluation of CIR. Both academia and industry face limitations due to the absence of a robust definition of success in this field. As CIR continues to evolve, there is an urgent need for an evaluation paradigm that transcends the traditional Cranfield Paradigm, especially in frontier tasks such as personalized evaluation and transparency.
These two directions are highly interconnected. The definition of success relies on the proper modelling of conversational interactions, while precise measures support the modelling process. This paper introduces a new potential contribution to this field: introducing user simulation to CIR.
In this paper, we present a framework for the automated evaluation of CIR systems via user simulation. This paper also includes the recent advancements within this framework, while exploring the encountered opportunities and challenges. Section 3 details the two-stage user simulation prototype, which amalgamates psychological principles and ML to enhance explainability, and utilizes Large Language Models (LLMs) to sustain high performance. Section 4 focuses on the application of this prototype in assessing CIR systems. The methodology proposed aims to connect user simulation with well-established CIR conversation modelling approaches, such as effort and cost, and seeks to align the simulations closely with real user interactions through indirect assessment. Section 5 discusses the challenges and future perspectives in this field.
Conversational search, a well-established field, continues to be a popular research topic due to its relevance for modern devices with small or no screens [1].
Despite significant progress, the evaluation of CIR remains relatively underdeveloped [6,7,8]. While CIR extends functionalities from traditional IR systems [9], many studies still rely on conventional metrics such as Mean Average Precision (MAP), Normalized Discounted Cumulative Gain (nDCG), and Mean Reciprocal Rank (MRR) [7,10,11]. At the heart of these metrics is the concept of relevance, where the documents retrieved are fitting to the topics (keywords) sought by the user [12]. Additionally, metrics from other domains like ROUGE and BLEU are also utilized [6,13,14]. However, recent research indicates that without real user interaction, these metrics may not accurately reflect user satisfaction [15,16].
User satisfaction, a highly abstract and subjective measure, pertains to the overall experience and interaction with a search system [10,8]. It is defined as the fulfillment users achieve in pursuing their goals [17]. Extensive research has been conducted to understand this measure [10,15,18,19,20,21,22]. Studies like Yilmaz et al. [18] offer various metrics to reflect user satisfaction, considering factors such as effort. Some studies further break down user satisfaction into query-level satisfactions [19,20], although others, such as Järvelin et al. [21], argue that summing query-level satisfaction misses the contextual journey where user queries are interconnected. Unlike traditional search systems, conversational search systems allow users to ask follow-up questions to refine their answers [22].
Despite widespread adoption, measuring user satisfaction remains an open question. Many studies rely on real-time user participation to gather feedback, providing fresh and realistic insights but requiring significant resources and participant incentives [10,15]. Alternatively, satisfaction prediction proxies using deep learning models offer a computational approach, overcoming temporal and spatial constraints but demanding high-quality computational resources and datasets [8].
Emerging trends in user simulation offer promising solutions to these challenges [5,2], though prior studies are not yet comprehensive. Gao et al. [23] noted that earlier simulators relied on randomly generated scenarios to approximate users' states of mind. While such models prove beneficial in specific domains like recommender systems [24], issues of explainability and scrutability remain unresolved. Our proposed framework addresses these issues by incorporating a user-profile-based personalized simulation approach, facilitating easier alignment with actual user behaviours and providing a scrutable means to control the simulation by modifying textual user profiles.
Azzopardi et al. [25] define simulation as the imitation of the operation of real-world phenomena. Simulations enable detailed experimental design and control tailored to specific research questions. These high-level controls allow for experiments with user simulations to be conducted with several advantages [25].
Firstly, what-if experiments can be performed by setting up different scenarios [26,27]. Secondly, user simulators ensure the repeatability of experimental results. Additionally, user simulations can achieve these benefits at a low cost [28].
In the IR community, user simulation methods are primarily divided into cognitive and statistical approaches [29]. Cognitive approaches were among the first used in this field. Belkin [30] described users, information resources, and IR models, characterizing users by their objectives, problems, and knowledge. Subsequent studies expanded on this foundation [31,32,33].
In contrast, statistical approaches focus on analyzing user behaviours and satisfaction [29,34,35,36,37]. These approaches underpin early user simulators based on statistical models [38,39]. Although these simulators heavily relied on corpora, they faced limitations such as the diversity of user intent [29].
Performance of models trained with priming factors in predicting users' actions, while AUC refers to the area under P-R curves. Adopted form Fu and Lipani [46].
Dataset Precision Recall F1 AUC All 0.844 0.574 0.684 0.835 Topi [47] 0.977 0.951 0.963 0.995 TREC 1 1.000 0.625 0.769 0.800 Cran [28] 0.795 0.875 0.833 0.883 QReCC [48] 0.953 0.997 0.974 0.990 ORC [49] 0 Agenda-based user simulations are popular due to their realistic responses and straightforward dialogue strategies [40,41,42]. The latest trend involves employing deep learning models, including adversarial generative approaches [43], reinforcement learning [44], and inverse reinforcement learning to abstract knowledge from data [45].
Evaluating CIR with user simulations is becoming a key trend, enabling efficient and cost-effective evaluation at various levels of CIR [2]. Our proposed prototype integrates benefits from the aforementioned approaches. Initially, it simulates user behaviour guided by statistical signals derived from the context, enhancing explainability. Subsequently, the prototype employs deep learning models, utilizing textual user profiles to produce realistic and diverse responses within controlled parameters, thereby enabling further exploration of the target system. The details of the user simulation prototype are in the next section.
In this section, we introduce an two-stage prototype for constructing a robust user simulation for CIR. The main structure of the prototype is divided into two parts to control simulated users: the Action Predictor and the Response Generator, as illustrated in Fig 2.
The Action Predictor controls the simulated users' actions during the current conversation. This concept originates from the psychology community's notion of priming, which refers to the unconscious influence of past experiences on current performance or behaviour [50,51]. Many studies have established models to explain this mechanism [52,50]. For instance, Tulving et al. [53] conducted an experiment where participants viewed a list of 96 words and later completed graphemic word fragments both one hour and seven days after studying the list. The results demonstrated a significant influence of the word list on subsequent tests.
As shown in Table 1, results from Fu and Lipani [46] demonstrate the potential of predicting users' next actions. The benefits of using lexical and textual patterns to predict users' next actions include low cost and ease of interpretation, which are valuable for evaluation.
In the current prototype, three user actions are modelled based on the dataset available. Stopping is defined as the action where users opt to end a conversation, typically indicating the conclusion of the exchange. This action is critical for evaluating effects such as the principle of least effort [54]. Sessions of conversation often include Following up on queries that build upon previous interactions, acknowledging missing contexts and references to earlier discussed topics [49]. As noted by Stede and Schlangen [55], an inquisitive user engaged in an ongoing dialogue may express interest in further related subjects as a response to the information provided. Switching topics is commonly seen in information-seeking dialogues, especially when using search systems for data acquisition [56].
As output from the Action Predictor, a general action as described above will be predicted, with the details elaborated in the Response Generator.
The Response Generator will generate realistic and diversified responses based on the conversation, user profiles and the predicted actions from the Action Predictor LLMs such as GPTs and Llama, renowned for their sophisticated natural language processing capabilities, present unique opportunities to enhance Conversational Systems through mechanisms like pre-training, fine-tuning, and prompting [57]. The ability of LLMs to mimic diverse demographic characteristics offers a novel approach to simulating user behaviour and preferences [58]. High-quality user simulations, which closely mirror real user behaviour distributions, can significantly advance CRS development, currently dependent on real data for training, with its inherent constraints and disadvantages.
Ramos et al. [59] offers a valuable method for generating user profiles from the Amazon dataset, introducing a more compact style of personalization into the user simulator. Table 2 demonstrates the agreement between simulated and real users' responses to the same items in the Amazon dataset. Since the simulated users are based on LLMs and prompts with user profiles, the results suggest the potential of generating personalized responses based on textual user profiles.
Agreements (Cohen's 𝐾 𝑐 , Randolph's 𝐾 𝑟 and Krippendorff's 𝛼) between simulated users and actual users in response.Cohen's index considers the marginal distribution of categories, Randolph's assumes a uniform distribution, and Krippendorff's offers a broader approach to assessing agreement. Fair agreements for each metric (>0.2) are marked as bold. Three simulation methods are evaluated: Real UP, where users are simulated based on their profiles; Rand UP, involving users simulated with random profiles; and Rand Sc, where scores are randomly generated based on the dataset's historical distribution.
𝐾 𝑐 𝐾 𝑟 𝛼 Real UP 0.20 0.34 0.20 Rand UP 0.14 0.26 0.14 Rand Sc -0.03 0.14 -0.03
In this simulation, responses are tailored based on user profiles, which consist of concise text that summarizes the attributes of users in a few succinct sentences. This can include motivations for task-oriented CIR systems. Furthermore, the Response Generator is tasked with handling clarification questions posed by the CIR system. At the conclusion of this phase, the user simulator is equipped to interact with CIR systems. The subsequent section proposes a linkage as the remaining component of this framework, specifically addressing the evaluation of the CIR system using this user simulator, given the absence of a direct indicator from the user simulation on the quality of the target CIR system.
This section discusses potential methods for applying evaluation tasks based on the user simulation prototype established in Section 3.
As introduced in Section 2, the target of evaluation originates from modelling users and conversations. According to Yilmaz et al. [18], user satisfaction can be reflected by the effort exerted. Effort also forms the foundation of modelling user actions, particularly stopping behaviours.
In the IR community, several studies have been conducted to depict stopping behaviours [60,61,62,63]. These studies aim to quantify the feeling of having "enough. " For instance, users may decide to stop a conversation when they feel frustrated or satisfied.
A previous study by Fu and Lipani [46] provided a reliable method for predicting stopping behaviours. The subsequent step is to explore the relationship between each stopping point and user satisfaction. The final part of the evaluation focuses on assessing the user simulation. To effectively perform evaluation for CIR, the user simulation must exhibit not only a diversity of reasonable responses but also a high alignment with real users, particularly in reflecting satisfaction or frustration.
Following the study by Fu et al. [28], where direct and indirect assessments in CIR show substantial agreement, the alignment between user simulation and real users can be evaluated. Real users will review the conversations between the user simulation and the system to determine if the simulated user appears satisfied.
Figure 3 illustrates the operational flow within the evaluation component of the framework. In this component, a classifier, integrated with the user simulator, is utilized to predict user satisfaction. This classifier is aligned with annotations from real users. Feedback from these real annotators is employed to train both the user simulator and the classifier. The user simulator aims to accurately mimic real user behaviour through the Actions Predictor and the Response Generator. Similarly, the classifier is trained to align its judgments with those of real users regarding the same conversation.
During the development of this framework, numerous emerging trends were observed, particularly in LLMs. These observations are presented in the following section.
This represents a significant transformation since 2021, as the term Large Language Models (LLMs) has gained popularity. This shift has introduced both opportunities and challenges for contemporary research.
The integration of LLMs is not limited to the IR community; the ML community is also embracing IR techniques. A prominent development in this space is Retrieval-Augmented Generation (RAG), which enhances LLMs in domain-specific or knowledge-intensive tasks [64].
RAG involves multiple retrieval processes to enrich the context, going beyond traditional single retrieval methods. For instance, Self-RAG [65] refines the RAG framework by enabling LLMs to actively determine the optimal moments and content for retrieval, thus improving the efficiency and relevance of the sourced information. The key aspect here is not merely multiple retrievals, but the reliance on the judgment of LLMs, indicating that LLMs can further participate in the processing with minimal human intervention.
For the evaluation of RAG, integrating typical LLMs alone may not suffice. A realistic inquiry is how agents based on LLMs can assess the responses from RAG systems that incorporate retrieved documents. One feasible approach is using RAG to evaluate itself. Here, the crucial aspects include not only the quality of the conversation and the retrieval process but also how effectively the documents are presented.
Moreover, in the specialized domain of CIR, where the systems are relatively light, LLMs can still serve as experts. The following section provides an example.
The TREC Interactive Knowledge Assistance Track (iKAT) builds upon the foundational work of the TREC Conversational Assistance Track (CAsT) [66], with a key difference being the addition of personal context for each user in the dataset. The primary task remains similar to TREC CAsT-retrieving and ranking documents from the corpus at each turn of the given conversations.
The best performance in iKAT 2023 introduced a novel approach [67]. In this approach, the LLM generates an initial answer to the user's query based on the context of the conversation and the user profile. This answer is derived through reasoning over the context and the user's profile, but it is not grounded in the documents within the collection. Subsequently, the LLM generates a set of five queries to achieve this answer.
The controversial aspect of this approach is using the LLM-generated answer as the target without initial retrieval, followed by employing an IR system to achieve it. This process assumes that LLMs' answers are sufficiently accurate. Alternatively, it suggests that the documents have likely been exposed to the LLMs, raising concerns of potential data leakage.
Training LLMs from scratch is a challenging task for most research groups due to high costs and the lack of storage and computational resources. The most common practice involves fine-tuning a public base version of LLMs and accessing them via APIs. Top LLMs are trained on a substantial portion of internet text documents, making it nearly impossible to prevent data leakage once any public data is used in a study.
Furthermore, the widespread use of LLMs will inevitably introduce LLM-generated text back into the internet, posing a significant challenge that has already raised considerable concerns within the community.
This paper proposes a framework for evaluating CIR systems automatically using a user simulator prototype. The framework comprises two main components:
1. A prototype of user simulation that leverages the advancements from both psychology and ML fields to conduct realistic and scrutable simulations targeted at CIR systems. 2. A component that employs sophisticated conversation modelling concepts from the IR community to provide reasonable feedback aimed at predicting user satisfaction alongside the user simulation prototype.
In addition to the framework, this study also presents emerging trends observed with the promising development of LLMs. The advent of systems such as RAG introduces both opportunities and challenges. It raises concerns that the current practices within the community utilizing LLMs may lead to increased data leakages.