Recent advancements in the field of Conversational Information Retrieval (CIR) have increased the demand for more sophisticated modelling and evaluation approaches. This paper introduces a novel framework for user simulation in CIR, aimed at enhancing the modeling and evaluation of user interactions. Additionally, this study explores the potential integration of large language models (LLMs) within this domain. Furthermore, the paper anticipates future developments in CIR, particularly in the context of the widespread use of LLMs. The study emphasizes the necessity for robust evaluation paradigms that go beyond traditional methods to efectively measure the success of CIR systems.
Recent advancements in Machine Learning (ML), Natural Language Processing (NLP), and the
proliferation of smart devices have significantly enhanced conversational AI. This progress has led to a variety
of commercial conversational services that enable natural spoken interactions, thereby increasing the
demand for more human-centric approaches in information retrieval (IR) [
CEUR
ceur-ws.org
In contrast, CIR systems emphasize the importance of context in the retrieval process. Previous
queries and responses influence the current response. The second part of Figure 1 shows a conversation
between a user and a CIR system from the FaithDial dataset [
This shift in communication style introduces several diferences between traditional IR and CIR. The conversation-like query format in CIR allows for richer, more interactive responses. Firstly, CIR systems can provide more detailed responses compared to traditional IR, which typically returns documents directly. CIR systems can refine information needs and the search space through user or system revealment. Secondly, users have more strategic options in their interactions. If unsatisfied with a response, users can ask follow-up questions to refine their search or request additional information. This necessitates the introduction of ad-hoc search methods (such as session-based and task-based search) in CIR, enabling the system to refine the search space based on the conversation, querying related sections instead of the entire database. Thirdly, context is crucial in CIR for understanding user queries, unlike traditional IR where queries are treated independently. For example, the meaning of pronouns can change depending on their order in a conversation.
Currently, CIR systems are predominantly used for simple tasks, as they are not suficiently
effective for complex and exploratory information-seeking conversations. Nonetheless, advancements
in key components, particularly in ML, are driving a trend within the IR community towards more
conversational methodologies [
Despite the progress, several critical questions remain unresolved, presenting challenges to the
development of CIR systems. Zamani et al. [
This paper primarily focuses on the first and last directions, which are both challenging and interconnected. The first direction involves modelling and producing conversational interactions, addressing the uncertainty of information needs between multiple agents through a mixed-initiative approach. Additionally, understanding long-term conversational interactions and addressing associated privacy and transparency concerns are critical topics in this direction.
The final direction pertains to the measurement and evaluation of CIR. Both academia and industry face limitations due to the absence of a robust definition of success in this field. As CIR continues to evolve, there is an urgent need for an evaluation paradigm that transcends the traditional Cranfield Paradigm, especially in frontier tasks such as personalized evaluation and transparency.
These two directions are highly interconnected. The definition of success relies on the proper modelling of conversational interactions, while precise measures support the modelling process. This paper introduces a new potential contribution to this field: introducing user simulation to CIR.
In this paper, we present a framework for the automated evaluation of CIR systems via user simulation. This paper also includes the recent advancements within this framework, while exploring the encountered opportunities and challenges. Section 3 details the two-stage user simulation prototype, which amalgamates psychological principles and ML to enhance explainability, and utilizes Large Language Models (LLMs) to sustain high performance. Section 4 focuses on the application of this prototype in assessing CIR systems. The methodology proposed aims to connect user simulation with well-established CIR conversation modelling approaches, such as efort and cost, and seeks to align the simulations closely with real user interactions through indirect assessment. Section 5 discusses the challenges and future perspectives in this field.
Conversational search, a well-established field, continues to be a popular research topic due to its
relevance for modern devices with small or no screens [
Despite significant progress, the evaluation of CIR remains relatively underdeveloped [
User satisfaction, a highly abstract and subjective measure, pertains to the overall experience and
interaction with a search system [
Despite widespread adoption, measuring user satisfaction remains an open question. Many studies
rely on real-time user participation to gather feedback, providing fresh and realistic insights but
requiring significant resources and participant incentives [ 10, 15]. Alternatively, satisfaction prediction
proxies using deep learning models ofer a computational approach, overcoming temporal and spatial
constraints but demanding high-quality computational resources and datasets [
Emerging trends in user simulation ofer promising solutions to these challenges [
Azzopardi et al. [25] define simulation as the imitation of the operation of real-world phenomena. Simulations enable detailed experimental design and control tailored to specific research questions. These high-level controls allow for experiments with user simulations to be conducted with several advantages [25].
Firstly, what-if experiments can be performed by setting up diferent scenarios [26, 27]. Secondly, user simulators ensure the repeatability of experimental results. Additionally, user simulations can achieve these benefits at a low cost [ 28].
In the IR community, user simulation methods are primarily divided into cognitive and statistical approaches [29]. Cognitive approaches were among the first used in this field. Belkin [30] described users, information resources, and IR models, characterizing users by their objectives, problems, and knowledge. Subsequent studies expanded on this foundation [31, 32, 33].
In contrast, statistical approaches focus on analyzing user behaviours and satisfaction [29, 34, 35, 36, 37]. These approaches underpin early user simulators based on statistical models [38, 39]. Although these simulators heavily relied on corpora, they faced limitations such as the diversity of user intent [29].
Agenda-based user simulations are popular due to their realistic responses and straightforward dialogue strategies [40, 41, 42]. The latest trend involves employing deep learning models, including adversarial generative approaches [43], reinforcement learning [44], and inverse reinforcement learning to abstract knowledge from data [45].
Evaluating CIR with user simulations is becoming a key trend, enabling eficient and cost-efective
evaluation at various levels of CIR [
In this section, we introduce an two-stage prototype for constructing a robust user simulation for CIR.
The main structure of the prototype is divided into two parts to control simulated users: the Action Predictor and the Response Generator, as illustrated in Fig 2.
The Action Predictor controls the simulated users’ actions during the current conversation. This concept originates from the psychology community’s notion of priming, which refers to the unconscious influence of past experiences on current performance or behaviour [50, 51]. Many studies have established models to explain this mechanism [52, 50]. For instance, Tulving et al. [53] conducted an experiment where participants viewed a list of 96 words and later completed graphemic word fragments both one hour and seven days after studying the list. The results demonstrated a significant influence of the word list on subsequent tests.
As shown in Table 1, results from Fu and Lipani [46] demonstrate the potential of predicting users’ next actions. The benefits of using lexical and textual patterns to predict users’ next actions include low cost and ease of interpretation, which are valuable for evaluation.
In the current prototype, three user actions are modelled based on the dataset available. Stopping is defined as the action where users opt to end a conversation, typically indicating the conclusion of the exchange. This action is critical for evaluating efects such as the principle of least efort [
Sessions of conversation often include
Following up on queries that build upon previous interactions, acknowledging missing contexts and references to earlier discussed topics [49]. As noted by Stede and Schlangen [55], an inquisitive user engaged in an ongoing dialogue may express interest in further related subjects as a response to the information provided. Switching topics is commonly seen in information-seeking dialogues, especially when using search systems for data acquisition [56].
As output from the Action Predictor, a general action as described above will be predicted, with the details elaborated in the Response Generator.
The Response Generator will generate realistic and diversified responses based on the conversation, user profiles and the predicted actions from the Action Predictor
LLMs such as GPTs and Llama, renowned for their sophisticated natural language processing capabilities, present unique opportunities to enhance Conversational Systems through mechanisms like pre-training, fine-tuning, and prompting [ 57]. The ability of LLMs to mimic diverse demographic characteristics ofers a novel approach to simulating user behaviour and preferences [ 58]. High-quality user simulations, which closely mirror real user behaviour distributions, can significantly advance CRS development, currently dependent on real data for training, with its inherent constraints and disadvantages.
Ramos et al. [59] ofers a valuable method for generating user profiles from the Amazon dataset, introducing a more compact style of personalization into the user simulator. Table 2 demonstrates the agreement between simulated and real users’ responses to the same items in the Amazon dataset. Since the simulated users are based on LLMs and prompts with user profiles, the results suggest the potential of generating personalized responses based on textual user profiles. Agreements (Cohen’s , Randolph’s and Krippendorf’s ) between simulated users and actual users in response.Cohen’s index considers the marginal distribution of categories, Randolph’s assumes a uniform distribution, and Krippendorf’s ofers a broader approach to assessing agreement. Fair agreements for each metric (>0.2) are marked as bold. Three simulation methods are evaluated: Real UP, where users are simulated based on their profiles; Rand UP, involving users simulated with random profiles; and Rand Sc, where scores are randomly generated based on the dataset’s historical distribution.
Setting Real UP Rand UP Rand Sc
In this simulation, responses are tailored based on user profiles, which consist of concise text that summarizes the attributes of users in a few succinct sentences. This can include motivations for task-oriented CIR systems. Furthermore, the Response Generator is tasked with handling clarification questions posed by the CIR system.
At the conclusion of this phase, the user simulator is equipped to interact with CIR systems. The subsequent section proposes a linkage as the remaining component of this framework, specifically addressing the evaluation of the CIR system using this user simulator, given the absence of a direct indicator from the user simulation on the quality of the target CIR system.
This section discusses potential methods for applying evaluation tasks based on the user simulation prototype established in Section 3.
As introduced in Section 2, the target of evaluation originates from modelling users and conversations. According to Yilmaz et al. [18], user satisfaction can be reflected by the efort exerted. Efort also forms the foundation of modelling user actions, particularly stopping behaviours.
In the IR community, several studies have been conducted to depict stopping behaviours [60, 61, 62, 63]. These studies aim to quantify the feeling of having ”enough.” For instance, users may decide to stop a conversation when they feel frustrated or satisfied.
A previous study by Fu and Lipani [46] provided a reliable method for predicting stopping behaviours. The subsequent step is to explore the relationship between each stopping point and user satisfaction.
The final part of the evaluation focuses on assessing the user simulation. To efectively perform evaluation for CIR, the user simulation must exhibit not only a diversity of reasonable responses but also a high alignment with real users, particularly in reflecting satisfaction or frustration.
Following the study by Fu et al. [28], where direct and indirect assessments in CIR show substantial agreement, the alignment between user simulation and real users can be evaluated. Real users will review the conversations between the user simulation and the system to determine if the simulated user appears satisfied.
Figure 3 illustrates the operational flow within the evaluation component of the framework. In this component, a classifier, integrated with the user simulator, is utilized to predict user satisfaction. This classifier is aligned with annotations from real users. Feedback from these real annotators is employed to train both the user simulator and the classifier. The user simulator aims to accurately mimic real user behaviour through the Actions Predictor and the Response Generator. Similarly, the classifier is trained to align its judgments with those of real users regarding the same conversation.
During the development of this framework, numerous emerging trends were observed, particularly in LLMs. These observations are presented in the following section.
This represents a significant transformation since 2021, as the term Large Language Models (LLMs) has gained popularity. This shift has introduced both opportunities and challenges for contemporary research. 5.1. CIR and RAG The integration of LLMs is not limited to the IR community; the ML community is also embracing IR techniques. A prominent development in this space is Retrieval-Augmented Generation (RAG), which enhances LLMs in domain-specific or knowledge-intensive tasks [ 64].
RAG involves multiple retrieval processes to enrich the context, going beyond traditional single retrieval methods. For instance, Self-RAG [65] refines the RAG framework by enabling LLMs to actively determine the optimal moments and content for retrieval, thus improving the eficiency and relevance of the sourced information. The key aspect here is not merely multiple retrievals, but the reliance on the judgment of LLMs, indicating that LLMs can further participate in the processing with minimal human intervention.
For the evaluation of RAG, integrating typical LLMs alone may not sufice. A realistic inquiry is how agents based on LLMs can assess the responses from RAG systems that incorporate retrieved documents. One feasible approach is using RAG to evaluate itself. Here, the crucial aspects include not only the quality of the conversation and the retrieval process but also how efectively the documents are presented .
Moreover, in the specialized domain of CIR, where the systems are relatively light, LLMs can still serve as experts. The following section provides an example.
The TREC Interactive Knowledge Assistance Track (iKAT) builds upon the foundational work of the TREC Conversational Assistance Track (CAsT) [66], with a key diference being the addition of personal context for each user in the dataset. The primary task remains similar to TREC CAsT—retrieving and ranking documents from the corpus at each turn of the given conversations.
The best performance in iKAT 2023 introduced a novel approach [67]. In this approach, the LLM generates an initial answer to the user’s query based on the context of the conversation and the user profile. This answer is derived through reasoning over the context and the user’s profile, but it is not grounded in the documents within the collection. Subsequently, the LLM generates a set of five queries to achieve this answer.
The controversial aspect of this approach is using the LLM-generated answer as the target without initial retrieval, followed by employing an IR system to achieve it. This process assumes that LLMs’ answers are suficiently accurate. Alternatively, it suggests that the documents have likely been exposed to the LLMs, raising concerns of potential data leakage.
Training LLMs from scratch is a challenging task for most research groups due to high costs and the lack of storage and computational resources. The most common practice involves fine-tuning a public base version of LLMs and accessing them via APIs. Top LLMs are trained on a substantial portion of internet text documents, making it nearly impossible to prevent data leakage once any public data is used in a study.
Furthermore, the widespread use of LLMs will inevitably introduce LLM-generated text back into the internet, posing a significant challenge that has already raised considerable concerns within the community.
This paper proposes a framework for evaluating CIR systems automatically using a user simulator prototype. The framework comprises two main components: 1. A prototype of user simulation that leverages the advancements from both psychology and ML ifelds to conduct realistic and scrutable simulations targeted at CIR systems. 2. A component that employs sophisticated conversation modelling concepts from the IR community to provide reasonable feedback aimed at predicting user satisfaction alongside the user simulation prototype.
In addition to the framework, this study also presents emerging trends observed with the promising development of LLMs. The advent of systems such as RAG introduces both opportunities and challenges. It raises concerns that the current practices within the community utilizing LLMs may lead to increased data leakages. [9] A. Anand, L. Cavedon, H. Joho, M. Sanderson, B. Stein, Conversational Search (Dagstuhl Seminar 19461), Dagstuhl Reports 9 (2020). doi:10.4230/DagRep.9.11.34. [10] J. Kiseleva, K. Williams, A. Hassan Awadallah, A. C. Crook, I. Zitouni, T. Anastasakos, Predicting user satisfaction with intelligent assistants, in: Proceedings of the 39th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’16, Association for Computing Machinery, New York, NY, USA, 2016. doi:10.1145/2911451.2911521. [11] J. Dalton, C. Xiong, J. Callan, Trec cast 2019: The conversational assistance track overview, 2020. [12] T. Saracevic, Relevance reconsidered, in: Proceedings of the second conference on conceptions of library and information science (CoLIS 2), 1996, pp. 201–218. [13] K. Papineni, S. Roukos, T. Ward, W.-J. Zhu, Bleu: A method for automatic evaluation of machine translation, in: Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, ACL ’02, Association for Computational Linguistics, USA, 2002. doi:10.3115/1073083. 1073135. [14] E. Reiter, A structured review of the validity of BLEU, Computational Linguistics 44 (2018).
doi:10.1162/coli_a_00322. [15] J. Kiseleva, K. Williams, J. Jiang, A. Hassan Awadallah, A. C. Crook, I. Zitouni, T. Anastasakos, Understanding user satisfaction with intelligent assistants, in: Proceedings of the 2016 ACM on Conference on Human Information Interaction and Retrieval, CHIIR ’16, Association for Computing Machinery, New York, NY, USA, 2016. doi:10.1145/2854946.2854961. [16] C.-W. Liu, R. Lowe, I. Serban, M. Noseworthy, L. Charlin, J. Pineau, How NOT to evaluate your dialogue system: An empirical study of unsupervised evaluation metrics for dialogue response generation, in: Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics, Austin, Texas, 2016. doi:10.18653/v1/ D16- 1230. [17] D. Kelly, Methods for evaluating interactive information retrieval systems with users, Foundations and Trends® in Information Retrieval 3 (2009). doi:10.1561/1500000012. [18] E. Yilmaz, M. Verma, N. Craswell, F. Radlinski, P. Bailey, Relevance and efort: An analysis of document utility, in: Proceedings of the 23rd ACM International Conference on Conference on Information and Knowledge Management, CIKM ’14, Association for Computing Machinery, New York, NY, USA, 2014, pp. 91–100. doi:10.1145/2661829.2661953. [19] J. Kiseleva, E. Crestan, R. Brigo, R. Dittel, Modelling and detecting changes in user satisfaction, in: Proceedings of the 23rd ACM International Conference on Conference on Information and Knowledge Management, CIKM ’14, Association for Computing Machinery, New York, NY, USA, 2014, pp. 1449–1458. doi:10.1145/2661829.2661960. [20] J. Kiseleva, J. Kamps, V. Nikulin, N. Makarov, Behavioral dynamics from the serp’s perspective: What are failed serps and how to fix them?, in: Proceedings of the 24th ACM International on Conference on Information and Knowledge Management, CIKM ’15, Association for Computing Machinery, New York, NY, USA, 2015, pp. 1561–1570. doi:10.1145/2806416.2806483. [21] K. Järvelin, S. L. Price, L. M. L. Delcambre, M. L. Nielsen, Discounted cumulated gain based evaluation of multiple-query ir sessions, in: C. Macdonald, I. Ounis, V. Plachouras, I. Ruthven, R. W. White (Eds.), Advances in Information Retrieval, Springer, Springer Berlin Heidelberg, Berlin, Heidelberg, 2008, pp. 4–15. [22] A. Al-Maskari, M. Sanderson, P. Clough, The relationship between ir efectiveness measures and user satisfaction, in: Proceedings of the 30th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’07, Association for Computing Machinery, New York, NY, USA, 2007, pp. 773–774. doi:10.1145/1277741.1277902. [23] J. Gao, M. Galley, L. Li, Neural approaches to conversational ai, in: The 41st international ACM
SIGIR conference on research & development in information retrieval, 2018, pp. 1371–1374. [24] S. Zhang, K. Balog, Evaluating conversational recommender systems via user simulation, in: Proceedings of the 26th acm sigkdd international conference on knowledge discovery & data mining, 2020, pp. 1512–1520. [25] L. Azzopardi, K. Järvelin, J. Kamps, M. D. Smucker, Report on the sigir 2010 workshop on the simulation of interaction, SIGIR Forum 44 (2011) 35–47. URL: https://doi.org/10.1145/1924475. 1924484. doi:10.1145/1924475.1924484. [26] M. I. Kellner, R. J. Madachy, D. M. Rafo, Software process simulation modeling: why? what? how?, Journal of Systems and Software 46 (1999) 91–105. [27] K. Balog, D. Maxwell, P. Thomas, S. Zhang, Report on the 1st simulation for information retrieval workshop (sim4ir 2021) at sigir 2021, SIGIR Forum 55 (2022). URL: https://doi.org/10.1145/3527546. 3527559. doi:10.1145/3527546.3527559. [28] X. Fu, E. Yilmaz, A. Lipani, Evaluating the cranfield paradigm for conversational search systems, in: Proceedings of the 2022 ACM SIGIR International Conference on Theory of Information Retrieval, 2022, pp. 275–280. [29] P. Erbacher, L. Soulier, L. Denoyer, State of the art of user simulation approaches for conversational information retrieval, arXiv preprint arXiv:2201.03435 (2022). [30] N. Belkin, Cognitive models and information transfer, Social Science Information Studies 4 (1984) 111–129. URL: https://www.sciencedirect.com/science/article/pii/014362368490070X. doi:https:// doi.org/10.1016/0143- 6236(84)90070- X, special Issue Seminar on the Psychological Aspects of Information Searching. [31] C. C. Kuhlthau, Inside the search process: Information seeking from the user’s perspective, J. Am.
Soc. Inf. Sci. 42 (1991) 361–371. [32] P. Ingwersen, K. Järvelin, The Turn: Integration of Information Seeking and Retrieval in Context, 2005. doi:10.1007/1- 4020- 3851- 8. [33] D. Ellis, A behavioral approach to information retrieval system design, J. Doc. 45 (1989) 171–212.
URL: https://doi.org/10.1108/eb026843. doi:10.1108/eb026843. [34] N. Craswell, O. Zoeter, M. Taylor, B. Ramsey, An experimental comparison of click position-bias models, in: Proceedings of the international conference on Web search and web data mining, WSDM ’08, ACM, New York, NY, USA, 2008, pp. 87–94. URL: http://doi.acm.org/10.1145/1341531. 1341545. doi:10.1145/1341531.1341545. [35] O. Chapelle, Y. Zhang, A dynamic bayesian network click model for web search ranking, in: In
WWW, 2009. [36] G. E. Dupret, B. Piwowarski, A user browsing model to predict search engine click data from past observations., in: SIGIR ’08: Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval, ACM, New York, NY, USA, 2008, pp. 331–338.
URL: http://portal.acm.org/citation.cfm?id=1390334.1390392. doi:10.1145/1390334.1390392. [37] A. Chuklin, P. Serdyukov, M. Rijke, Modeling clicks beyond the first result page, 2013. doi: 10.
1145/2505515.2507859. [38] W. Eckert, E. Levin, R. Pieraccini, User modeling for spoken dialogue system evaluation, 1997
IEEE Workshop on Automatic Speech Recognition and Understanding Proceedings (1997) 80–87. [39] K. Schefler, S. J. Young, Probabilistic simulation of human-machine dialogues, 2000 IEEE International Conference on Acoustics, Speech, and Signal Processing. Proceedings (Cat. No.00CH37100) 2 (2000) II1217–II1220 vol.2. [40] K. Komatani, S. Ueno, T. Kawahara, H. Okuno, User modeling in spoken dialogue systems to generate flexible guidance, User Modeling and User-Adapted Interaction 15 (2005) 169–183. doi:10.1007/s11257- 004- 5659- 0. [41] J. Schatzmann, B. Thomson, K. Weilhammer, H. Ye, S. Young, Agenda-based user simulation for bootstrapping a pomdp dialogue system, 2007, pp. 149–152. doi:10.3115/1614108.1614146. [42] X. Li, Z. C. Lipton, B. Dhingra, L. Li, J. Gao, Y.-N. Chen, A user simulator for task-completion dialogues., CoRR abs/1612.05688 (2016). URL: http://dblp.uni-trier.de/db/journals/corr/corr1612. html#LiLDLGC16. [43] X. Zhao, L. Xia, Z. Ding, D. Yin, J. Tang, Toward simulating environments in reinforcement learning based recommendations, ArXiv abs/1906.11462 (2019). [44] X. Chen, S. Li, H. Li, S. Jiang, Y. Qi, L. Song, Generative adversarial user model for reinforcement learning based recommendation system, in: K. Chaudhuri, R. Salakhutdinov (Eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, PMLR, 2019, pp. 1052–1061. URL: https://proceedings.mlr.press/v97/chen19f. html. [45] S. Chandramohan, M. Geist, F. Lefèvre, O. Pietquin, User simulation in dialogue systems using inverse reinforcement learning, in: INTERSPEECH, 2011. [46] X. Fu, A. Lipani, Priming and actions: An analysis in conversational search systems, SIGIR, 2023. [47] V. Adlakha, S. Dhuliawala, K. Suleman, H. de Vries, S. Reddy, TopiOCQA: Open-domain conversational question answering with topic switching, volume 10, 2022, pp. 468–483. URL: https://doi.org/10.1162/tacl_a_00471. doi:10.1162/tacl_a_00471. arXiv:https://direct.mit.edu/tacl/articlepdf/doi/10.1162/tacl_a_00471/2008126/tacl_a_00471.pdf. [48] R. Anantha, S. Vakulenko, Z. Tu, S. Longpre, S. Pulman, S. Chappidi, Open-domain question answering goes conversational via question rewriting, Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (2021). [49] C. Qu, L. Yang, C. Chen, M. Qiu, W. B. Croft, M. Iyyer, Open-Retrieval Conversational Question
Answering, in: SIGIR, 2020. [50] D. L. Schacter, R. L. Buckner, Priming and the brain, Neuron 20 (1998) 185–195. [51] J. A. Bargh, T. L. Chartrand, The mind in the middle: A practical guide to priming and automaticity research. (2014). [52] C. Janiszewski, R. S. Wyer Jr, Content and process priming: A review, Journal of consumer psychology 24 (2014) 96–118. [53] E. Tulving, D. L. Schacter, H. A. Stark, Priming efects in word-fragment completion are independent of recognition memory., Journal of experimental psychology: learning, memory, and cognition 8 (1982) 336. [54] G. K. Zipf, Human behavior and the principle of least efort: An introduction to human ecology,
Ravenio Books, 2016. [55] M. Stede, D. Schlangen, Information-seeking chat : Dialogue management by topic structure, 2004. [56] A. Spink, H. Özmutlu, S. Özmutlu, Multitasking information seeking and searching processes,
JASIST 53 (2002) 639–652. doi:10.1002/asi.10124. [57] W. Fan, Z. Zhao, J. Li, Y. Liu, X. Mei, Y. Wang, J. Tang, Q. Li, Recommender systems in the era of large language models (llms), arXiv preprint arXiv:2307.02046 (2023). [58] G. V. Aher, R. I. Arriaga, A. T. Kalai, Using large language models to simulate multiple humans and replicate human subject studies, in: International Conference on Machine Learning, PMLR, 2023, pp. 337–371. [59] J. Ramos, H. A. Rahmani, X. Wang, X. Fu, A. Lipani, Natural language user profiles for transparent and scrutable recommendations, 2024. arXiv:2402.05810. [60] D. Maxwell, L. Azzopardi, K. Järvelin, H. Keskustalo, Searching and stopping: An analysis of stopping rules and strategies, in: Proceedings of the 24th ACM International on Conference on Information and Knowledge Management, CIKM ’15, Association for Computing Machinery, New York, NY, USA, 2015, p. 313–322. URL: https://doi.org/10.1145/2806416.2806476. doi:10.1145/ 2806416.2806476. [61] W. S. Cooper, On selecting a measure of retrieval efectiveness part ii. implementation of the philosophy, Journal of the American Society for information Science 24 (1973) 413–424. [62] D. H. Kraft, T. Lee, Stopping rules and their efect on expected search length, Information
Processing & Management 15 (1979) 47–58. [63] K. R. Nickles, Judgment-based and reasoning-based stopping rules in decision-making under uncertainty, University of Minnesota, 1995. [64] Y. Gao, Y. Xiong, X. Gao, K. Jia, J. Pan, Y. Bi, Y. Dai, J. Sun, H. Wang, Retrieval-augmented generation for large language models: A survey, arXiv preprint arXiv:2312.10997 (2023). [65] A. Asai, Z. Wu, Y. Wang, A. Sil, H. Hajishirzi, Self-rag: Learning to retrieve, generate, and critique through self-reflection, arXiv preprint arXiv:2310.11511 (2023). [66] M. Aliannejadi, Z. Abbasiantaeb, S. Chatterjee, J. Dalton, L. Azzopardi, Trec ikat 2023: The interactive knowledge assistance track overview, arXiv preprint arXiv:2401.01330 (2024). [67] Z. Abbasiantaeb, C. Meng, D. Rau, A. Krasakis, H. A. Rahmani, M. Aliannejadi, Llm-based retrieval and generation pipelines for trec interactive knowledge assistance track (ikat) 2023, TREC, 2023.