Conversational Information Retrieval (CIR) is an emerging field of Information Retrieval (IR) at the intersection of interactive IR and dialogue systems for open domain information needs. In order to optimize these interactions and enhance the user experience, it is necessary to improve IR models by taking into account sequential heterogeneous user-system interactions. Reinforcement learning has emerged as a paradigm particularly suited to optimize sequential decision making in many domains and has recently appeared in IR. However, training these systems by reinforcement learning on users is not feasible. One solution is to train IR systems on user simulations that model the behavior of real users. Our contribution is twofold: 1) reviewing the literature on user modeling and user simulation for information access, and 2) discussing the diferent research perspectives for user simulations in the context of CIR.
1. Introduction
of reformulation of requests, clicks on documents,
conversations in natural language) and 2) anticipate and
Conversational information retrieval (CIR) is a well- guide the user in his/her sequence of search actions
usestablished research field in the Information Retrieval ing, e.g., query clarification or document ranking [ 5, 6].
(IR) community [
The objective of a CIR system is to help the optimize an IR policy has become a hot topic [13, 15]. For user in his/her interactions with a search engine IR, policies would be learned to optimize in the long term through a conversational tool. This whole search en- the sequences of user interactions with the CIR system gine/conversational system will aim to: 1) respond to according to diferent criteria : reducing/increasing the a complex information need expressed in natural lan- number of interactions with the user, increasing the diguage and/or through various feedbacks (e.g., sequences versity of retrieved information, predicting future needs, guiding users actions, etc. Ideally, CIR system policies should train with real users. In practice, this is impossible because reinforcement learning algorithms are currently very ineficient: it would take millions of interactions with users to see the emergence of a correct policy.
Training directly on data limits the exploration to scenarios collected from real users, so CIR systems can only Causality in Search and Recommendation (CSR) and Simulation of Information Retrieval Evaluation (Sim4IR) workshops at SIGIR, 2021 Editors of the proceedings (editors): Krisztian Balog, Xianjie Chen, Xu Chen, David Maxwell, Paul Thomas, Shuo Zhang, Yi Zhang, and Yongfeng Zhang " pierre.erbacher@lip6.fr (P. Erbacher); laure.soulier@lip6.fr (L. Soulier); denoyer@fb.com (L. Denoyer) © 2021 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
CEUR Workshop Proceedings (CEUR-WS.org) visit a very small part of the state space. In addition, even Historically, US have been extensively studied for conif a lot of search trajectories (sequence of interactions versational systems [26, 27, 28, 29, 30]. One of the first between user and system) are collected, only a few would attempts is to use statistical user models as a simulabe optimal [16]. tor [26]. This simulates the users from the appearance
One of the solutions is to train CIR systems with user probabilities of a sentence according to the previous sensimulations (US). US generate synthetic data close to data tence. This simulation is limited, because it only takes generated by real users. This makes it possible to : 1) into account the previous response of the system, the estimate the most probable actions of users, particularly probabilities sufer from the lack of data and moreover it in new scenarios and 2) best optimize CIR systems poli- is not guided by a dialogue objective. To add consistency cies by exploring trajectories not contained in collected to users’ responses, Schefler et al. [ 30] introduces into scenarios. In this article we are interested in the problem its simulator a dialogue objective, drawn from a list of of user modeling and simulation in diferent contexts of possible objectives. The model follows an imposed diaaccess to information (e.g., IR, recommendation, conver- logue objective: the probabilities are conditioned by the sational systems). objectives of the real users in the corpus. 1. We review the diferent approaches for modeling Komatani et al.[27] brings diversity by conditioning and simulating user for information access and the probability of utterance on the user’s ability to use a goal oriented conversational systems (Section 2). search system as well as its degree of urgency. From annoModeling and simulation are often confused in tated data, a decision tree is used to classify the sentences the literature. For a source system, in our case the of the corpus according to the urgency and the compeuser whose behavior is unknown, the associated tence of the users. For the simulation, a degree of urgency model is a set of instructions, rules, markov chain is drawn and imposed on the model for the generation or equations constructed from the observed data of sentences. One way to implement rule based US is to coming from this user [17]. Simulation is an agent use agenda, as in [31]. In this work, the authors build that generates behaviors by following the rules a conversational US using constraints and rules which of a model. Together, the goal is to emulate user make it possible to explicitly encode the user’s state as behavior [17]. well as his objective. In addition to generating sentences 2. We discuss state-of-the-art SU evaluation, the dif- according to a certain user model, the simulator is asifculty being to measure whether the simulated signed a dialogue objective as well as a dialogue strategy behavior is realistic (Section 3). (Agenda) which conditions the sentences generated by 3. Finally, we discuss the research perspectives to the model.
build user simulations for CIR (Section 4). Agenda-based approaches are widely used to train conversational recommendation systems because they allow to build dialogue strategies for close domain using simple 2. Model and simulate user rules [32, 33]. Zhang et al. [34] show that US Agendas behavior can be particularly realistic: when recommending movies, 36% of human reviewers mistook an US for a real user The IR community produced lot of work in studying and and 23% of reviewers were undecided. modelling users behaviors either based on cognitive or Li et al. [33] propose a more realistic hybrid model that statistical approaches. First user modeling approaches relies on the agenda to maintain user dialogue strategy in IR were based on cognitive approaches. For instance, but which takes advantage of the power of language modBelkin [18] produces a description of the users character- els seq2seq to improve the natural language generation ized by an objective or a problem as well as a knowledge and language understanding. Asri et al. [35] and Kreyssig of the world. He also describes the conditions necessary et al. [16] propose a neural model that breaks free from for an IR system to be efective with ambiguous queries. the agenda by using seq2seq language models to generate Many derivatives were then proposed (e.g., ISP [19], IS&R and interpret natural language. On the other hand, they [20], Ellis [21]). The second line of work mainly remains keep the generation of objectives (constraints / requests) on statistical models describing users’ click behavior and at the semantic level. To simulate the dynamic behavsatisfaction during search sessions [22, 23, 24, 25]. Be- ior of the user, the constraints are drawn independently yond being used for building predictive models (from the according to a probability observed in the dataset. IR system point of view), these user models serve also Some work relies on deep learning to simulate users as basis of user simulations. In what follows, we will from data [36, 37, 38]. For example, Zhao et al. [38] learn focus on the literature review of user simulation models. a user simulation which provides information on the Please note that since CIR is a young research domain, relevance of the recommendations made by the system. most of the cited work below belongs to orthogonal and They use an adversarial generative approach to converge related fields, namely recommendations and dialog. to a US. The discriminator must diferentiate between simulated and real feedback based on user search his- plexity [24]). However, in the context of dialogue systory. The generator learns to create feedbacks based on tems, [46] demonstrates that automatic metrics that asa history of interactions. By relying on reinforcement sess word overlap with reference sentences like BLUE, learning approaches, Chen et al. [37] propose to jointly RED and METEOR have very little correlation with judgmodel the user click policy and the reward function us- ment human, in particular because there is a great diing a generative approach. Indeed, the user policy is versity of valid answers for each round of conversation optimized to generate the actions which maximize the [46, 34]. Human evaluation is also used to evaluate the reward signal. In parallel, the reward function has the quality of the responses of a user simulation. For exrole of the discriminator, it is optimized to minimize the ample, Zhang et al. [34] observe the number of times diference in reward between the action taken by the user human evaluators cannot diferentiate between real user policy and the action taken by a real user observed in responses and US-generated dialogues to compare the the database. A data driven approach is based on reverse quality between US. reinforcement learning, the objective of which is to learn Indirect evaluation measures the impact of a US on the the reward function from the behavioral observations of performance of the retrieval system [29]. Shi et al. [47] another (often human) agent [39]. This approach con- assess the realism of its user model thanks to direct and trasts with previous models which relied on explicitly indirect human annotation. For direct evaluation, they defined reward functions. In this context, Chandramohan directly judge the quality of sentences from diferent US et al. [36] propose to use inverse reinforcement learn- through diferent criteria: the quality of the language ing to find the reward function describing user policies (such as grammar), the consistency of the sentences beobserved in data. An experiment is carried out using tween the diferent turns as well as the diversity of the conversational data in the field of tourist information. sentences. The evaluators also give an overall rating
Some US relies on active learning. Because learning of the simulations. The indirect evaluation of the US is a user model requiring a large amount of data. For this, measured through the performance of the recommender Zhang et al. [40] proposes to include sparsely human systems that have interacted with the US. Human reviewfeedback in the learning loop. In this model, the user ers analyze the quality of the recommendation. They rate model is learned jointly with the recommendation sys- their satisfaction, the naturalness of the conversations tem. The goal is to optimize training to limit human and the efectiveness of the recommendations. intervention. For this, the user model is allocated a lim- This section summarizes how human annotations and ited budget. At each turn, the model can choose how to automatic metrics are used in the literature to assess US allocate her credits to respond to the recommendation quality via direct and indirect evaluations. [34] shows system: 0 credit to respond using the learned policy, 1 that assessing IR systems using US is possible. There are credit to access a human response, 2 credit to collect a still lot of work to be done to have reliable and realist human discussion - human. The user model learns from user simulation in order to benchmark IR systems. the new data collected. At the start of each conversation, a goal generator pulls in unsuccessful or unexplored goals. Despite a parsimonious use of human interven- 4. Research perspectives tions, including humans in the learning loop remains time-consuming.
Towards multi-modal user simulation models 3. Evaluation of User Simulation for open domains (CIR). To generate more realistic search sessions, US should be able to express complex As explained earlier, one of the objectives of US is to eval- queries on many diferent domains and simulate uate [26] information retrieval systems. It is therefore document selection (click). Currently, conversational needed to ensure the quality of the US used for the eval- US do not embed user click models. However, clicks uation. According to [41], a good metric, to evaluate US, are present in many IR scenarios for example when should be able to measure US quality independently of the the system is returning keywords for disambiguating task and should be correlated with real user evaluations. the query or selecting document. Generating complete Following [41], we distinguish two types of evaluation. user behaviors in the form of sequences of clicks and
Direct evaluation directly measures US through met- requests or dialogue seems essential to have realistic rics. For example, an evaluation approach consists in evaluation frameworks for CIR systems. To overcome measuring whether the policy has achieved its objec- the lack of data, one idea would be to incorporate tive, the quality of sentences generated in a dialogue existing click models [24] into US [31]. Another idea is system thanks to BLEU / RED / METEOR measurements to jointly learn the language model and the user click [42, 43, 44, 45], likely click behavior measured by per- model. Simulating both conversations and clicks seems • To open domain conversation • To various level of query complexity • To various user behavior essential for learning CIR systems that return a list of documents. Also, CIR imposes to be able to evolve on a variety of domains (open domains). The vast majority of US are built for conversational recommendation systems (CRS) for a specific domain, for example,(e.g., restaurant suggestion). CIR systems need to ofer services covering a growing number of areas. A first step towards this open domain would be to benefit from the research in multi-domain CRS [48], with the publication of several multi-domain datasets [48, 49].
ulation of unseen US having diferent persona and behavior.
Learning users behaviors in CIR context directly from collected data might be limited because: 1) Collecting conversational search session data is computationally expensive. 2) Collected data is entangled with used the Towards diversity-driven user simulation IR system. One idea is to build US with diferent level One of the dificulties brought by neural models is to of abstractions. From the intention level to the natural generate diversity in behavior. Indeed, such models learn language generation. In order to facilitate future works, to satisfy the average behaviors observed in the data. the IR community should have a common platform for However, there is a strong variability in user behavior building and testing US for sequential IR systems. In [50]. Some are more persistent or curious in their this direction, some environments simulating users have research, while others give up quickly. In addition, the already been developed for learning recommendation context and world knowledge is diferent between users. system using sequential learning such as [53, 54]. Learning a single user model on data does not reflect this diversity of users. Recently proposed US, model one 5. Conclusion behavior from data generated by many real users. One of the ideas would be to generate realistic user policies This article focuses on CIR and above all on setting up but with diferent styles. One method is to regroup a training and evaluation framework for user-centered and model by hand the diferent users according to a approaches. Although human assessment is the most certain number of criteria (curiosity, perseverance, world suitable, it is however too time-consuming for models knowledge, expertise, ...) such as [27]. A second method based on deep learning or reinforcement learning. The is to extract automatically these information from a objective of this paper is to synthesize the state of the art corpus of conversations. Some works propose to use US of user modeling and simulation approaches in various with attributes and characteristics (persona) [51, 52] to information access tasks to open up prospects for the obtain both diversity and consistency in requests and implementation of CIR systems. dialogues. Li et al. [51] used user embedding encoding speaker specific information (persona) such as age, dialect, context, gender and other personal information 6. Acknowledgments that influences generated responses. Balog et al. [ 52] proposed a more convenient way to encode information We would like to thank the ANR JCJC project SESAMS about users persona using knowledge graph. Ideally, (Projet-ANR-18-CE23-0001) for supporting Pierre ErCIR should use knowledge they have about the user in bacher and Laure Soulier from Sorbonne Univeristé in order to optimize this user’s search session. In a cold this work. start scenario, one possibility for the CIR is to probe users to extract a maximum number of information with References a limited probing budget. We believe that using diversity driven US is essential to learn such probing policies.
Towards the establishment of benchmarks for CIR systems. Currently, human annotators are the most reliable benchmark to assess CIR systems performances.
But, human evaluation is time-consuming to test the robustness of models for various information needs and behaviors. One of the solutions is to establish US to measure the performance of the various proposed CIR models.
Note that [34] proposed this evaluation framework for CR systems. In addition, CIR systems should also be evaluated to on their generalization capacity: [5] L. Azzopardi, M. Dubiel, M. Halvey, J. Dalton, Con- [22] N. Craswell, O. Zoeter, M. Taylor, B. Ramsey, An exceptualizing agent-human interactions during the perimental comparison of click position-bias modconversational search process, 2018. els, in: WSDM, 2008, pp. 87–94. [6] F. Radlinski, N. Craswell, A Theoretical Frame- [23] O. Chapelle, Y. Zhang, A Dynamic Bayesian Network for Conversational Search, in: SIGIR, 2017, p. work click model for web search ranking, in: 117–126. WWW, ACM Press, 2009, pp. 1–10. [7] A. Sordoni, Y. Bengio, H. Vahabi, C. Lioma, J. G. Si- [24] G. Dupret, B. Piwowarski, A user browsing model monsen, J. Y. Nie, A hierarchical recurrent encoder- to predict search engine click data from past obserdecoder for generative context-aware query sug- vations, in: ACM SIGIR, 2008, pp. 331–338. gestion, in: CIKM, 2015, pp. 553–562. [25] A. Chuklin, P. Serdyukov, M. De Rijke, Modeling [8] L. Zou, L. Xia, Y. Gu, X. Zhao, W. Liu, J. X. Huang, clicks beyond the first result page, in: International D. Yin, Neural interactive collaborative filtering, in: Conference on Information and Knowledge ManACM SIGIR, SIGIR ’20, ACM, 2020, p. 749–758. agement, Proceedings, ACM, 2013, pp. 1217–1220. [9] M. Mitsui, C. Shah, Bridging gaps: Predicting user [26] W. Eckert, E. Levin, R. Pieraccini, User modeling and task characteristics from partial user informa- for spoken dialogue system evaluation, in: IEEE, tion, in: ACM SIGIR, SIGIR’19, ACM, 2019, p. IEEE, 1997, pp. 80–87.
415–424. [27] K. Komatani, S. Ueno, T. Kawahara, H. G. Okuno, [10] Y. Sun, Y. Zhang, Conversational Recommender User modeling in spoken dialogue systems to gener
System, SIGIR (2018) 235–244. ate flexible guidance, in: User Modelling and User[11] Z. Tang, G. H. Yang, Dynamic search - optimizing Adapted Interaction, volume 15, Springer, 2005, pp. the game of information seeking, CoRR (2019). 169–183.
arXiv:1909.12425. [28] O. Pietquin, A Framework for Unsupervised Learn[12] H. Yang, M. Sloan, J. Wang, Dynamic information ing of Dialogue Strategies, Ph.D. thesis, Faculté retrieval modeling, in: ACM SIGIR, SIGIR ’14, 2014, Polytechnique de Mons, TCTS Lab (Belgique), p. 1290. 2004. URL: http://theses.ulb.ac.be:8002/ETD-db/ [13] W. Zhang, X. Zhao, L. Zhao, D. Yin, G. H. Yang, collection/available/FPMSetd-11082004-163102/.
A. Beutel, Deep Reinforcement Learning for Infor- [29] J. Schatzmann, K. Weilhammer, M. Stuttle, S. Young, mation Retrieval: Fundamentals and Advances, in: A Survey of Statistical User Simulation Techniques SIGIR, 2020, pp. 2468–2471. for Reinforcement-Learning of Dialogue Manage[14] R. S. Sutton, A. G. Barto, Reinforcement Learning: ment Strategies, The Knowledge Engineering Re
An Introduction, MIT Press, 1998. view (2006) 0–1. [15] A. Montazeralghaem, H. Zamani, J. Allan, A Re- [30] K. Schefler, S. Young, Probabilistic simulation of inforcement Learning Framework for Relevance human-machine dialogues, in: ICASSP, IEEE, volFeedback, in: SIGIR, 2020, p. 59–68. ume 2, IEEE, 2000, pp. 1217–1220. [16] F. L. Kreyssig, I. Nigo Casanueva, P. Budzianowski, [31] J. Schatzmann, B. Thomson, K. Weilhammer, H. Ye, M. G. Gaši´c, Neural User Simulation for Corpus- S. Young, Agenda-Based User Simulation for Bootbased Policy Optimisation for Spoken Dialogue Sys- strapping a POMDP Dialogue System, Technical tems, Technical Report, 2018. Report, 2007. [17] B. P. Zeigler, Theory of Modelling and Simulation [32] Y. Hou, M. Fang, W. Che, T. Liu, A Corpus-free | Guide books, Krieger Publishing Co., Inc, 1984. State2Seq User Simulator for Task-oriented DiaURL: https://dl.acm.org/doi/10.5555/537562. logue, Technical Report, 2019. [18] N. J. Belkin, Cognitive models and informa- [33] X. Li, Z. C. Lipton, B. Dhingra, L. Li, J. Gao, Y.tion transfer, Social Science Information Studies N. Chen, A User Simulator for Task-Completion 4 (1984) 111–129. doi:10.1016/0143-6236(84) Dialogues *, Technical Report, 2016. URL: https: 90070-X. //github.com/MiuLab/UserSimulator. [19] C. C. Kuhlthau, Inside the search process: Informa- [34] S. Zhang, K. Balog, Evaluating Conversational Rection seeking from the user’s perspective, JASIST 42 ommender Systems via User Simulation, in: ACM (1991) 361–371. SIGKDD, Association for Computing Machinery, [20] P. Ingwersen, K. Järvelin, The Turn: Integration of 2020, pp. 1512–1520.
Information Seeking and Retrieval in Context (The [35] L. E. Asri, J. He, K. Suleman, A sequence-toInformation Retrieval Series), Springer-Verlag New sequence model for user simulation in spoken diaYork, Inc., 2005. logue systems, in: ISCA, 2016. [21] D. G. Ellis, A Behavioral Approach to Information [36] S. Chandramohan, M. Geist, F. Lefèvre, O. Pietquin, Retrieval System Design, Journal of Documentation User Simulation in Dialogue Systems using Inverse 45 (1989) 171–212. Reinforcement Learning, Technical Report, 2011. [37] X. Chen, S. Li, H. Li, S. Jiang, Y. Qi, L. Song, Gen- [52] K. Balog, T. Kenter, Personal knowledge graphs: A erative adversarial user model for reinforcement research agenda, in: ICTIR, 2019, p. 217–220. learning based recommendation system, in: ICML, [53] E. Ie, C. wei Hsu, M. Mladenov, V. Jain, S. Narvekar, volume 97 of PMLR, PMLR, 2019, pp. 1052–1061. J. Wang, R. Wu, C. Boutilier, Recsim: A config[38] X. Zhao, L. Xia, L. Zou, D. Yin, J. Tang, Simulating urable simulation platform for recommender sysUser Feedback for Reinforcement Learning Based tems (2019).
Recommendations, Technical Report, 2019. [54] D. Rohde, S. Bonner, T. Dunlop, F. Vasile, A. Karat[39] S. Russell, Learning agents for uncertain environ- zoglou, Recogym: A reinforcement learning ments (extended abstract), in: COLT’ 98, ACM environment for the problem of product recomPress, 1998, pp. 101–103. mendation in online advertising, arXiv preprint [40] Z. Zhang, X. Li, J. Gao, E. Chen, Budgeted policy arXiv:1808.00720 (2018).
learning for task-oriented dialogue systems, in:
ACL, 2019, pp. 3742–3751. [41] O. Pietquin, H. Hastie, A survey on metrics for the evaluation of user simulations, The Knowledge
Engineering Review (2008) 0–1. [42] K. Papineni, S. Roukos, T. Ward, W.-J. Zhu, {B}leu: a Method for Automatic Evaluation of Machine
Translation, in: ACL, 2002, pp. 311–318. [43] C.-Y. Lin, ROUGE: A package for automatic evaluation of summaries, in: Text Summarization Branches Out, Association for Computational Linguistics, 2004, pp. 74–81. [44] S. Banerjee, A. Lavie, METEOR: An automatic metric for MT evaluation with improved correlation with human judgments, in: ACL, Ann Arbor, Michigan, 2005, pp. 65–72. [45] A. Sordoni, M. Galley, M. Auli, C. Brockett, Y. Ji,
M. Mitchell, J.-Y. Nie, J. Gao, B. Dolan, A Neural Network Approach to Context-Sensitive Generation of Conversational Responses, in: NAACL, 2015, pp. 196–205. [46] C.-W. Liu, R. Lowe, I. Serban, M. Noseworthy,
L. Charlin, J. Pineau, How {NOT} To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response
Generation, in: EMNLP, 2016, pp. 2122–2132. [47] W. Shi, K. Qian, X. Wang, Z. Yu, How to Build User
Simulators to Train {RL}-based Dialog Systems, in:
EMNLP-IJCNLP, 2019, pp. 1990–2000. [48] A. Rastogi, X. Zang, S. Sunkara, R. Gupta, P. Khaitan, Towards scalable multi-domain conversational agents: The schema-guided dialogue dataset, AAAI 34 (2020) 8689–8696. [49] P. Budzianowski, T.-H. Wen, B.-H. Tseng,
I. Casanueva, S. Ultes, O. Ramadan, M. Gašić, MultiWOZ - A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modelling, in:
EMNLP, 2018, pp. 5016–5026. [50] R. W. White, S. M. Drucker, Investigating behavioral variability in web search, in: WWW 2007, 2007, pp.
21–30. [51] J. Li, M. Galley, C. Brockett, G. Spithourakis, J. Gao,
B. Dolan, A Persona-Based Neural Conversation Model, in: ACL, 2016, pp. 994–1003.