In Conversational Recommendation Systems (CRS), a user can provide natural language feedback on suggested items, which the recommender uses to produce improved suggestions. Therefore, the success of a user's conversation with the CRS is determined by how well the system is able to interpret the user's feedback and the quality of the recommendations. Knowing whether a conversation is likely to be successful may allow the CRS to adjust accordingly - for instance, changing its retrieval strategy, or asking a clarifying question. Existing work on Query Performance Prediction (QPP) has examined a number of predictors that indicate the efectiveness of a search engine's ranking in response to a query. Inspired by existing work in QPP, we propose a framework for Conversational Performance Prediction (CPP) that aims to predict conversation failures by considering the recommendation ranking at diferent turns of a conversation, either one turn at a time, or by considering multiple consecutive turns. In this regard, we adapt post-retrieval predictors to address the multi-turn nature of the CRS task. We conduct our analysis on Shoes and FashionIQ Shirts & Dresses datasets. In particular, as a ground truth, we measure conversation dificulty by the efectiveness of the ranking at a given turn of the conversation. Overall, we find some promise in score-based retrieval predictors for CPP, obtaining medium strength correlations with conversation dificulty - for instance, observing a Spearman's of 0.423 on the Shoes dataset, which is comparable to correlations observed for standard QPP predictors on adhoc search tasks.
Traditionally, Recommender Systems (RS) help users to ifnd items of interest on the basis of user feedback in terms of ratings, clicks or reviews. In contrast, Conversational Recommendation Systems (CRS), such as personal digital assistants [1], have facilitated more complex recommendation settings by suggesting items in response to voice or (natural language) chat interactions. In particular, a CRS allows a multi-turn dialogue with users and aim to assist them with achieving a number of taskoriented goals [2]. Indeed, at each turn users can provide their feedback or critique [3], which helps the system to improve recommendations [4].
One important aspect of natural language-based CRS is that they allow users to explore the range of available options and elicit their preferences. For example, Bursztyn et al. [7] created a multi-modal system, where users navigate in a setting of limited options, such as finding a restaurant near their location. In this setting, users start exploring an initial set of restaurants and have the opportunity to see their details by clicking through the options, while they are asked about the reasons for any negative feedback they provide. Another example of user exploration is the MusicBot [8, 9], a music chatbot that ifrst collects users’ preferences and then makes suggestions based on diferent techniques of critiquing the song recommendations. In our work, we are focused upon conversational fashion image recommendation [5, 6, 10], an example of which is shown in Figure 1. In this task, the user has a target item in mind, and provides textual feedback (critiques) to direct the system towards retrieving images of fashion products that are more similar to to their perceived target item.
However, not all conversations may lead to a satisfying outcome for the user. This can be easily quantified in ofline evaluation scenarios, where the CRS is evaluated explore a list of options for a number of turns, the sys- 2.1. Query Performance Prediction tem was unable to find a relevant recommendation by turn 7, which might mean that the algorithm was still Traditionally, QPP is used to predict the efectiveness of exploring the space. Also, in Wu et al. [10], the target a search results page performed in response to a query item was found by the system at rank 1 in only 42% of in the absence of human relevance judgments [15]. It conversations after (a maximum of) 10 turns. Therefore, has applications to selective retrieval approaches [17, 18] exploration might result in an increased number of turns, and query features for learning-to-rank [19], to name which on one hand might mean more engaged users [8], but a few. Query performance predictors are generally but at the same time suggests that often the conversa- grouped into pre-retrieval, and post-retrieval, which we tions might fail (i.e., target item not found). In this regard, discuss further below. we are interested in identifying indicators that can detect how this happens – for example, a conversation could 2.1.1. Pre-retrieval Query Performance Predictors fail because the system is unable to find the target item Pre-retrieval predictors are used to estimate the perforor because the target item is not available. mance of queries before the retrieval stage, and therefore,
In what follows, inspired by existing work on Query are independent of the search performed and the ranked Performance Prediction (QPP) (e.g., [12, 13, 14, 15]), we list of results [14]. This means that pre-retrieval predicaim to predict conversational failures by identifying spe- tors base their predictions on properties of query-terms cific indicators that are correlated with failure. In par- or corpus-based statistics [12, 13, 14, 20, 21, 22]. Examticular, we aim to determine the quality of multi-turn ples of pre-retrieval predictors that describe the statistical critiquing-based CRS recommendation by proposing pre- properties of the query terms or the corpus include the dictors that consider the multi-turn aspect of conver- query length (number of non-stop words in the query), sational recommendation. The proposed predictors ad- the standard deviation of the inverse document frequency dress characteristics of the retrieved scores of the top- of the query terms, the simplified query clarity score recommended items and can predict poor performance (SCS), which measures the occurrence of a query term across a shorter or longer number of turns in the con- in the query relative to its occurrence in the collection, versation, which we call prediction horizons. In summary, and AvICTF, which considers the overall informativethis work makes the following contributions: (i) We pro- ness of the query terms using the collection model [23]. pose a framework for Conversational Performance Pre- Another class of pre-retrieval predictors refers to linguisdiction (CPP), which extends the existing work on QPP to tic features of the queries, such as syntactic complexity a conversational recommendation setting; (ii) We show (distance between syntactically linked words) and word how to adapt QPP evaluation methodology to a multi- polysemy (number of semantic classes a word belongs turn conversational setting which allows to evaluate CPP to) [22]. Overall, with limited information available bepredictors for both short- and long-term prediction hori- fore retrieval commences, pre-retrieval predictors are zons; (iii) We evaluate some of our proposed predictors widely considered less accurate for performance predicon the Shoes [5, 6] dataset and the Fashion IQ Dresses tion than post-retrieval predictors [14]. and Shirts categories [16], using a state-of-the-art user simulator [6]. The rest of the paper is structured as fol- 2.1.2. Post-retrieval Query Performance lows: Section 2 presents the existing research on QPP, Predictors including pre- and post-retrieval predictors, as well as their probabilistic interpretation; Section 4 outlines our On the other hand, post-retrieval predictors are applied on new proposed framework and predictors; Section 5 de- the list of the top-ranked retrieved documents, and therescribes our experimental setup; Sections 6 & 7 present fore use the relevance scores or the (textual) contents of our results and provide concluding remarks. the returned items. A first group of post-retrieval predictors examines the diference of the result list from the corpus, or the focus of the result list. For example, Clarity 2. Related Work [12] measures the focus of the resulting ranking with respect to the corpus using the KL divergence between In order to predict why a conversation with a CRS might their respective language models, while the Weighted fail, we need to identify indicators that show when the Information Gain (WIG) corresponds to the diference beuser is unable to find the target item during the interac- tween the average retrieval score of the result list and of tion. In this regard, we are inspired by existing work from that of the corpus [24]. A second group includes the disQuery Performance Prediction (QPP), which we discuss in tribution of the retrieval scores of the top-ranked items. Section 2.1; Later, in Section 2.2, we discuss applications Predictors in this group include Normalized Query Comof QPP in conversational contexts. mitment (NQC) [25] (the standard deviation of the retrieval scores in the result list). The standard deviation is considered to be negatively correlated with the amount However, while QPP has been widely explored for of query drift (the non-related information in the result (single turn) queries in search settings, the area of conlist) [26]. Also, this group includes the modeling of re- versational search or recommendation has seen much trieval scores; the top-ranked items could be modeled as a less work. For example, one recent work examines the certain mixture of distributions corresponding to relevant predicted efectiveness of the top-retrieved documents and non-relevant items [27]. Another related predictor for deciding to generate clarifying questions, and specifiis autocorrelation [28], which assumes that documents cally some extracted features, such as noun phrases or whose vector space embeddings are closely related re- named entities [36]. Indeed, clarifications are useful for ceive similar scores, and therefore, closely related scores both the user and the system [37, 38, 39]. Also, Roitwould indicate similar performance. man et al. [40] examined a constrained retrieval setting,
A third group of post-retrieval predictors refers to namely the interaction with a conversational assistant, the relation of the top-ranked retrieval scores with a where the assistant needs to decide whether the provided particular reference list. Recently, a more generalised ap- answer could be accepted. The authors built a classifier proach for estimating the efectiveness of a ranking was that determines the answer quality by adapting some proposed, based on the assumption that high association existing QPPs to the answer level (using the score of the with pseudo-efective reference lists and low association top item, which is provided as the answer). with pseudo-inefective lists improves efectiveness [ 29]. However, QPP for conversational recommendation has One example refers to the utility estimation framework not been addressed. In particular, we are interested in (UEF) [30], which estimates the utility of a given ranking creating a prediction framework for identifying poorly with respect to how much it represents an underlying performing or failed conversations in a recommendation information need [31]. The utility is estimated by the setting. We postulate that these predictors can be useful expected similarity between a given document ranking in several use cases, for instance knowing when to ask and those induced by estimates of relevance language for clarifications, or when the users target item cannot models (these rankings are assumed to be representative be found. Towards achieving this goal, we explore scoreof the information need). [32]. A similar predictor to based predictors, adapting to the multiple turn nature the UEF approach is query feedback (QF) [24], which of the task. In the next section, we define the CRS task; measures the overlap of top items between the result Later in Section 4, we define our CPP framework. list and a reference list retrieved from the corpus using a language model induced from the result list.
Autocorrelation [28] could also fall under this category, if 3. Conversational Image we compare the result list of the original retrieval scores Recommendation with a reference list that contains either a perturbed version of the scores difused in space or a list with Figure 1 describes the context of dialog-based image recthe averaged values from multiple retrievals for the ommendation in a CRS. At each interaction turn, the same query. Lastly, an inter-list similarity predictor is user provides a critique of the current recommendation a measure of rank-biased overlap (RBO), which measures (candidate item) back to the system, aimed at directthe expected average overlap between two rankings [33] ing it towards the desired target item. More formally, and can be applied to the QPP task. at a given interaction turn , the user provides textual
Finally, we note some recent QPP work (e.g. [29, 34]) feedback on the current top-ranked candidate item has focused upon probabilistic frameworks for QPP, ,1. Based on this feedback, the conversational recwhich can integrate both pre-retrieval and post-retrieval ommendation system () provides a new ranking , i.e.: predictors. However, many of the underlying intuitions (,1, ) → , where is a ranking of items with encapsulated by these frameworks are already addressed corresponding descending retrieval scores 1 . . . , i.e.: in the previously described predictors. = [⟨+1,11⟩, . . . ⟨+1,, ⟩].
However, it is challenging to train and evaluate a natu2.2. Query Performance Prediction in ral language-based CRS. For training, reinforcement learning (RL) is widely used, as it allows optimising the recom
Conversational Search mendation model based on the long-term rewards [41], Natural language-based conversational systems allow i.e. based not just on retrieving the correct item in any users to express complex feedback through a dialogue, current iteration, but also retrieving it in later iterations. thus resulting in more natural interactions [35]. To be However, such a model needs to be trained while interactable to predict the likelihood of success of a conversation, ing with an environment, and obtaining many samples we need to consider the salient aspect of the conversa- is hard by relying on real users [41, 42]. For evaluation, tional setting, such as the users’ feedback and the iterative ideally human users are needed to judge the system’s turn-based nature of the interaction process. eficiency and user satisfaction [ 43]. Instead, user simulators are deployed as surrogates for human users, trained on textual queries and textual documents. In contrast, on relative caption data - a form of human-annotated di- in our fashion-based CRS, the “units of retrieval" are alogues on pairs of images. Recommendation models images, with embedded representations - this precludes trained and evaluated using user simulators have been the use of textual content-based predictors. Furthermore, found to be correlated with human satisfaction [6]. our “query units" are critiques, which are based on the
Specifically, for the purposes of training a user sim- retrieval of the previous turn. Therefore, it can be seen ulator with human-annotated dialogues, Guo et al. [6] that there is no clear distinction between pre-retrieval proposed the relative captioning task. In this task, hu- and post-retrieval predictors, since what is considered man annotators recruited through crowdsourcing are post-retrieval of one turn could be seen as a pre-retrieval placed in a context of online shopping, where the CRS predictor of the following turn. For this reason, we acts as the shopping assistant and they play the role of the propose a new framework for performance prediction customer. During the process, annotators are presented in a conversational setting, in particular conversational with candidate recommended images of items and they fashion retrieval, which we describe in Section 4.1 below. are asked to provide single instance critiques. In each Later in Section 4.2, we describe the initial score-based interaction round, they are shown a given candidate item predictors we can adapted to this framework. and based on a given target item, they provide a critique on the current candidate item. These diferences between 4.1. CPP Framework the candidate and the target image are described with natural language phrases and form the relative captions. We present a framework for Conversational Performance Hence, a relative captioning dataset contains tuples of Prediction (CPP) applied to the domain of fashion recomthe following form: ⟨, , ,⟩ where is a represen- mendation for image retrieval [6, 16]. In this regard, we tation of the target item (for instance an image), is define recommendation success as the identification of the current candidate item being presented to the user the target image item by the system before a maximum and , is the critique by the user on the candidate, number of turns is reached, which corresponds to a user intended to direct the system more towards the target. being satisfied with the conversation. More formally, the Relative captioning data can be used to train a user sim- CPP task can be described as a function of the form ulator, which is then deployed for training or evaluating a CRS [6, 10, 11, 16, 44, 45]. (, ) → R
Using a user simulator for evaluation, the overall success of a CRS system can be reliably measured, in a ofline where is a sequence each containing feedback criCranfield-like setting, by using ranking evaluation mea- tiques over 1 or more turns, and is a sequence of results sures, such as NDCG, upon the ranked list of recommen- lists consisting of retrieval scores, over 1 or more turns. dations produced at each turn. From such an evaluation, This framework can be instantiated for single-turns, it can be seen that even after 10 turns, some CRS models or multiple turns. For instance, in a single-turn setting, may not be able to identify the target item for some con- we can instance CPP task at a given turn , i.e.: versations. For this reason, making a prediction as to the single([], []). likelihood of a user being satisfied with a conversation may have utility to improving the user experience. In the On the other hand, for two consecutive turns, and +1, next section we introduce our proposal for conversation prediction takes the following form: performance prediction for CRS.
Our aim for conversational performance prediction difers from existing approaches on QPP in a number of ways. While QPP focuses on estimating the relevance of a ranking to a given single query (single-turn), to predict the user’s satisfaction of a conversation, we need to take into account the nature of the task, which is to consider the ranking quality across multiple turns. Another important diference is that many QPP techniques are based consecutive([, +1], [, +1]).
that CPP is a distinct task from QPP that can be addressed by diferent families of predictors. In this initial work, we adapt one category of score-based QPP predictors into the CPP framework, which we discuss further below.
that study the distributions of retrieval scores and the use of reference lists, as introduced in Section 2.1.2. In particular, we have the following initial intuitions concerning successful interactions in the CRS task: We now experiment to address salient aspects upon both the nature of the predictors (single-turn and consecutive turn), as well as upon the accuracy of the predictors on diferent prediction horizons, i.e., at what point can a prediction be made, and how does it correspond to the efectiveness of the CRS, as measured at a later turn. In particular, we measure short-term horizons (i.e., can we predict the efectiveness of the next turn?); and long-term horizons (i.e., can we predict the efectiveness of the last turn); as well as measuring the longevity of the prediction (i.e., how useful is an early prediction?). Focusing initially on single-turn predictors, our first research question is:
RQ1 Can we predict conversation performance with predictors based on retrieval scores of a single turn, in terms of (a) long-term and (b) short-term prediction, as well as (c) longevity? Secondly, we consider the consecutive-turn predictors:
scipy.stats.spearmanr.html
aims to determine the extent that the overall conversation can be successfully predicted (i.e. the ground truth is the rank of the target item at turn 10). Overall, the correlations2 are weak (-0.184 is the strongest observed for Shoes, and -0.160 for Shirts; Dresses is lower still at -0.080), yet significant ( < 0.05). This suggest the dificulty of the long-term prediction task. We do observe that correlations are relatively higher as the prediction turn increases - thus indicating that it is easier to predict performance at turn 10 using evidence of the ranking at turn 10. Finally, among the predictors, the maximum score at each turn, along with the mean score, exhibit higher correlations the standard deviation. To answer RQ1 (a), we cannot suficiently predict long-term conversation
observed correlations are negative, as our CRS system uses representation distances rather than similarities. performance using single-turn score-based predictors.
Turning next to the second group of columns, we observe stronger correlations. Indeed, the overall higher correlations suggests that predicting at turn 2 gives more accurate predictions, particularly when aiming to predict conversation performance at turn 2 or shortly thereafter.
In particular, for the Shoes datasets, medium strength correlations of -0.423 are observed - these are in line with the best accuracy of some QPP predictors for adhoc search tasks [12, 25, 30, 24]. Correlations of -0.305 and − 0.281 are observed for Shirts and Dresses, respectively.
Among the predictors, top-1@k is again most successful Figure 2: Results of the diference in the top-1 ranked item on Shoes, but on Dresses and Shirts, where correlations (maximum score) between pairs of consecutive turns as a conare lower, the overall picture is less clear across diferent secutive turn CPP predictor for each of the datasets. prediction horizons (i.e. as the ground truth is varied).
For these datasets, mean is the most accurate for most values of ≥ 2. In general, when predicting conver- turn. In RQ2 below, we focus on short-term (next turn sation performance using single-turn retrieval scores, prediction), as the most promising CPP setting. prediction becomes less accurate as the longevity of the prediction increases, thus answering RQ1(c). 6.2. RQ2 - Consecutive-Turn predictors
Finally, the last set of columns of the table shows the correlation of the scores of each turn (as a predictor) Figure 2 presents the results of our first consecutive-turn when the efectiveness of the following turn + 1 is used predictor, namely the diference in maximum score (topas the ground truth (i.e. applying a short-term horizon). 1 item) for each pair of turns , + 1 when predicting The scores of both the top-ranked item and the average the rank of the target item at turn + 1. Within the score of the top-ranked items at turn suficiently pre- ifgure, each dataset is represented as a separate curve. dict the rank of turn + 1, especially for early turns. This Considering the diferent datasets, for Shirts and Dresses, trend weakens as the number of turns increases, but the we observe a similar trend across turns, starting from observed correlations remain quite high for some cases. a correlation of -0.18 (the maximum value obtained for For example, for Shoes, we start with a correlation of this predictor) at turns 2-3, which gradually decreases as -0.423 (maximum score) and -0.413 (average score) for the number of turns increases. In contrast, Shoes does turns 2, 3 and at turns 8 − 9 the correlation is still -0.20. not achieve any correlation stronger than -0.016 at turns For Shirts, the maximum and average score of top items 3-4. Therefore, we observe only weak correlations for suficiently predict the ranking of turn 3 at -0.30 and the this predictor at short-term prediction, although some score of turn 8 still at -0.20. Finally, although weaker correlations are significant. To answer RQ2(a), using the than the other two datasets, the two predictors work rea- scores of two consecutive turns, does not suficiently presonably well for Dresses, achieving a maximum value of dict conversation performance, and is indeed generally -0.26 for predicting the rank of turn 3. These values sug- less efective than the predictors examined in RQ1. gest some evidence for short-term prediction when using Next, we test our final predictor, which considers the single-turn score-based predictors, to answer RQ1(b). overlap of top-ranked items (i.e., the size of intersection)
Overall, we observe that there is some evidence for between consecutive turns. We considered various rank short (score of one turn predicting the rank of the fol- cutof values for calculating the overlap, ranging from lowing turn) and early prediction (a score of initial turn rank 5 to rank 1000, and all pairs of turns. Figure 3 repredicting the rank of some turns ahead). The score ports the observed correlations (y-axis), where each pair of the top-ranked item and the mean scores of the rec- of turns is a curve, and the x-axis is the rank cutof at ommendation list are shown to be the most promising which overlap is calculated. Recall that we expect that single-turn predictors. However, contrary to previous when the retrieved items are generally similar, this may QPP research [25], the results for the standard deviation be indicative that the CRS is reaching a stable concluare not as encouraging. The results for long-term predic- sion of the likely relative items. If this occurs at a later tion are weaker, but still, the score of the initial turn is turn, we may be further confident in the likely positive predictive of later stages. In general, prediction of the performance of the system. system performance (whether it finds the target in the On analysing Figure 3, we note that Dresses & Shirts context of a conversation) is possible by using single-turn (Figure 3(b) & (c), respectively) – which are both Fashion score-based predictors, particularly for the success of the IQ datasets – we observe a strengthening trend in the conversation at early turns and prediction of the next correlations as we increase the rank cutof value (more (a) Shoes (b) Dresses (c) Shirts items are considered). This happens for all pairs of turns indicative of the ranking of the users’ target items in the except the initial turn. In addition, the correlations are recommendation list. stronger for later turns than earlier turns, indicating that In our analysis of the proposed single-turn predictors, this predictor is more useful for later turns (as expected). we found that examining the score of the top-ranked Indeed, improved prediction at later turns is particularly items had a medium correlation with the efectiveness of notable, as this contrasts with our results in RQ1, where the conversation, particularly the efectiveness at early earlier prediction was more accurate. turns. Indeed, we observed a Spearman’s of 0.423 on
On the other hand, for the Shoes dataset, the highest the Shoes dataset, which is comparable to correlations correlations are observed for turns 3-4 and 4-5, and for observed for standard QPP predictors on adhoc search cutof values at 50 and 100. The correlations for item over- tasks [12, 24, 25, 30]. However, these single-turn lap in Shoes are weaker than the other two datasets, con- predictors became less useful at predicting the success of trasting with the observations in RQ1 (where Shoes ex- later turns. On the other hand, among our consecutive hibited higher correlations for the single-turn predictors turn predictors, simply examining the overlap of the than Dresses or Shirts). We note that, as a CRS dataset, retrieved lists had a weak-medium correlation with late Shoes is “easier” than Dresses (e.g. the GRU model can turn efectiveness on two out of our three datasets. attain Mean Reciprocal Rank 0.2 at turn 10 on Shoes, Overall, the weak-medium correlations observed for compared to Mean Reciprocal Rank 0.075 at turn 10 on our simple unsupervised predictors of diferent families Dresses [10]). We postulate that early single-turn pre- suggests that there is significant scope to extend this diction works well on Shoes, as more conversations are work, for instance by introducing supervised predictors. answered at earlier turns; in contrast, on Dresses, more Moreover, our proposed framework for CPP is generaliscritiques are required for successful conversations, and able - for instance, we can also envisage predictors that the overlap-based evidence later in the conversation is examine aspects of the critiques (for instance, repeated therefore more useful for prediction. critiques), or characteristics of the retrieved images (are
Overall, these results suggest some weak-medium cor- item colours or styles varied). We leave these for future relations (upto -0.25 ) on the overlap-based consecutive work. Furthermore, we also aim to extend our analyses turn predictor, thereby answering RQ2(b). to a classification task that aims to predict whether a conversation would fail, as well as testing the eficacy of interventions for failing conversations. 7. Conclusions Finally, this study takes place in the context of user simulators for evaluation of CRS - such user simulators are common in the training and evaluation of conversational systems. Logging the interactions of a deployed CRS would allow to verify the results depicted here.
We have presented a novel framework for conversational performance prediction (CPP) that aims to detect the factors that indicate efective performance by taking into account the multi-turn aspect of the task of conversational interactive image retrieval. In this regard, we proposed a number of predictors that can be used for Acknowledgments both short-term and long-term prediction, and explored the retrieval scores and retrieved items, of both a single Maria Vlachou’s work was supported by the UKRI Centre turn and consecutive turns. We conducted our analy- for Doctoral Training in Socially Intelligent Artificial ses on three widely-used relative captioning datasets for Agents, Grant number EP/S02266X/1. conversational recommendation systems (CRS) and examined the extent to which our proposed predictors are [27] R. Cummins, Document score distribution models 1257–1260.
for query performance inference and prediction, [39] H. Zamani, S. Dumais, N. Craswell, P. Bennett, ACM Transactions on Information Systems (TOIS) G. Lueck, Generating clarifying questions for in32 (2014) 1–28. formation retrieval, in: Proceedings of the Web [28] F. Diaz, Performance prediction using spatial au- conference 2020, 2020, pp. 418–428. tocorrelation, in: Proceedings of the 30th annual [40] H. Roitman, S. Erera, G. Feigenblat, A study of international ACM SIGIR conference on Research query performance prediction for answer quality and development in information retrieval, 2007, pp. determination, in: Proceedings of the 2019 ACM 583–590. SIGIR International Conference on Theory of Infor[29] A. Shtok, O. Kurland, D. Carmel, Query perfor- mation Retrieval, 2019, pp. 43–46. mance prediction using reference lists, ACM Trans- [41] W. Shi, K. Qian, X. Wang, Z. Yu, How to build user actions on Information Systems (TOIS) 34 (2016) simulators to train RL-based dialog systems, arXiv 1–34. preprint arXiv:1909.01388 (2019). [30] A. Shtok, O. Kurland, D. Carmel, Using statistical [42] X. Li, Z. C. Lipton, B. Dhingra, L. Li, J. Gao, Y.decision theory and relevance models for query- N. Chen, A user simulator for task-completion performance prediction, in: Proceedings of the 33rd dialogues, arXiv preprint arXiv:1612.05688 (2016). international ACM SIGIR conference on Research [43] N. Tintarev, J. Masthof, A survey of explanations and development in information retrieval, 2010, pp. in recommender systems, in: Proc. IEEE data engi259–266. neering workshop, IEEE, 2007, pp. 801–810. [31] J. Laferty, C. Zhai, Document language models, [44] Y. Wu, C. Macdonald, I. Ounis, Partially observable query models, and risk minimization for informa- reinforcement learning for dialog-based interactive tion retrieval, in: Proceedings of the 24th annual recommendation, in: Proceedings of ACM RecSys, international ACM SIGIR conference on Research 2021. and development in information retrieval, 2001, pp. [45] Y. Wu, C. Macdonald, I. Ounis, Multimodal con111–119. versational fashion recommendation with positive [32] V. Lavrenko, W. B. Croft, Relevance-based language and negative natural-language feedback, in: Promodels, in: ACM SIGIR Forum, volume 51, ACM ceedings of ACM Conversational User Interfaces, New York, NY, USA, 2017, pp. 260–267. 2022. [33] W. Webber, A. Mofat, J. Zobel, A similarity mea- [46] B. Hidasi, A. Karatzoglou, L. Baltrunas, D. Tikk, sure for indefinite rankings, ACM Transactions on Session-based recommendations with recurrent Information Systems (TOIS) 28 (2010) 1–38. neural networks, arXiv preprint arXiv:1511.06939 [34] O. Kurland, A. Shtok, S. Hummel, F. Raiber, (2015).
D. Carmel, O. Rom, Back to the roots: A probabilistic framework for query-performance prediction, in: Proceedings of the 21st ACM international conference on Information and knowledge management, 2012, pp. 823–832. [35] J. Kang, K. Condif, S. Chang, J. A. Konstan, L. Terveen, F. M. Harper, Understanding how people use natural language to ask for recommendations, in:
Proc. RecSys, 2017, pp. 229–237. [36] I. Sekulić, M. Aliannejadi, F. Crestani, Exploiting document-based features for clarification in conversational search, in: European Conference on
Information Retrieval, Springer, 2022, pp. 413–427. [37] M. Aliannejadi, H. Zamani, F. Crestani, W. B.
Croft, Asking clarifying questions in open-domain information-seeking conversations, in: Proceedings of the 42nd international acm sigir conference on research and development in information retrieval, 2019, pp. 475–484. [38] J. Kiesel, A. Bahrami, B. Stein, A. Anand, M. Hagen, Toward voice query clarification, in: The 41st international ACM SIGIR conference on research & development in information retrieval, 2018, pp.