Conversational recommender systems (CRS) are interactive agents that support their users in recommendation-related goals through multi-turn conversations. Generally, a CRS can be evaluated in various dimensions. Today's CRS mainly rely on ofline (computational) measures to assess the performance of their algorithms in comparison to diferent baselines. However, ofline measures can have limitations, for example, when the metrics for comparing a newly generated response with a ground truth do not correlate with human perceptions, because various alternative generated responses might be suitable too in a given dialog situation. Current research on machine learning-based CRS models therefore acknowledges the importance of humans in the evaluation process, knowing that pure ofline measures may not be suficient in evaluating a highly interactive system like a CRS.
Conversational recommender systems (CRS) support their users in finding items of interest through multidialogs, often in natural language [ 1]. A CRS is generally considered a highly interactive system, where users converse with the agent and seek for recommendations. Due erally complex and consist of multiple components, see e.g., [1, 2, 3, 4]. Overall, the eventual goal of a CRS is to support non-trivial yet useful interactions with their users [5]. vironment is generally challenging and can be both time volved in the loop, see also [6]. For example, assessing the quality of responses and thereby dialogs is as important 0000-0001-9418-753 (A. Manzoor); 0000-0002-4698-8507 (D. Jannach)
© 2022 Copyright for this paper by its authors. Use permitted under Creative Commons License meaningful given the previous dialog history. Moreover, in reality, the system might respond to a user’s utterance in a meaningful way, but may not match the ground truth [11]. Ultimately, ofline linguistic metrics may therefore not fully inform us about the users’ quality perceptions in practice.
Evaluating the usefulness of CRS in the academic en- sistency, naturalness or fluency as a proxy for the lanand resource intensive in particular when humans are in- in its own. In this context, to assess language quality, as assessing the quality of the underlying recommenda- responses with their ground truths using the BLEU [7] or for most recent “end-to-end” learning approaches, where as proxies. In addition to recommendation quality, ofdeep neural network models are trained using recom- fline experiments are common for assessing the dialog mendation dialogs collected between humans, see, e.g., quality. Specifically, linguistic measures such as distinct [12, 13, 14, 15, 16]. In these recent works, we therefore N-gram and perplexity to assess the diversity and fluency ifnd studies involving humans, and experiments are con- of the system-generated responses have been applied ducted using various evaluation methodologies. How- in various recent works on CRS, see, e.g., [2, 12, 13, 14]. ever, such evaluations often have limitations. For exam- Similarly, inspired by the machine translation domain, ple, in [17] human judges were asked to provide a relative metrics like BLEU or NIST are applied in several works ranking of the responses by diferent systems. In case of on CRS, where the system response is compared with a relative comparison, it remains unfortunately unclear if given ground truth in order to estimate the overall quality any of the compared systems are useful at all [18]. More- of the generated responses, see, e.g., [2, 15, 19, 22]. over, the scope of such studies seems limited as in many Given the interactive nature of CRS, studies involving cases there are only a few evaluators involved and some- humans in the evaluation process are not uncommon in times the details regarding the background of the human the CRS literature. Such studies mainly assess the quality judges are missing too. Also, often only the language aspects from a user’s perspective. See [23, 24, 6] for a quality of the responses is the main focus of such studies. set of relevant quality attributes. Looking at most recent An assessment if the made recommendations are suitable works, diferent studies were conducted in which human in an ongoing dialog context is sometimes missing, see, judges were tasked to rate or rank system responses in e.g., [14, 19]. various dimensions. For example, in [17], ten evaluators
In this work, we present a user-centric evaluation were asked to rank the responses by baseline recomapproach to CRS that can be used to assess both lin- menders and the proposed system in terms of the overall guistic and recommendation quality aspects along with quality. Similarly, in [19], the authors reported a study the INFACT, an onlIne humaN evaluation Framework in which human evaluators had to rate the responses on for conversAtional reCommender sysTems. We have a scale from 1-5 in terms of Fluency, Consistency, Natuapplied our evaluation approach for a number of stud- ralness, Persuasiveness, and Engagingness. Similarly, in a ies [11, 20, 21], using the INFACT framework as a ba- recent work [14], five human judges were given the task sis. To easily involve a larger set of subjects than in to rate the responses generated by the proposed system earlier studies, the INFACT framework is prepared to and various baselines on a scale from 1-3 in terms of support the evaluation task through online crowdwork- Fluency, Coherence, Informativeness, and Interoperability. ing platforms. We believe that our study design may Similar examples of such studies can also be found in serve as a blueprint for future human evaluation studies [13, 12, 15, 25]. for CRS. It can furthermore be easily extended to evalu- Interestingly, such studies mainly focused on linguistic ate dialogs systems, machine translation, or Q&A tasks. aspects of the systems’ responses. If the recommendaWe release the source code of the INFACT framework at tions themselves were considered meaningful—a main https://github.com/ahtsham58/INFACT. aspect in terms of a system’s usefulness—was not assessed with the help of human judges but rather evaluated through ofline analyses. On the other hand, the 2. Related Work generic concept of meaningfulness of a response can be used to evaluate both aspects, i.e., language and recommendation quality, see, e.g., [11, 20, 21]. Moreover, in such studies the details about the study setup and background of the evaluators were quite brief and sometimes missing at all. Furthermore, several studies were conducted with a small number of judges, for example, in [12, 13], three judges were involved and no information was provided regarding their linguistic expertise.
Given the potential limitations of ofline experiments and of user studies with unclear significance, we provide a study design that may serve as a template for scalable human-centric evaluation studies of dialog systems.
Next, we explain the experiment design of our evaluation approach and highlight the features that the INFACT framework ofers to support the evaluation task through online crowdworking platforms.
According to recent surveys on CRS [1, 6], we can generally observe three main dimensions in which a CRS can be evaluated: (i) efectiveness of task support , i.e., the ability of the system to support a recommendation-related task, (ii) eficiency of task support , i.e., how much efort is required by the user, and (iii) conversation quality and usability, which may cover aspects like fluency, naturalness, or the consistency of the system responses. All of these aspects can contribute to the success of a CRS in practice.
From a methodological standpoint, quality measurements are typically either made with the help of computational (“ofline”) experiments or with studies involving humans in the loop. In ofline experiments, system efectiveness is often evaluated in terms of recommendation quality, where metrics like precision or recall are used
INFACT
based on the concept that user-centric evaluations are vital to assess the efectiveness of highly interactive systems like CRS. Moreover, evaluating such systems requires studies at scale in order to investigate the quality of both linguistic and recommendation aspects in practice.
Specifically, in our approach we ask human subjects to assess dialog continuations (“system responses”) provided by a CRS given a piece of dialog (“dialog situation”) using one or more quality criteria. In our own studies, we use the ReDial dataset [17] consisting of real-world dialogs for such evaluations. We note that various recent current CRS approaches rely on this dataset or similar ones to generate or retrieve suitable responses, e.g., [15, 19, 26].
To avoid biases and to receive feedback for all stages of the dialog, the dialog situations to be evaluated are selected from such datasets at random [27, 28]. In addition, we assume to have a larger set of participants than many existing studies; in particular we consider crowdsourcing to be helpful.
Such an approach can be instantiated in various ways, depending on the research question(s). For example, an experiment could include one or more algorithms to evaluate for each participant. Similarly, there could be one or more questions regarding the dialog quality and for each dialog continuation. Moreover, each participant can be tasked to assess only one or more dialog situations and the feedback scale could be diferent too.
When deciding on these specifics, it is important to keep the cognitive load and the overall workload for the study participants in mind. Moreover, the specific design also determines how many participants are required to achieve a suficient number of human judgements. In our own experiments, as discussed later, we decided to ask participants to assess exactly three diferent dialog • Study overview • Instructions • Data privacy • Informed consent • Dialog Situation • Responses • User ratings
Chatbot
Experience • Naturalness • Realism • Usefulness • Intention to reuse
About you
Done! • Demographic
features • Chatbot usage • Greeting message • Hit‐code • Exit continuations, and they had to assess ten such dialog dialog situation. In case an item recommendation(s) is insituations. As a result, we obtain multiple assessments cluded in the response, it has to match to the user’s stated from each participant, which helps to keep the number interest and preferences. If the system response does not of participants low. However, using such a design, it is include a movie recommendation, e.g., chit-chat sentence, important to check for intraclass (per user) correlations the participants are supposed to rate the meaningfulness in the statistical analyses. of the response as a reply to the user’s last utterance
Arbitrary dialog datasets can be used with our frame- while considering also the context of the ongoing dialog. work, as long as they follow the format used in the frame- Overall, human judges are supposed to provide ratings work, which is currently based on the ReDial dataset, as based on their subjective quality perceptions. mentioned. Finally, arbitrary post-task questionnaire el- From the linguistic perspective, a deep discussion of ements can be introduced, and the INFACT framework the concept ‘meaningfulness’ is provided in [29], where implements a number of typically required functional- the author makes a distinction between ‘grammaticalities, e.g., for persistently storing the feedback into a ity’ and ‘meaningfulness’. In our study design, instead database. of challenging participants with complex linguistic concepts or considerations, we provide examples and guideA Specific Implementation In our experiments [11, lines when a response can be considered meaningful or 20, 21], each participant is presented with dialog situa- meaningless.2 tions that always start from the first utterance and end In this way, each evaluator assessed ten such dialog with a user utterance. Below the dialog situation, as situations. However, this is a configurable parameter shown in Figure 1, we show three responses to the last that can changed based on the experiment design. On user utterance by three diferent CRS under compari- submission, we store the rating scores, including the son. To highlight and diferentiate item recommenda- dialog situation, corresponding responses, and the overall tions from regular language words in the dialog situation time it took for the evaluator to rate, in a NoSQL cloud and responses, we enclosed item names, e.g., in this case database. We relied on a NoSQL database as it ofers movie titles, in double quotes1. lfexible, afordable, and scalable database management.
In our experiments, the only question for the study In order to check if participants are attentive during participants was to independently assess (or rate) the the study, one of the ten dialog situations that appear in quality of each response in terms of the meaningfulness of random order is used as an attention check. Specifically, the responses in the given dialog context. To obtain fine- in one of the three responses for this dialog situation, we grained assessments, we use a 5-point scale labeled from asked the participants to select a particular rating from ‘Entirely meaningless’ to ‘Perfectly meaningful’, which the given scale. The attention check was considered to can be modified depending on the research question. To be failed whenever a study participant did not select the avoid any sequential rating bias by the evaluators, the required score. In this case, we completely discard all data order of showing responses to the user is randomized. from such unreliable crowdworkers3. Furthermore, apart
On the landing page, we provide specific instructions to the evaluators about how to judge the meaningfulness of responses. For example, a response by the specific system should be logical continuation of the provided 2The term “meaningfulness” is also used in the context of a human evaluation in [9]. Diferently from our work, the term “meaningful” is used in [9] to summarize other evaluation dimensions in an informal way. We note that our framework can easily be configured to collect annotator feedback on several dimensions, e.g., fluency, coherence, or informativeness as in [14]. 3We provide the Python script to automatically parse the study data
explained in the online repository. garding fluency or interpretability as in [ 14], appropriate measures must be taken to ensure that the crowdworkers are able to fulfill the task reliably, e.g., by providing more instructional material or by requiring certain skills.
After the submission of ratings for ten such dialog situations, a post-task questionnaire is shown to the participants, where we collect general feedback regarding the quality of dialogs, demographics, and general remarks or suggestions. Table 1 shows parts of a questionnaire for dialog quality, asking, for example, if the shown dialogs feel natural, realistic, or useful. An example list of demographic questions is shown in Table 2. These questionnaires can be modified depending on the research question(s). The overall workflow of our experiment design is visualized in Figure 2.
To enable and support studies through crowdworking platforms like Amazon Mechanical Turk, Prolific, etc., we used a pre-implemented feature like hit-code generation. Our assumption is that a large number of human judges are needed for the evaluation, hence the INFACT framework is prepared accordingly. Technically, the INFACT framework is a web-based application developed using the Django framework in Python 3.0, and Bootstrap 4.4.
has attracted increased attention in recent years. The most recent proposals on CRS, and in particular ones Education level that follow an end-to-end learning paradigm, mainly rely on computational measures in order to demonstrate the efectiveness of their systems in comparison to diferent baselines. However, the aspects that contribute to the Frequency of watching movies success or failure of a CRS may not be fully assessed without involving humans in the evaluations process.
In this work we provide a user-centric evaluation apEver interacted with a chat-bot proach for CRS, which can be used to investigate both recommendation and linguistic quality aspects of system Ever interacted with a chat-bot for responses in a given dialog. Since the scope of several getting movie recommendations studies reported in the context of recent CRS seems limited, we propose an online evaluation tool which can be used to perform human evaluations at scale with the from the explicit attention check, the INFACT framework help of crowdworkers. Due to the modular and flexible is equipped with various implicit checks to deal with nature of the architecture underlying the INFACT framethe potentially unreliable crowdworkers such as time work, it can be modified and adapted in the context of interval (in seconds) for each individual event, overall a similar study design. For example, replacing the scale study completion time, etc. or metric requires only a few modifications on a single
In our specific experiments, no particular training or template page. Ultimately, we hope that our user-centric expertise is required by the crowdworkers to participate. evaluation approach can be considered as a template to To fulfill the task, the crowdworkers were asked for their facilitate the design of similar studies in domains like subjective assessment regarding the generated system dialog systems, Q&A or machine translation. responses in terms of their meaningfulness. In case participants should answer more complex questions, e.g., restored in the JSON format on the cloud. system, in: ICCL ’20, 2020, pp. 4128–4139. [16] J. Zou, E. Kanoulas, P. Ren, Z. Ren, A. Sun, C. Long, [1] D. Jannach, A. Manzoor, W. Cai, L. Chen, A survey Improving conversational recommender systems on conversational recommender systems, ACM via transformer-based sequential modelling, in: Computing Surveys 54 (2021) 1–36. SIGIR ’22, 2022, pp. 2319–2324. [2] K. Chen, S. Sun, Knowledge-based conversational [17] R. Li, S. E. Kahou, H. Schulz, V. Michalski, L. Charlin, recommender systems enhanced by dialogue policy C. Pal, Towards deep conversational recommendalearning, in: IJCKG ’21, 2021, pp. 10–18. tions, in: NIPS ’18, 2018, pp. 9725–9735. [3] A. Rana, D. Bridge, Navigation-by-preference: A [18] D. Jannach, A. Manzoor, End-to-end learning for new conversational recommender with preference- conversational recommendation: A long way to based feedback, in: IUI ’20, 2020, p. 155–165. go?, in: IntRS Workshop at RecSys ’20, Online, [4] D. Jannach, ADVISOR SUITE – A knowledge-based 2020.
sales advisory system, in: ECAI ’04, 2004, pp. [19] S. A. Hayati, D. Kang, Q. Zhu, W. Shi, Z. Yu, IN720–724. SPIRED: Toward sociable recommendation dialog [5] D. Jannach, L. Chen, Conversational Recommen- systems, in: EMNLP ’20, 2020. dation: A Grand AI Challenge, AI Magazine 43 [20] A. Manzoor, D. Jannach, Conversational recom(2022). mendation based on end-to-end learning: How far [6] D. Jannach, Evaluating conversational recom- are we?, Computers in Human Behavior Reports mender systems, Artificial Intelligence Review (2021) 100139.
forthcoming (2022). [21] A. Manzoor, D. Jannach, Generation-based vs. [7] K. Papineni, S. Roukos, T. Ward, W.-J. Zhu, Bleu: A retrieval-based conversational recommendation: A method for automatic evaluation of machine trans- user-centric comparison, in: RecSys ’21, 2021. lation, in: ACl ’02, 2002, p. 311–318. [22] A. Bartl, G. Spanakis, A retrieval-based dialogue [8] G. Doddington, Automatic evaluation of machine system utilizing utterance and context embeddings, translation quality using n-gram co-occurrence in: ICMLA ’17, 2017, pp. 1120–1125. statistics, in: HLTR ’02, 2002, pp. 138–145. [23] P. Pu, L. Chen, R. Hu, A user-centric evaluation [9] L. Nie, W. Wang, R. Hong, M. Wang, Q. Tian, Mul- framework for recommender systems, in: RecSys timodal dialog system: Generating responses via ’11, 2011, pp. 157–164.
adaptive decoders, in: MM ’19, 2019, pp. 1098–1106. [24] Y. Jin, L. Chen, W. Cai, P. Pu, Key qualities of con[10] Q. Chen, J. Lin, Y. Zhang, H. Yang, J. Zhou, J. Tang, versational recommender systems: From users’ perTowards knowledge-based personalized product de- spective, in: HAI ’21, 2021, pp. 93–102. scription generation in e-commerce, in: KDD ’19, [25] F. Pecune, S. Murali, V. Tsai, Y. Matsuyama, J. Cas2019, pp. 3040–3050. sell, A model of social explanations for a conversa[11] A. Manzoor, D. Jannach, Towards retrieval-based tional movie recommendation system, in: HAI ’19, conversational recommendation, Information Sys- 2019, p. 135–143.
tems (2022) 102083. [26] D. Kang, A. Balakrishnan, P. Shah, P. Crook, Y.-L. [12] Q. Chen, J. Lin, Y. Zhang, M. Ding, Y. Cen, H. Yang, Boureau, J. Weston, Recommendation as a commuJ. Tang, Towards knowledge-based recommender nication game: Self-supervised bot-play for goaldialog system, in: EMNLP-IJCNLP ’19, 2019, pp. oriented dialogue, in: EMNLP-IJCNLP ’19, 2019, pp. 1803–1813. 1951–1961. [13] K. Zhou, W. X. Zhao, S. Bian, Y. Zhou, J.-R. Wen, [27] W. Cai, L. Chen, Towards a taxonomy of user J. Yu, Improving conversational recommender sys- feedback intents for conversational recommendatems via knowledge graph based semantic fusion, tions, in: RecSys’ 19 Late-Breaking Results, 2019, in: KDD ’20, 2020, pp. 1006–1014. pp. 572–573. [14] J. Zhou, B. Wang, R. He, Y. Hou, CRFR: Improv- [28] S. Lyu, A. Rana, S. Sanner, M. R. Bouadjenek, A ing conversational recommender systems via flexi- workflow analysis of context-driven conversational ble fragments reasoning on knowledge graphs, in: recommendation, in: WWW ’21, 2021, pp. 866–877.
EMNLP ’21, 2021, pp. 4324–4334. [29] Y. Wilks, Decidability and natural language, Mind [15] K. Zhou, Y. Zhou, W. X. Zhao, X. Wang, J.-R. Wen, (1971) 497–520.
Towards topic-guided conversational recommender