Conversational Recommender Systems (CRS) have received increased interest in recent years due to advances in natural language processing and the wider use of voice-controlled smart assistants. One technical approach to build such systems is to learn, in an end-to-end way, from recorded dialogs between humans. Recent proposals rely on neural architectures for learning such models. These models are often evaluated both with the help of computational metrics and with the help of human annotators. In the latter case, the task of human judges may consist of assessing the utterances generated by the models, e.g., in terms of their consistency with previous dialog utterances. However, such assessments may tell us not enough about the true usefulness of the resulting recommendation model, in particular when the judges only assess how good one model is compared to another. In this work, we therefore analyze the utterances generated by two recent end-to-end learning approaches for CRS on an absolute scale. Our initial analyses reveals that for each system about one third of the system utterances are not meaningful in the given context and would probably lead to a broken conversation. Furthermore, about less than two third of the recommendations were considered to be meaningful. Interestingly, none of the two systems “generated” utterances, as almost all system responses were already present in the training data. Overall, our works shows that (i) current approaches that are published at high-quality research outlets may have severe limitations regarding their usability in practice and (ii) our academic evaluation approaches for CRS should be reconsidered.
• Information systems → Recommender systems;
In Conversational Recommender Systems (CRS), a software agent
interacts with users in a multi-turn dialog with the goal of
supporting them in finding items that match their preferences [
In recent years, CRS have gained increased research interest
again, mostly due to technological progress in the context of
natural language processing (NLP) and the wide-spread use of
voicecontrolled smart assistants like, e.g., Apple’s Siri, Amazon’s Alexa
or Google Home. Diferently from many previous works that are
based on substantial amounts of statically defined domain
knowledge (e.g., about item properties or pre-defined dialog states and
transitions), some recent approaches try to adopt an end-to-end
learning approach [
The evaluation of the usefulness of such CRS in academic environments in general is, however, challenging. To assess the quality of a system, it is not only important to check if the recommendations are adequate in a given dialog situations, one has also to assess the quality of the dialog itself. Dialog quality could, for example, also relate to the question if a system is able to react to chit-chat utterances (phatic expressions) in an appropriate way.
In some recent works, researchers use a combination of objective
and subjective measures to assess a CRS. Objective measures can,
for example, be typical recommendation accuracy measures, but
may also include linguistic measures like perplexity that capture the
lfuency of natural language [
In this work, we therefore analyze quality aspects of two
stateof-the-art end-to-end learning systems ([
Our analysis shows that about one third of the utterances
generated by the investigated systems are considered to be not
meaningful in the given dialog context. Moreover, in both of the systems,
about more than one third of the recommendations did not suit the
assumed preferences of the recommendation seeker. These
observations raise questions both regarding the practical usefulness of
the proposed systems and the way we evaluate CRS in academia. A
main implication of our analyses is that more realistic ways of
evaluating CRS are needed. In particular, such evaluation approaches
should help us understand if a (i) proposed end-to-end learning
system reaches a quality level, in terms of generating plausible
responses, that is actually acceptable for users and if it is (ii) able to
avoid bad recommendations that can be detrimental to the system’s
quality perception and use [
Technical Approaches – DeepCRS and KBRD. We analyzed two
recent approaches from the literature. The first one, which we denote
as DeepCRS, was published at NeurIPS 2018 [
The second approach is called KBRD [
Underlying Data – The ReDial Dataset. Both approaches, DeepCRS and KBRD, are trained on the ReDial1 dataset. This dataset was collected by the authors of DeepCRS with the help of crowdworkers and consists of more than 10,000 conversations between a recommendation seeker and a recommender. The crowdworkers were given specific instructions regarding the conversation: For example, each participant had to take one of two roles, seeker or recommender. The seeker had to specify which movies she likes, and the recommender’s task was to make recommendations based 1https://redialdata.github.io/website/ on the assumed interests of the seeker. The conversations were collected through a web-based interface, where the crowdworkers typed their utterances in natural language. At least four movie mentions were required per dialog, and each dialog had to have at least ten utterances. The resulting conversations were then further enriched. Movie mentions were tagged with movie names and release years. Furthermore, diferent labels were assigned to the movies, e.g., whether or not a dialog seeker has seen or liked it.
Original Evaluation. The DeepCRS system was evaluated in
different dimensions, including the quality of sentiment classification,
recommendation quality, and overall dialog assessment. Accuracy
was measured in terms of the Cohen’s kappa coeficient [
The KBRD approach was evaluated in three dimensions. Two computational metrics, perplexity and distinct n-gram measure the lfuency and diversity of the natural language. Recommendation quality was measured in terms of Recall. KBRD proved to be favorable over two baselines (DeepCRS and a Transformer model) in terms of all computational metrics. For the human evaluation, ten annotators with knowledge in linguistics were asked to assess the consistency of a generated utterance with the previous dialog history on a scale from 1 to 3. An average consistency rating of 1.99 was obtained, which was about 15% higher than the average rating for the DeepCRS baseline.
Overall, the (relative) ranking exercise for DeepCRS unfortunately does not tell us much about the absolute meaningfulness and usefulness of the system-generated utterances. For KBRD, the evaluation was done on an absolute scale. The average score was at about 2 on the 1-3 scale, i.e., in the middle. No details are, however, provided regarding the distribution of the ratings. It is, for example, not clear if trivial system responses like goodbye to a seeker’s goodbye were counted as consistent dialog continuations.
Given the dificulty of assessing the usefulness of the proposed systems from the reported studies, our goal was to assess the quality of the system responses through a complementary analysis. 3
Looking at example conversations published in the supplementary
material of [
First, we randomly selected 70 dialogs from the ReDial test dataset. We then used the code provided by the authors of DeepCRS and KBRD to reproduce the systems and to generate system responses after each seeker utterance, given the dialog history up to that point. As a result, we obtained 70 dialogs, which not only contained the original seeker and human recommender utterances, but also the recommender sentences that were generated by the respective CRS.
In total, 758 system responses, 399 by DeepCRS and 359 by KBRD, were generated this way. We analyzed the responses both through manual and automated processes. For replicability, we share all study materials online3. The following main analyses were made: (1) Creativeness or novelty of responses wrt. training data; (2) Meaningfulness of system responses in the given context; (3) Quality of the recommendations;
In the context of analysis (1), we were wondering how diferent the system-generated responses are from the training data. This is in particular relevant as the authors of KBRD measure perplexity and n-gram distance for the generated sentences. In our analysis, we therefore counted which fraction of the system responses was contained in an identical or almost identical form4 in the training data. In case the generated sentences were mostly identical to sentences appearing in the training data, measuring perplexity and the n-gram distance of what are mostly genuine human utterances is not very informative.
To measure aspects (2) and (3), we relied on human annotators who marked each generated system response in the 70 dialogs as being meaningful or not. Furthermore, we asked them to label each utterance as being chit-chat or containing a recommendation. Two of the three annotators were PhD students at two universities with no background in conversational recommendation. One was evaluating the DeepCRS responses, the other annotated the KBRD responses. They were not informed about the background of their task. To obtain a second opinion and to avoid potential biases, both datasets were also manually labeled by one of the authors of this paper. The annotator agreement was generally very high (92.73% for DeepCRS and 93.89% for KBRD).
When instructing the external annotators, we did not provide specific instructions what “meaningful” means. Our analysis shows 2https://papers.nips.cc/paper/8180-towards-deep-conversational-recommendations 3https://drive.google.com/drive/folders/10gPOmaiFrZjIULIa3LsdmuyvJvnCV_Xq 4We considered sentences to be almost identical if the same set of words appeared in them with the same frequency, i.e., only the order was changed. that typical non-meaningful sentences include situations where the system ignores the last user-utterance intent, repeats questions, abruptly ends the conversation, provides broken or incomplete response, or makes a bad recommendation. The judgment of what represents a bad recommendation is in many cases clear, e.g., when the system recommended a movie that the seeker has just mentioned, but to some extent it remains a subjective assessment. 4
Analysis of Generated Sentences. Table 2 shows the characteristics of the utterances that are generated by DeepCRS and KBRD. Even though both algorithms were fed with the same dialogs to compete, the number of generated sentences for KBRD is lower as this method did not always return a response.
Regarding the novelty of the returned sentences, we, to some surprise, found that DeepCRS almost exclusively returns sentences that are found in identical form in the training data (except for the placeholder for movie names that are eventually replaced by the algorithm). KBRD also mostly returns sentences that are contained in the training data or are tiny modifications of such sentences. Five of the generated sentences were not in the training data. However, there were also 11 generated sentences that were broken.
Overall, it is surprising that both systems mostly return sentences they found in the training data, which resembles more a retrieval approach than a language generation problem. Measuring linguistic properties, e.g., perplexity on the sentence level, of what are genuinely human sentences, therefore is not too meaningful.
Analysis of Dialog and Recommendation Quality. In Table 3, we show the results of the labeling process by the annotators. The numbers in the table correspond to the rounded average of two annotators who, as mentioned above, have a very high agreement.
The results show that for both systems a substantial fraction of the generated responses—about 40%—were considered not meaningful by the annotators. As a result, there are only 5 (7%) dialogs for which there is not at least one issue. Overall, these findings raise the question if such high failure rates would be acceptable by users in practice?
A major fraction of the generated sentences (33% for DeepCRS, 24% for KBRD) were considered chit-chat. The analyzed systems were performing better in terms of generating such chit-chat messages than they were when generating other types of utterances. The percentage of meaningful chit-chat responses is 85% and 87% for DeepCRS and KBRD respectively. However, a larger fraction of these chit-chat exchanges consist of trivial responses to ’hello’, ’hi’, ’goodbye’ and ’thank you’ utterances by the recommendation seeker. Overall, the chit-chat messages account for 39% of all generated sentences that were marked as meaningful.
The analysis of the quality of the recommendations themselves is what we thought would be a more subjective part of our evaluation. The agreement between the annotators was, however, very high (93% for DeepCRS and 92% for KBRD). The annotators both relied on their own expertise in the movie domain and used external sources like movie databases such as IMDb to check the plausibility of the recommendations. Recommendations were typically considered not meaningful when the annotators could not establish any plausible link between seeker preferences and recommendations. An example is the system’s recommendation of the movie “The Secret Life of Pets” after the seeker mentioned that s/he liked “Avengers Infinity Wars”. Quite interestingly, the subjective performance of KBRD is lower than for DeepCRS even though KBRD includes a knowledge graph, called DBpedia 5, that contains with information about movies and their relationships.
Overall, the results in Table 3 show that the perceived recommendation quality is modest. While for the DeepCRS and KBRD systems, less than two third of the movie recommendations were considered meaningful. However, DeepCRS produced not even a single recommendation in 7 (10%) dialogs and this was the case for 6 (8.5%) dialogs with KBRD.
Limitations of the ReDial Dataset. The existence of large-scale datasets containing human conversations is a key prerequisite for building a CRS based on end-to-end learning. The ReDial dataset is an important step in that direction. However, the dataset also has a number of limitations. This mostly has to do with the way it was created with the help of crowdworkers, which were given specific instructions about the minimum number of interactions and the minimum number of movie mentions.
As a result, many dialogs are not much longer than the minimum length, and the dialogs do not enter deeper discussions. The expression of preferences is very often based on movie mentions and only to a lesser extent based on preferences regarding certain features like genre or directions. The responses by the human recommenders are also mainly movie mentions. An explanation why a recommendation is a good match for the seeker’s preferences are not very common. Developing an end-to-end system that is capable of providing explanations, which might be a helpful feature in any CRS, therefore might remain challenging. 5https://wiki.dbpedia.org/
The DeepCRS and KBRD systems did not provide explanations
in the dialogs that we examined, except in cases where the system
generated some sort of confirmatory utterances ( “it is a very good
movie” ). Such utterances appeared in the training data and were
correspondingly sometimes selected by the systems. The number of
user intents that are actually supported by DeepCRS and KBRD are
generally quite low, see [
Finally, when examining the dialogs that we sampled, we observed that a few conversations were broken. This was for example the case when a crowdworker had not understood the instructions. In one case, for example, the seeker was not interested in a recommendation, but rather told the recommender that he would like to watch a certain movie. In our sample of 70 dialogs, we found 9 cases we considered to be broken. To what extent such noise in the data impacts the performance of end-to-end learning systems however requires more investigations in the future. 5
We performed an alternative and independent evaluation of two recently published end-to-end learning approaches to building conversational recommender systems. A manual inspection of the responses of the two systems reveals that these systems in many cases fail to react in a meaningful way to user utterances. The quality of the recommendations in these dialogs also appears to be limited.
Our findings have important implications. First, current
evaluation practices, at least those from the analyzed papers, seem to be
not informative enough to judge the practical usefulness of such
systems. Relying on relative subjective comparisons (as in [
Measuring linguistic aspects like perplexity—or the BLEU score
as done in some other works [
As a result, our work calls for extended, alternative, and more realistic evaluation practices for CRS. In particular, in practical applications, certain guarantees regarding the quality of the system responses and recommendations might be required, which might be dificult to achieve with current end-to-end learning approaches.