Conversational recommender systems (CRS) that are able to interact with users in natural language often utilize recommendation dialogs which were previously collected with the help of paired humans, where one plays the role of a seeker and the other as a recommender. These recommendation dialogs include items and entities that indicate the users' preferences. In order to precisely model the seekers' preferences and respond consistently, CRS typically rely on item and entity annotations. A recent example of such a dataset is INSPIRED, wich consists of recommendation dialogs for sociable conversational recommendation, where items and entities were annotated using automatic keyword or pattern matching techniques. An analysis of this dataset unfortunately revealed that there is a substantial number of cases where items and entities were either wrongly annotated or annotations were missing at all. This leads to the question to what extent automatic techniques for annotations are efective. Moreover, it is important to study impact of annotation quality on the overall efectiveness of a CRS in terms of the quality of the system's responses. To study these aspects, we manually fixed the annotations in performance of several benchmark CRS using both versions of the dataset. Our analyses suggest that the improved version of the dataset, i.e., INSPIRED2, helped increase the performance of several benchmark CRS, emphasizing the importance of data quality both for end-to-end learning and retrieval-based approaches to conversational recommendation. We release our improved dataset (INSPIRED2) publicly at https://github.com/ahtsham58/INSPIRED2.
Sociable conversational recommender systems (CRS) aim
to build rapport with users while interacting with them in
natural language [1, 2]. CRS that rely on natural language
processing (NLP) nowadays commonly utilize datasets of
previously recorded dialogs between humans, where one
plays the role of a recommendation-seeker and the other
as human-recommender, see e.g., [
WA, USA. ∗Corresponding author. nEvelop-O LGOBE
0000-0001-9418-753 (A. Manzoor); 0000-0002-4698-8507 (D. Jannach) generation-based CRS approaches as well as for retrievalbased approaches to build natural language conversational systems [17]. For both types of systems, the question arises to what extent better data quality, i.e., having correct annotations and noise-free conversations, leads to better results in terms of the quality of the responses terms of consistency and plausibility.
Therefore, it is important to develop datasets like IN- economically expensive process [
In this work, we study the recent INSPIRED dataset, A number of new datasets for conversational
recomin which the items and entities that were mentioned in mendation were published in recent years, e.g., [3, 21,
the recorded utterances are explicitly annotated. These 22, 23]. Such datasets, which are commonly collected
annotations were created with the help of automatic ap- with the help of crowdworkers, can however have
limiproaches using keyword or pattern matching methods. tations and may be not fully representative in terms of
However, looking at the data, we observed a substantial what we would observe in reality. In some cases, for
number of cases where items and entities were either example, crowdworkers were instructed to mention a
wrongly annotated or missing annotations at all, e.g., minimum number of movies in the conversations. This
“My favorite [ MOVIE_GENRE_1] are Groundhogs Day, leads to mostly “instance-based” conversations, where
[ MOVIE_TITLE_2] and Borat”. In addition, there were crowdworkers rather mention individual movies they
several cases where the utterances included noise, e.g., like than their preferred genres, see also, [
To understand the severity of the problem and the po- adapt a semi-automatic approach or rely on NLP-assisted
tential efects of data issues on the quality of a CRS, we tools that visualize the entities in a text in order to
rehave manually corrected the dataset by fixing the annota- duce the required manual efort [
of evaluation approaches for CRS can be found in [37].
Number of dialogs (conversations) 1,001 Like in the recommender systems literature in general,
AAvveerraaggee ttoukrnesnspeprerdiuatltoegrance 170.9.733 computational experiments that do not involve humans
in the loop are the predominant instrument to assess
Number of human-recommender utterances 18,339 the quality of a CRS. Common metrics to evaluate the
Number of seeker utterances 17,472 quality of the recommendations include Recall, Hit Rate,
or Precision [
Technological advancements particularly in fields like tem responses in general [35]. For example, when using
NLP, speech recognition, and machine learning in general metrics like the BLEU score, usually a system response is
led to the design of today’s end-to-end learning-based compared with one particular given ground truth. Such
CRS. In such approaches, recorded recommendation di- a comparison has limitations when used to estimate the
alogs between paired humans are used to train the deep average quality of a system’s responses, because there
neural models, see, e.g., [
In terms of the underlying data, the DeepCRS [
Unlike generation-based systems, in retrieval-based annotated using a regular expression to match a set of CRS the idea is to retrieve and adapt suitable responses predefined tokens. Regarding actors and directors and from the dataset of recorded dialogs. One main advan- other entities, a pattern matching technique was used, tage of retrieval-based approaches is that the retrieved where words starting with a capital letter were searched responses were genuinely made by humans and thus in the TMDB database2. A similar technique was used for are grammatically usually correct and in themselves se- movie titles. However, as mentioned, we observe a large mantically meaningful [35]. Recent examples of such number of cases where items and entities were either retrieval-based systems are RB-CRS [17] and CRB-CRS wrongly annotated or missing annotations. To answer [36], which we designed and evaluated based on the Re- our research question on the impact of the quality of the Dial dataset in our own previous work. underlying data on the quality of the responses of a CRS, Observed Issues During the annotation process, we we fixed the annotations as follows. recorded the observed issues in the original annotations. Since the original annotations were created using autoProcedure To fix the annotations, we interviewed a matic techniques, many issues were related to the liminumber of university students to assess their knowledge tations of the simple keyword or pattern matching techin the movies domain and their ability to do the correction niques. Overall, we observed a number of cases where task. Subsequently, we hired two students and instructed minor spelling mistakes or incomplete movie titles made them on how to annotate and clean the dataset. First, the exact string matching approaches inefective. they were briefed on the logical format of the original For example, in one of the utterances, “I think I am annotations and how to retain that format. Second, they waiting for Star Wars The Rise of Skywalker”, the annotawere asked to read each utterance individually, to detect tion was missing because the correct title is “Star Wars: potential noise, and to analyze which items or entities Episode IX – The Rise of Skywalker ”. Similarly, we ob(e.g., title, genre, actor, or director) are mentioned in it. serve a significant number of cases where an utterance
In case of ambiguity or obscurity, they were allowed was only partially annotated, e.g., “ok is it scary like into access online portals, e.g., IMDb3. Note that regarding cidious or [ MOVIE_GENRE_2] [ MOVIE_TITLE_5]”. In the genres, a set of 27 keywords was provided to them, addition, at places where two entities were separated which we curated and used in our earlier research [36]. with ‘/’ instead of a space, the automatic technique often After the briefing, the dataset was split evenly for both failed to create proper annotations, e.g., “Since you like annotators. On weekly basis, their performance and the [MOVIE_GENRE_1] drama/mystery, I’m going to send you accuracy of the annotations was checked by one of the the trailer to the movie [MOVIE_TITLE_3]”. authors. Finally, after annotating the complete dataset, a Also, the automatic approach used for INSPIRED somenumber of additional validation steps were applied. times had dificulties to deal with ambiguity. We found
First, using a Python script, we ensured that every a number of cases where a regular word was annotated, placeholder is enclosed by ‘[’ and ‘]’ as was done origi- although such a word did not belong to any item or ennally, e.g., [MOVIE_TITLE_1]. Second, another thorough tity. For example, in one of the cases, “Are you interested manual examination of the entire improved dataset was in a current movie in the box ofice? ”, the utterance was performed to fix any missing annotations or noise. In annotated as “Are you interested in a current movie in that context, we also double-checked the consistency of the box [ MOVIE_TITLE_0]”, where the word ‘ofice ’ was the format and of the annotations. mistakenly annotated as an item, i.e., The Ofice (2005) . Overall, the main observed issues are the following.
The INSPIRED2 Dataset In total, 1,851 new annotations were added to INSPIRED, leading to the INSPIRED2 dataset. The most mistakes or inconsistencies were found for the items, i.e., movie titles, which is the most pertinent information for developing a CRS. We present the statistics about new annotations in Table 2. Overall, we added around 20% new annotations in INSPIRED2. The number of issues that were fixed, e.g., duplicate annotations in an utterance, noise or factually wrong information in the original annotations, are not shown in the presented statistics. We release the INSPIRED2 both in the TSV and JSON format online.
Total
We included the following recent end-to-end learning
approaches in our experiments: DeepCRS [
egy labels4 [1]. This selection of models covers various design approaches for CRS, e.g., using an additional knowledge graph or not. We used the open-source toolkit CRSLab5 for our evaluations. This framework was used in earlier research as well, for example in [10, 39, 40]. For our analyses, we first trained the aforementioned CRS models using the original split ratio, i.e., 8:1:1, for each dataset. Afterwards, given the trained models and test data for each dataset, we ran three trials for each CRS and subsequently averaged the results for ofline evaluation metrics. Note that the same procedure was adapted for both versions of the dataset, i.e., INSPIRED and INSPIRED2.
Recommendation Quality Table 3 shows the accuracy results for the evaluated CRS models. Specifically, we provide the results for the diferent benchmark CRS models in terms of the performance diference when using the original and improved annotations. Overall, we can observe an almost consistent gain in performance for all models and on all metrics except Hit@50 when the improved dataset is used. The obtained improvements can be quite substantial, indicating that improved data quality can be helpful for CRS of diferent types, including (i) CRS, which do not rely on additional knowledge sources, (ii) CRS that leverage additional knowledge sources, (iii) CRS that are guided by a topic policy, and (iv) CRS that rely on pre-trained language models like BERT.
Interestingly, we see negative efects for two measurements in which Hit@50 is used as a metric. A deeper investigation of this phenomenon is needed, in particular as the other metrics at this (admittedly rather uncommon) list length, MRR@50 and NDCG@50, indicate that the improved dataset is helpful to increase recommendation accuracy. At the moment, we can only speculate that the improved annotations in the ongoing dialog histories led to more diverse or niche recommendations compared to the original dataset. We might assume that the missing annotations in many cases referred to less popular movies, so that the recommendations without the improved annotations will more often recommend popular movies, which is commonly advantageous in terms of hit rate and recall.
study to compare the perceived quality of system responses using either INSPIRED and INSPIRED2. Specifically, we randomly sampled same 50 dialog situations from each dataset. To create the dialog continuations, we used the retrieval-based CRS approaches, RB-CRS and CRB-CRS, which we proposed in our earlier work, see [36].
In order to obtain fine-grained assessments, three
human judges6 were involved. The specific task of the
judges was to assess (rate) the meaningfulness of a
system response as a proxy of its quality and consistency in a
dialog situation, see [
We used a 3-point scale for these ratings, from ‘Com- uators assessed the linguistic quality of the system repletely meaningless (1)’ to ‘Somewhat meaningless and sponses (dialog continuations), which were created either meaningful (2)’ to ‘Completely meaningful (3)’. The hu- based on the INSPIRED or the INSPIRED2 dataset. As unman judges were provided with specific instructions on derlying CRS systems, we considered the retrieval-based how to evaluate the meaningfulness of a response, e.g., approaches RB-CRS and CRB-CRS, as mentioned above. they should assess if a response represents a logical dialog For our analysis, we averaged the scores by the three continuation and evaluate the overall language quality evaluators. Table 4 shows the mean ratings across all of the given response. Overall, the human judges were dialog situations as well as the standard deviations. We provided 50 dialogs (446 responses to rate) that were pro- find that also in the case of retrieval-based approaches, duced using the INSPIRED and INSPIRED2 datasets. We improving the quality of the underlying dataset was helpalso explained the meanings and purpose of various place- ful, leading to higher mean scores, without observing holders contained in the responses to the human judges. larger standard deviations. A Student’s t-test reveals that Moreover, to avoid any bias in the evaluation process, the the observed diferences in the means are statistically judges were not made aware which response was created significant ( p<0.001). 7 for which dataset by which CRS. Also, the order of the dialogs and the system responses were randomized.
5https://github.com/RUCAIBox/CRSLab 6These judges were PhD students and were diferent than the ones who fixed the annotations.
To understand the impact of the new annotations on the responses in terms of the richness of knowledge concepts, we compute the number of items and entities that
appeared in the system responses. Specifically, we
compute the number of placeholders in the responses, before
they would be replaced by the recommendation
component, see [36]. In Table 5, we present the statistics for
RB-CRS and CRB-CRS for both dataset versions.
Overall, we find that the responses for the improved dataset
contain between 20% and 27% more concepts and
entities. We note that an increase in concepts is expected, as
INSPIRED2 has almost 20% more annotations. However,
the important observation here is that the retrieval-based
CRS approaches actually surfaced these richer system
responses frequently.
BLEU Score Analysis Finally, in order to understand
to what extent (ofline) linguistic scores correlate with
the perceived quality of responses as was done in [
are the basis for many modern CRS. In this work, we have analyzed the recent INSPIRED dataset, which was developed to build the next generation of sociable CRS.
We found that automatic entity and concept labeling has its limitations and we have improved the quality of the dataset through a manual process. We then conducted both computational experiments as well as experiments with users to analyze to what extent improved data quality impacts recommendation accuracy and the quality perception of the system’s responses by users. The analyses clearly indicate the benefits of improved data quality across diferent technical approaches for building CRS.
We release the improved dataset publicly and hope to thereby stimulate more research in sociable conversational recommender systems in the future. prove conversational recommendation, in: ICCBR conversational recommendation, Information Sys’10, 2010, pp. 480–494. tems (2022) 102083. [31] L. Chen, P. Pu, Critiquing-based recommenders: [37] D. Jannach, Evaluating conversational recomsurvey and emerging trends, User Modeling and mender systems, Artificial Intelligence Review User-Adapted Interaction 22 (2012) 125–150. forthcoming (2022). [32] Q. Chen, J. Lin, Y. Zhang, H. Yang, J. Zhou, J. Tang, [38] T. Zhang, Y. Liu, P. Zhong, C. Zhang, H. Wang, Towards knowledge-based personalized product de- C. Miao, KECRS: Towards knowledge-enriched scription generation in e-commerce, in: KDD ’19, conversational recommendation system, 2021. 2019, pp. 3040–3050. a r X i v : 2 1 0 5 . 0 8 2 6 1 . [33] J. Devlin, M.-W. Chang, K. Lee, K. Toutanova, Bert: [39] Y. Zhou, K. Zhou, W. X. Zhao, C. Wang, P. Jiang, Pre-training of deep bidirectional transformers for H. Hu, C2-crs: Coarse-to-fine contrastive learnlanguage understanding, in: NAACL-HLT, 2019. ing for conversational recommender system, in: [34] L. Wang, H. Hu, L. Sha, C. Xu, K.-F. Wong, D. Jiang, WSDM ’22, 2022, pp. 1488–1496.
Finetuning large-scale pre-trained language models [40] Y. Li, B. Peng, Y. Shen, Y. Mao, L. Liden, Z. Yu, for conversational recommendation with knowl- J. Gao, Knowledge-grounded dialogue generation edge graph, 2021. a r X i v : 2 1 1 0 . 0 7 4 7 7 . with a unified knowledge representation, 2021. [35] A. Manzoor, D. Jannach, Conversational recom- a r X i v : 2 1 1 2 . 0 7 9 2 4 .
mendation based on end-to-end learning: How far [41] D. Jannach, A. Manzoor, End-to-end learning for are we?, Computers in Human Behavior Reports conversational recommendation: A long way to (2021) 100139. go?, in: IntRS Workshop at RecSys ’20, Online, [36] A. Manzoor, D. Jannach, Towards retrieval-based 2020.