In this paper, we investigate the combination of Virtual Assistants and Conversational Recommender Systems (CoRSs) by designing and implementing a framework named ConveRSE, for building chatbots that can recommend items from diferent domains and interact with the user through natural language. An user experiment was carried out to understand how natural language influences both the cost of interaction and recommendation accuracy of a CoRS. Experimental results show that natural language can indeed improve user experience, but some critical aspects of the interaction should be mitigated appropriately.
presents a work that was previously published in Decision Support Systems [
A Conversational Recommender System (CoRS) is defined as a system that provides
recommendations to users via a multi-turn dialogue [
The idea of combining together Virtual Assistants and recommender systems has been
introduced in [
CoRSs have been developed using many diferent input and output modalities, such as forms
[
Interaction with ConveRSE is divided into three main steps, which are also found in [
The architecture of ConveRSE is shown in Figure 1. In particular, ConveRSE implements the following components:
Natural Language Understanding (NLU): It is in charge of understanding the user’s utterance. It performs three tasks: (i) intent recognition, which classifies the action or request expressed in the message (e.g. providing a preference, requesting a recommendation), (ii) entity recognition, which extracts items (e.g. movies, actors, directors) that are mentioned in the message, and (iii) sentiment analysis, which then assigns a sentiment score to each item. Intent
NLU Intent Recognition Entity Recognition
Sentiment Analysis Dialogue Manager Dialogue State Tracking
Dialogue Policy
Response Generation Response
Recommender System recognition is implemented using Google Dialogflow 1, Sentiment Analysis is performed using the Stanford CoreNLP2 Sentiment Tagger, while Entity Recognition is developed in-house.
Dialogue Manager (DM): It supervises the interaction process, by coordinating the activity of all other components. It performs three tasks: Dialogue State Tracking, which keeps track of all information exchanged with the user, and updates it accordingly, Dialogue Policy, which selects the best action to perform based on the current intent and the dialogue state (e.g. generate a recommendation, ask for clarification), and Response Generation, that generates a textual feedback by filling a template with contextual information. This component is developed in-house.
Recommender System: It handles all functions related to profile building and generation of suggestions. In particular, ConveRSE uses a graph-based recommendation algorithm based on the PageRank with Priors [18]. It exploits a knowledge graph extracted from Wikidata [19], in which both items and their properties are represented as nodes in the graph. This component is also responsible for generating explanations, exploiting the connections between the recommended item and the items in the user profile.
We created three instances of ConveRSE that are able to generate recommendations for diferent domains: movies, books and music, with respectively 15, 954, 7, 592 and 12, 926 recommendable items. Aside from this, we also implemented three diferent interaction modes for the CoRS: (i) Natural Language (NL), in which users express their preferences in the form of short text messages, and all the steps described in Section 3 are performed via text; (ii) Buttons, which uses a traditional system-driven approach for profile elicitation, in which the CoRS proposes a set of popular items, and users express a rating by pressing buttons; (iii) Mixed, which is an extension of the NL interface, in which users can answer certain system questions using buttons (e.g. 1https://dialogflow.cloud.google.com/ 2https://stanfordnlp.github.io/CoreNLP/ when multiple entities in the knowledge base match an item mentioned in the user utterance). Therefore, there are nine configurations in total.
We performed three within-subjects experiments (one for each domain), which involved 50 people for the movie domain, 55 for the book domain, and 54 for the music domain. The results of the experiment will answer the following Research Questions: RQ1: Can natural language improve a Conversational Recommender System in terms of cost of interaction?; RQ2: Can natural language improve a Conversational Recommender System in terms of quality of the recommendations?
During the experiment, participants were briefly instructed on how to use the system. Then, they performed all steps described in Section 3. After providing at least three preferences, they received a set of five recommended movies, each of which could be accepted, rejected, or more complex feedback could be provided. During the experiment, we collected several metrics related to the interaction cost and recommendation accuracy. For the interaction cost, we recorded the number of questions (NQ) asked by the system, the time needed to answer those questions (TPQ), the total interaction Time (IT), and the Query Density (QD) [20], which measures the average number of new concepts (i.e. entities) introduced in each utterance. For the recommendation quality we measured the Accuracy and the Mean Average Precision (MAP).
Results are shown in Table 1. We can observe that the NL and Mixed configurations ask a lower number of questions compared to the button-based one. Also, the NL configuration requires longer IT and TPQ, while the mixed configuration obtained the lowest values. This suggests that an interaction based entirely on natural language can become ineficient in specific cases, for example, when a disambiguation of the user input is required. Integrating a button-based interface for these cases leads to reduced typing time and less mistakes, which dramatically reduces the interaction cost. We also observe that the mixed interaction mode obtains the best recommendation accuracy results in all domains, which means that it allows users to express their preferences more efectively, thus improving the quality of the suggestions.
In this paper, we presented an experimental study on the efect of introducing natural language interaction into a CoRS. Although a dialogue in natural language has the potential to improve interaction cost and quality of recommendations of a CoRS, a purely NL-based interface poses some issues that need to be addressed. Specifically, when the user has to choose among a set of possible options, the integration of buttons drastically improves user experience. Modeling and User-Adapted Interaction 22 (2012) 125–150. URL: http://link.springer.com/ 10.1007/s11257-011-9108-6. doi:10.1007/s11257-011-9108-6. [16] G. Wu, K. Luo, S. Sanner, H. Soh, Deep Language-based Critiquing for Recommender Systems, in: Proceedings of the 13th ACM Conference on Recommender Systems, RecSys ’19, ACM, New York, NY, USA, 2019, pp. 137–145. URL: http://doi.acm.org/10.1145/3298689. 3347009. doi:10.1145/3298689.3347009, event-place: Copenhagen, Denmark. [17] K. Christakopoulou, F. Radlinski, K. Hofmann, Towards Conversational Recommender Systems, in: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining - KDD ’16, ACM Press, San Francisco, California, USA, 2016, pp. 815–824. URL: http://dl.acm.org/citation.cfm?doid=2939672.2939746. doi:10.1145/2939672.2939746. [18] T. H. Haveliwala, Topic-sensitive pagerank: A context-sensitive ranking algorithm for web search, IEEE transactions on knowledge and data engineering 15 (2003) 784–796. [19] D. Vrandečić, M. Krötzsch, Wikidata: a free collaborative knowledgebase, Communications of the ACM 57 (2014) 78–85. Publisher: ACM New York, NY, USA. [20] J. Glass, J. Polifroni, S. Senef, V. Zue, Data collection and performance evaluation of spoken dialogue systems: The mit experience, in: Sixth International Conference on Spoken Language Processing, 2000.