Conversational Recommender Systems (CoRSs) implement a paradigm where users can interact with the system for denfiing their preferences and discovering items that best fit their needs. A CoRS can be straightforwardly implemented as a chatbot. Chatbots are becoming more and more popular for several applications like customer care, health care, medical diagnoses. In the most complex form, the implementation of a chatbot is a challenging task since it requires knowledge about natural language processing, human-computer interaction, and so on. In this paper, we propose a general framework for making easy the generation of conversational recommender systems. The framework, based on a contentbased recommendation algorithm, is independent from the domain. Indeed, it allows to build a conversational recommender system with diferent interaction modes (natural language, buttons, hybrid) for any domain. The framework has been evaluated on two state-of-the-art datasets with the aim of identifying the components that mainly influence the ifnal recommendation accuracy.
Conversational Recommender Systems (CoRSs) are
characterized by the capability of interacting with the user during
the recommendation process [
Users can provide functional requirements or technical constraints used by the recommender for finding the items that Knowledge-aware and Conversational Recommender Systems (KaRS) Workshop 2018 (co-located with RecSys 2018), October 7, 2018, Vancouver, Canada. 2018. ACM ISBN Copyright for the individual papers remains with the authors. Copying permitted for private and academic purposes. This volume is published and copyrighted by its editors.. best fit their needs. Accordingly, the acquisition of preferences is an incremental process that might not be necessarily ifnalized in a single step. CoRSs can provide several interaction modes and can ofer explanation mechanisms. Hence, the goal of these systems is not only to improve the accuracy of the recommendations, but also to provide an efective user-recommender interaction.
In this paper we propose a framework, not dependent on the domain, for generating conversational recommender systems. Our framework implements most of the capabilities that a recommender should ofer, such as preference acquisition, profile exploration, critiquing strategies, and explanation capability. Furthermore, it ofers three diferent interaction modes: natural language, buttons, and a combination of the two previous ones.
The most complex interaction mode is certainly the one based on natural language. Indeed, a conversational recommender based on natural language needs at least four components: an intent recognizer, an entity recognizer, a sentiment analyzer, and a recommendation algorithm. The next sections explain the tasks that each component carries out. Usually, the first three components are also used for purposes diferent from the recommendation task. For example, an entity recognizer is useful in several applications where the identification of named entities in a given text is needed, such as news classification, search algorithms, customer support. In this work we first generalized, combined, and integrated the aforementioned components in order to make easy the development of a new conversational recommender system, then we investigated the impact of each component on the recommendation process.
By exploiting our framework, we implemented instances of a conversational recommender system in three diferent domains: movies, music, and books 1.
The rest of the paper is organized as follows: the relevant literature is analyzed in Section 2; Section 3 describes the architecture of our framework, and finally, the experimental evaluation and the discussion of results are reported in Section 4. Section 5 draws the conclusion and the future work. 2
Conversational Recommender Systems fall in the area of
Goal-Oriented Dialog Systems. A Goal-Oriented Dialog
System, also known as Chatbot, is designed for helping users
to achieve a given goal (e.g. to book a restaurant). These
systems are generally closed-domain, thus can be exploited
in scenarios like recommendation [
There are several work in the literature that tried to
improve various aspects of the conversational recommendation
process [
A commercial solution is proposed by Microsoft with the Bot Framework3 that provides tools for building, connecting, testing, and deploying intelligent bots. Even though the framework is not designed for generating conversational recommender systems, it allows to integrate services from Microsoft Azure like the recommendation engine4. However, the efort for integrating and connecting the diferent components is borne by the user of the framework. Furthermore, the framework does not ofer features like critiquing strategies or explanation functions. Also, the diferent interaction modes (e.g., buttons) have to be implemented by the user. Last but not least, in this work we studied the accuracy of each component integrated in our framework. Conversely, other solutions can be used only as black box models.
3https://dev.botframework.com/ 4https://azure.microsoft.com/en-us/resources/videos/building-arecommender-system-in-azure-ml-studio/ 3
The architecture of our framework is depicted in Figure 1. The main goal of this framework is to make easy the building of a new CoRS. Therefore, the components have been generalized making them independent from a specific domain. When the user wants to build e new CoRS for a new domain she should update the configuration file, and provide the list of entities and properties in the Wikidata5 format. This requirement will be better explained in the next sections and depends on the Entity Recognizer. The interaction follows a slot-filling model, where a set of features need to be filled in order to accomplish the user goal. As an example, the recommendation step requires that the user preferences have been filled. In the following we analyze each component in detail.
Dialog Manager This is the core component of the framework whose responsibility is to supervise the whole recommendation process. The Dialog Manager (DM) is the component that keeps track of the dialog state. DM receives the user message, invokes the components needed for answering to the user request, and returns the message to be shown to the user. When all the information for fulfilling the user request are available, the Dialog Manager returns the message to the client. DM is completely independent from the client, indeed it receives a text message and returns a json message as a response. In this way the client can be any messaging platform like Facebook Messenger, Telegram, Web apps, and so on.
Intent Recognizer
This component has the goal of defining the intent of the user formulated by natural language. The Intent Recognizer (IR) is based on DialogFlow6 developed by Google. Our framework uses the DialogFlow APIs for sending the user message and to receive the intent recognized. DialogFlow requires a set of example sentences for each intent. There are four main intents to be recognized:
- preference: the user is providing a preference. The preference can be expressed on a new item, or on a recommended item. In the latter case, the preference is also considered as a critique (e.g., I like this movie, but I don’t its director).
- recommendation: the user asks to receive a recommendation. This intent is the condition for other sub-intents such as explanation where the user asks the motivation for a given recommendation, critiquing where the user expresses a critique on one or more features of the recommended item, more info where the user asks more details on the recommended item (e.g., the plot, the trailer).
- show profile: the user asks to visualize (and modify) her list of preferences.
- help: the user asks for help from the system to complete a given task.
Each intent can be composed of a set of sub-intents that activate specific functions. For example the intent profile has delete preference, update preference, reset profile as sub intents. The motivation behind this hierarchical organization is that, generally, the sub-intent can be activated only when the parent intent is activated too. The activation of a parent intent is managed by the Dialog Manager.
Sentiment Analyzer The Sentiment Analyzer (SA) is based on the Sentiment Tagger of Stanford CoreNLP7. The Sentiment Tagger takes as input the user sentence and returns the sentiment tags identified. Afterwards, SA assigns the sentiment tags to the right entity identified into the sentence. For example, given the sentence I like The Matrix, but I hate Keanu Reeves, the Sentiment Tagger identifies a positive sentiment (i.e. like) and a negative one (i.e. hate). SA associates the positive sentiment to the entity The Matrix and the negative sentiment to the entity Keanu Reeves. The association sentiment-entity is performed by computing the distance between the sentiment tag and the entity into the sentence. The distance is in terms of number of tokens that separate the sentiment tag from the entity. Given the aforementioned example, the distance between like and The Matrix is zero, as well as the distance between hate and Keanu Reeves. The sentiment tag identified by CoreNLP is thus associated to the closest entity in the sentence. Furthermore, SA implements a Property Type Recognizer that is able to identify property-mentions in the sentence. Given the sentence I like The Matrix, but I hate the director, SA identifies the property director and assigns the negative sentiment to it. Afterwards, SA will retrieve the entity associated to that property, Larry e Andy Wachowski in the given example. The list of properties the framework is able to recognize is provided in the configuration file.
Entity Recognizer
The aim of the Entity Recognizer (ER) module is to find relevant entities mentioned in the user sentence and then to link them to the correct concept in the Knowledge Base (KB). The KB chosen for building our framework is Wikidata since it is a free and open knowledge base and acts as a hub of several structured data coming from Wikimedia sister projects8. Moreover, Wikidata covers several domains and this is a key feature for developing a domain-independent framework. We choose to develop a custom entity recognizer 7https://stanfordnlp.github.io/CoreNLP/ 8Wikipedia, Wikivoyage, Wikisource, and others for two reasons: 1) existing entity recognizer/entity linking algorithms are hard to customize to a specific domain; 2) entity recognizers included in existing dialog manager toolkits generally require annotated data to build a new model for a specific domain, while we implemented a knowledge based approach that does not need any annotated data. The task is challenging since more than one surface form Spielberg, Steven Spielberg can refer to Steven Spielberg:director, and the same surface form Spielberg can refer to more than one concept Steven Spielberg:director and Sasha Spielberg:actor in case of ambiguous entities. Moreover, we need to limit the entities’ type according to the domain of the CoRS and the list of concepts and properties provided during the configuration of the framework.
In order to recognize the entities in the user request, we build a search engine based on a classical Vector Space Model in which for each entity we store all the possible alias provided by Wikidata. For example, for the concept Q8877 (Steven Spielberg), we store the alias Steven Allan Spielberg, Spielberg and Steven Spielberg. The index is exploited for retrieving a list of candidate concepts according to the input text. In particular, given a text as input, the ER module performs a chunking operation in order to identify nominal chunks by using the Apache OpenNLP library. Each nominal chunk is sent as a query to the search engine in order to retrieve a list of candidate concepts. The output of this first recognition step is a list of candidate concepts assigned to each nominal chunk. The list is sorted according to the score assigned by the search engine. We use Apache Lucene as library for implementing our search engine.
The last step consists in selecting the correct concept for each chunk. The idea is to choose the concept that is more similar to the other concepts occurring in the text following the hypothesis of one topic for discourse. The motivation behind this approach is that the user tends to cite in the same text entities that are in some way related. The score () assigned to each candidate concept for the chunk is computing according to the Equation 1, where is the set of the other nominal chunks in the text. The score () is the sum for each chunk in of the maximum similarity score between all the candidate concepts of and . () = ∑︁ ∈ (, ) ∈ (1)
In order to compute the score (), we need to define a
similarity function between concepts. In our approach we rely
on graph embeddings that have recently gained considerable
attention [
Recommendation Services This component collects
the services strictly related to the recommendation process.
The recommendation algorithm implemented is the
PageRank with Priors [
As before stated, all these components are independent from the domain. The only requirement is that the entities have to be available in Wikidata.
The goal of the experimental evaluation was to define the accuracy of each component involved in our framework. For this experiment we used the bAbI dataset developed by Facebook Research10. The dataset collects a list of utterances, and each utterance contains a list of preferences, followed by the recommendation request, and the recommended item. An example of utterance is: ”Beauty and the Beast, Aladdin, Schindler’s List, The Shawshank Redemption, and The Silence of the Lambs are movies
I loved. Would you recommend something I might like? Ghost”.
We used this dataset for testing the impact of the Entity Recognizer, the Sentiment Analyzer, and the Intent Recognizer on the recommendation process. Since the dataset has the goal of evaluating end-to-end dialog systems, the dataset is split in three sub sets: training, test, and dev set. Given the goal of our experiment, we excluded the training set due to its huge dimension, and we used the test, and dev sets composed respectively of 6,667 and 6,733 examples (each example contains the preference elicitation, the recommendation request, and the recommendation). We defined four diferent configurations of the framework:
- Upper bound (UB): this configuration tests the accuracy of our recommendation algorithm. The preferences and the recommendation requests are filled programmatically, so, except for the recommendation algorithm, the other components of the frameworks do not work.
- Intent Recognizer Test (IR): in this configuration the only component that works is the Intent Recognizer. The component detects the intention of the user of expressing a preference and receiving the recommendation. If both intents are correctly recognized, the recommendation is performed by setting the entities and their sentiments programmatically.
- Entity Recognizer Test (ER): in this configuration the only component that works is the Entity Recognizer. It detects the entities on which the user expressed a preference. The sentiment on the entities correctly identified are set programmatically.
- Sentiment Recognizer Test (SR): in this configuration the only component that works is the Sentiment Recognizer. The component detects the sentiments in the sentence. The entities on which the sentiments is expressed are set programmatically.
These configurations allow to test one component at time by excluding the influence of the other ones in the process. For each configuration we computed the HitRate@n as the ratio of the hits in the recommendation list with n= 5, 10, 20. In Table 1, the first row reports the upper bound in terms of HitRate@n: this is the best result that our recommendation algorithm can achieve on this dataset in the ideal situation where the other components work with a 100% of accuracy. The other rows report the loss in terms of HitRate@n of each configuration compared to the upper bound. Due to the space limit, we report only the results on the test set of bAbI, since the dev set follows the same trend.
It is worth to note that it is not surprising that the upper bound is very low. Indeed, in the bAbI dataset even though the top predictions contain items that might be good for the user, if the actual single true label is not recommended, the recommendation fails.
The analysis of the loss shows that the entity recognizer
(ER) is the component with the highest negative impact on
the recommendation process. Even though the entity
recognizer was able to recognize ∼ 85% of the entities on the bAbI
dataset, the error (∼ 15% entities not recognized) determined
a strong loss in terms of accuracy. The second component in
terms of negative impact on the recommendation accuracy
is the Intent Recognizer (IR). In this case, the component
was able to correctly recognize ∼ 77% of the intents (both
preference elicitation, and recommendation request). The
component with the lowest impact on the recommendation
accuracy is the Sentiment Recognizer (SR), in this case the
component was able to correctly recognize ∼ 83% of the
sentiments. By analyzing these results, it emerges that the
Entity Recognizer plays a crucial role in the
recommendation process of a conversational recommender. This aspect
is particularly crucial when a rich user profile is not
available, and the recommender has to work on a small number
of preferences as in the bAbI dataset. The second
component that negatively influences the recommendation is the
IR. Also in this case, if the CoRS is not able to correctly
identify the user intention, it will not be able to activate the
correct process to satisfy the request. Finally, the SR is the
component with the lowest negative impact. However, in the
bAbI dataset all the sentences have a positive sentiment, and
this facilitates the work of the component. We also tested our
framework on a dataset recently released by Grouplens [
UB IR ER SR
In this paper, we proposed a framework for building conversational content-based recommender systems in any domain. The only requirement to be satisfied is to provide a list of entities and properties in the Wikidata format. We will soon release the source code of the framework and make it available for the community. The preliminary experimental evaluation on two state-of-the-art datasets shows the impact of each component in a classical movie recommendation scenario. In this way, the user is aware of the limitations of the single modules implemented. In the next future, we plan to run an experimental evaluation on another synthetic dataset and to perform an in-vivo evaluation with real users on three diferent domains (movie, book, music) with the aim of investigating the impact of other capabilities (e.g. critiquing, explanation) on the recommendation process.
This work has been funded by the projects UNIFIED WEALTH MANAGEMENT PLATFORM - OBJECTWAY SpA - Via Giovanni Da Procida nr. 24, 20149 MILANO - c.f., P. IVA 07114250967, and PON01 00850 ASK-Health (Advanced system for the interpretations and sharing of knowledge in health care).