In this paper1 we present a methodology to generate context-aware natural language justifications supporting the suggestions produced by a recommendation algorithm. Our approach relies on a natural language processing pipeline that exploits distributional semantics models to identify the most relevant aspects for each different context of consumption of the item. Next, these aspects are used to identify the most suitable pieces of information to be combined in a natural language justification. As information source, we used a corpus of reviews. Accordingly, our justifications are based on a combination of reviews' excerpts that discuss the aspects that are particularly relevant for a certain context. In the experimental evaluation, we carried out a user study in the movies domain in order to investigate the validity of the idea of adapting the justifications to the different contexts of usage. As shown by the results, all these claims were supported by the data we collected.
Recommender Systems (RSs)
1Copyright ©2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
To this end, several attempts have been
recently devoted to investigate how to introduce
explanation facilities in RSs
In this paper we fill in this gap by proposing
an approach to generate a context-aware
justification that supports a recommendation. Our
methodology exploits distributional semantics models
To sum up, we can summarize the contributions of the article as follows: (i) we propose a methodology based on distributional semantics models and natural language processing to automatically learn a vector space representation of the different contexts in which an item can be consumed; (ii) We design a pipeline that exploits distributional semantics models to generate context-aware natural language justifications supporting the suggestions returned by any recommendation algorithm;
The rest of the paper is organized as follows: first, in Section 2 we provide an overview of related work. Next, Section 3 describes the main components of our workflow and Section 4 discusses the outcomes of the experimental evaluation. Finally, conclusions and future work of the current research are provided in Section 5. 2
The current research borrows concepts from review-based explanation strategies and distributional semantics models. In the following, we will try to discuss relevant related work and to emphasize the hallmarks of our methodology.
Review-based Explanations. According to
the taxonomy discussed in
Our workflow to generate context-aware justifications based on users’ reviews is shown in Figure 1. In the following, we will describe all the modules that compose the workflow.
Context Learner. The first step is carried out by the CONTEXT LEARNER module, which exploits DSMs to learn a vector space representation of the contexts. Formally, given a set reviews R and a set of k contextual settings C = fc1 : : : ckg, this module generates as output a matrix Cn;k that encodes the importance of each term ti in each contextual setting c j. In order to build such a representation, we first split all the reviews r 2 R in sentences. Next, let S be the set of previously obtained sentences, we manually annotated a subset of these sentences in order to obtain a set S0 = fs1 : : : smg, where each si is labeled with one or more contextual settings, based on the concepts mentioned in the review. Of course, each si can be annotated with more than one context. As an example, a review including the sentence ’a very romantic movie’ is annotated with the contexts company=partner, while the sentence ’perfect for a night at home’ is annotated with the contexts day=weekday. After the annotation step, a sentence-context matrix Am;k is built, where each asi;cj is equal to 1 if the sentence si is annotated with the context c j (that is to say, it mentions concepts that are relevant for that context), 0 otherwise.
Next, we run tokenization and lemmatization
algorithms (Manning et al., 1999) over the sentences
in S to obtain a lemma-sentence matrix Vn;m. In
this case, vti;sj is equal to the TF/IDF of the term
ti in the sentence s j. Of course, IDF is
calculated over all the annotated sentences. In order
to filter out non-relevant lemmas, we maintained
in the matrix V just nouns and adjectives. Nouns
were chosen due to previous research
In the last step of the process annotation matrix An;k and vocabulary matrix Vm;n are multiplied to obtain our lemma-context matrix Cn;k, which represents the final output returned by the CONTEXT LEARNER module. Of course, each ci; j encodes the importance of term ti in the context c j. The whole process carried out by this component is described in Figure 2.
Given such a representation, two different outputs are obtained. First, we can directly extract column vectors ~cj from matrix C, which represents the vector space representation of the context c j based on DSMs. It should be pointed out that such a representation perfectly fits the principles of DSMs since contexts discussed through the same lemmas will share a very similar vector space representation. Conversely, a poor overlap will result in very different vectors. Moreover, for each column, lemmas may be ranked and those having the highest TF-IDF scores may be extracted. In this way, we obtain a lexicon of lemmas that are relevant for a particular contextual setting, and this can be useful to empirically validate the effectiveness of the approach. In Table 1, we anticipate some details of our experimental session and we report the top-3 lemmas for two different contextual settings starting from a set of movie reviews.
Ranker. Given a recommended item (along
with its reviews) and given the context in which
the item will be consumed (from now on, defined
as ’current context’), this module has to identify
the most relevant review excerpts to be included
in the justification. To this end, we designed a
ranking strategy that exploits DSMs and similarity
measures in vector spaces to identify suitable
excerpts: given a set of n reviews discussing the item
i, Ri = fri;1 : : : ri;ng, we first split each ri in
sentences. Next, we processed the sentences through
a sentiment analysis algorithm
Vn;m a1;2 : : : a1;k 1 a2;2 : : :
::: ::: am;1 am;2 : : : am;k
Am;k
0c1;1 c1;2 : : : c1;k1 a2:::;k CCCA = BB@ ::: ::: ::: ::: CCA
Bc2;1 c2;2 : : : c2;kC cn;1 cn;2 : : : cn;k
Cn;k space representation of each sentence ~si. The sentences having the highest cosine similarity w.r.t. to the context of usage c j are selected as the most suitable excerpts and are passed to the GENERATOR.
Generator. Finally, the goal of GENERATOR is
to put together the compliant excerpts in a single
natural language justification. In particular, we
defined a slot-filling strategy based on the
principles of Natural Language Generation
The experimental evaluation was designed to
identify the best-performing configuration of our
strategy, on varying of different combinations of the
parameters of the workflow (Research Question
1), and to assess how our approach performs
in comparison to other methods (both
contextaware and non-contextual) to generate post-hoc
justifications (Research Question 2). To this
end, we designed a user study involving 273
subjects (male=50%, degree or PhD=26.04%,
age 35=49,48%, already used a RS=85.4%) in
the movies domain. Interest in movies was
indicated as medium or high by 62.78% of the
sample. Our sample was obtained through the
availability sampling strategy, and it includes students,
researchers in the area and people not skilled with
computer science and recommender systems. As
in
Experimental Design. To run the experiment, we deployed a web application2 implementing the methodology described in Section 3. Next, as a first step, we identified the relevant contextual dimensions for each domain. Contexts were selected by carrying out an analysis of related work of context-aware recommender systems in the MOVIE domain. In total, we defined 3 contextual dimensions, that is to say, mood (great, normal), company (family, friends, partner) and level of attention (high, low). To collect the data necessary to feed our web application, we selected a subset of 300 popular movies (according to IMDB data) discussed in more than 50 reviews in the Amazon Reviews dataset 3. This choice is motivated by our need of a large set of sentences discussing the item in each contextual setting. These data were processed by exploiting lemmatization, POS-tagging and sentiment analysis algorithms available in CoreNLP4 and Stanford Sentiment Analysis algorithm5. tool. Some statistics about the final dataset are provided in Table 2http://193.204.187.192:8080/filmando-eng 3http://jmcauley.ucsd.edu/data/amazon/ links.html - Only the reviews available in the ’Movies and TV’ category were downloaded.
4https://stanfordnlp.github.io/CoreNLP/ 5https://nlp.stanford.edu/sentiment/ 3.
In order to compare different configurations of
the workflow, we designed several variant
obtained by varying the vocabulary of lemmas. In
particular, we compared the effectiveness of
simple unigrams, of bigrams and their merge. In the
first case, we encoded in our matrix just single
lemmas (e.g., service, meal, romantic, etc.) while
in the second we stored combination of nouns and
adjectives (e.g., romantic location). Due to space
reasons, we can’t provide more details about the
lexicons we learnt, and we suggest to refer again
to Table 1 for a qualitative evaluation of some of
the resulting representations. Our representations
based on DSMs were obtained by starting from a
set of 1,905 annotations for the movie domain,
annotated by three annotators by adopting a majority
vote strategy. To conclude, each user involved in
the experiment carried out the following steps:
1. Training, Context Selection and Generation
of the Recommendation. First, we asked
the users to provide some basic demographic
data and to indicate their interest in movies.
Next, each user indicated the context of
consumption of the recommendation, by
selecting a context among the different
contextual settings we previously indicated (see
Figure 3-a). Given the current context, a
suitable recommendation was identified and
presented to the user. As recommendation
algorithm we used a content-based
recommendation strategy exploiting users’ reviews.
2. Generation of the Justification. Given the
recommendation and the current context of
consumption, we run our pipeline to
generate a context-aware justification of the item
adapted to that context. In this case, we
designed a between-subject protocol. In
particular, each user was randomly assigned to one
of the three configurations of our pipeline and
the output was presented to the user along
with the recommendation (see Figure 3-b).
Clearly, the user was not aware of the specific
configuration he was interacting with.
3. Evaluation through Questionnaires. Once
the justification was shown, we asked the
users to fill in a post-usage
questionnaire. Each user was asked to evaluate
transparency, persuasiveness, engagement
and trust of the recommendation process
through a five-point scale (1=strongly
disagree, 5=strongly agree). The questions the
users had to answer follow those proposed in
experiment, that allows to answer to Research Question 1, are presented in Table 4. The values in the tables represent the average scores provided by the users for each of the previously mentioned questions. As for the movie domain, results show that the overall best results are obtained by us#Items #Reviews #Sentences #Positive Sent. Avg. Sent./Item 307 153,398 1,464,593 560,817 4,770.66 Avg. Pos. Sent./Item 1,826.76 ing a vocabulary based on unigrams and bigrams.
This first finding provides us with an interesting outcome, since most of the strategies to generate explanations are currently based on single keywords and aspects. Conversely, our experiment showed that both adjectives as well as couples of co-occurring terms are worth to be encoded, since they catch more fine-grained characteristics of the item that are relevant in a particular contextual setting. Overall, the results we obtained confirmed the validity of the approach. Beyond the increase in TRANSPARENCY, high evaluations were also noted for PERSUASION and ENGAGEMENT metrics. This outcome confirms how the identification of relevant reviews’ excerpts can lead to satisfying justifications. Indeed, differently from feature-based justifications, that typically rely on very popular and well-known characteristics of the movie, as the actors or the director, more specific aspects of the items emerge from users’ reviews.
Next, in order to answer to Research Question 2, we compared the best-performing configurations emerging from Experiment 1 to two different baselines. The results of these experiments are reported in Table 5 which show the percentage of users who preferred our context-aware methodology based on DSMs to both the baselines. In particular, the first comparison allowed us to assess the effectiveness of a vector space representation of contexts based on DSMs with respect to a simple context-aware justification method based on a fixed lexicon of relevant terms, while the second comparison investigated how valid was the idea of diversifying the justifications based on the different contextual settings in which the items is consumed. As shown in the table, our approach was the preferred one in both the comparisons. It should be pointed out that the gaps are particularly large when our methodology is compared to a noncontextual baseline. In this case, we noted a statistically significant gap (p 0:05) for all the metrics, with the exception of trust. This suggests that diversifying the justifications based on the context of consumption is particularly appreciated by the users. This confirms the validity of our intuition, which led to a completely new research direction in the area of justifications for recommender systems. 5
In this paper we presented a methodology that exploits DSMs to build post-hoc context-aware natural language justifications supporting the suggestions generated by a RS. The hallmark of this work is the diversification of the justifications based on the different contextual settings in which the items will be consumed, which is a new research direction in the area. As shown in our experiments, our justifications were largely preferred by users. This confirms the effectiveness of our approach and paves the way to several future research directions, such as the definition of personalized justiMetrics / Configuration
Transparency Persuasion Engagement
Trust
fication as well as the generation of hybrid
justifications that combine elements gathered from
usergenerated content (as the reviews) with descriptive
characteristics of the items. Finally, we will also
evaluate the use of ontologies and rules