This paper reports and summarizes the methodology presented in [16] and accepted for publication at ACM RecSys 20191. In this work we present a methodology to justify recommendations that relies on the information extracted from users' reviews discussing the available items. The intuition behind the approach is to conceive the justification as a summary of the most relevant and distinguishing aspects of the item, automatically obtained by analyzing its reviews. To this end, we designed a pipeline of natural language processing techniques including aspect extraction, sentiment analysis and text summarization to gather the reviews, process the relevant excerpts, and generate a unique synthesis presenting the main characteristics of the item. Such a summary is finally presented to the target user as a justification of the received recommendation. In the experimental evaluation we carried out a user study in the movie domain (N=141) and the results showed that our approach is able to make the recommendation process more transparent, engaging and trustful for the users. Moreover, the proposed method also beat another review-based explanation technique, thus confirming the validity of our intuition.
A research line which is recently emerging in the context of the so-called Data
Explosion [
Copyright c 2019 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
However, differently from most of the current literature, in this work we
investigated the impact of the information conveyed by users’ reviews in a different
scenario, that is to say, the task of justifying a suggestion returned by a
recommendation algorithm. This topic is particularly relevant due to the recent
regulations in the area, as the General Data Protection Regulation3 (GDPR), which
emphasized the users’ right to explanation. Moreover, many studies showed that
a higher transparency leads to a higher trust [
In a nutshell, our methodology relies on the idea that an effective justification
can be conceived as a summary of relevant and distinguishing aspects of the item,
automatically obtained by analyzing its reviews. To this end, we designed a
pipeline of natural language processing techniques to obtain and process reviews
excerpts and to generate a unique synthesis presenting the main characteristics
of the item, which is finally presented to the target user as justification of the
received recommendation. It is worth to note that we preferred to use the term
justification over explanation, even if they are often used as synonyms. This
choice follows the definition provided in [
For a discussion of the literature in the area we suggest to refer to the original
paper [
planations; – The use of aspect extraction methods to automatically discover relevant and distinguishing aspects of the item, discussed in user reviews.
In the following, we describe the main building blocks of our methodology and we will show the results emerging from the user study in the movie domain.
Finally, we will discuss the main outcomes of this work, by also sketching some ideas for future research. 2
Our methodology to build natural language justifications by exploiting users’ reviews relies on four steps: aspect extraction, aspect ranking, sentence filtering and text summarization.
First, we assume that an effective justification should include relevant and distinguishing traits of the items, which are often discussed in the reviews (with a positive sentiment, of course). Accordingly, the goal of the first phase is identifying the aspects that are worth to be included in the justification. Formally, our strategy takes as input a set of reviews R = fr1; r2 : : : rng and produces a set of 4-tuples hri; aij ; rel(aij ; ri); sent(aij ; ri)i representing the review ri, the j-th aspect aij extracted from the review ri, the relevance rel(aij ; ri) of the aspect aij in the review ri, and the sentiment associated to that aspect.
To extract aspects from reviews, we used the Kullback-Leibler divergence [
Where p(t; ca) is the number of occurrences of the term t in corpus ca. In a nutshell, this measure rely on the idea that the use of language differs when talking about a specific domain with respect to a general topic, thus this method identifies the aspects whose distribution in a specific domain (e.g., movie reviews ) diverges from that in a general corpus (e.g., the British National Corpus BNC4). In other terms, the measure identifies those aspects which are mentioned in the reviews more often than usual. Given such a formulation, our strategy for identifying the main aspects mentioned in the reviews follows: Require: review ri, general corpus BN C, domain corpus d Ensure: set of main aspects A and relevance scores
A = fg , T = nouns(ri)andentities(ri) for all tk 2 T do if tk (djjBN C) > then aij tk A = A S faij g rel(aij ; ri) tk end if end for
In our case, given a review ri, we first extract all the nouns by running a POS-tagging algorithm. Next, we calculate KL-divergence for each noun by using as corpora our set of movie reviews and the BNC. The nouns having a KL-divergence greater then a threshold are labeled as aspects and the KLdivergence score is used as relevance score. Finally, we also run a sentiment analysis algorithm to identify the opinion is associated to the aspect aij in ri and we stored that value. More details on the algorithms used in this phase are provided in the next Section.
Aspect extraction is run over all the reviews contained in R, and a set of aspects is extracted from each review. However, the goal of our strategy is to identify the most relevant aspects that overall describe the item, thus we also designed an aspect ranking phase to merge the information we extracted from each single review. Specifically, for each aspect aj we calculate its global score as follows: score(aj ) =
PN i=1 naj;ri rel(aj ; ri) sent(aj ; ri)
N (2)
Our formula gives a higher score to the aspects that are often mentioned in the reviews (naj;ri is the number of the occurrences of aj in ri) with a positive sentiment. At the end of this step all the aspects are ranked and the top-K are labeled as main aspects. Typically, the aspects returned by our strategy include concepts such as actors, director, story, music and so on.
The Top-5 Aspects returned by our algorithm for three different movies are reported in Table 1. Such an output emphasizes how our strategy can make relevant and interesting aspects of the items immediately emerge, since different and distinguishing elements of the movies are highlighted.
Chinatown The Ring Titanic Once the main aspects are identified, we run a sentence filtering phase. The goal of this phase is to filter out non-compliant sentences that are supposed to be not useful in the final justification we want to build. In this case we first split each review ri 2 R in sentences si1 : : : sim. Next, we verify the compliancy of each sentence si and we maintain only the sentences matching the following criteria: (i) si contains a main aspect a1 : : : ak; (ii) si is longer than 5 tokens; (iii) si expresses a positive sentiment; (iv) si does not contain first-person personal or possessive pronouns.
The rationale behind these heuristics is straightforward: we want to include in the justification only the sentences including a relevant aspect that also express a positive sentiment about the item. Moreover, we decided to filter out very short and non-informative sentences as well as those using the first person. In this case, the intuition is to prefer excerpts having a more impersonal style (e.g., ’The movie has a great cast’ ) than those expressing personal opinions (e.g., I liked the cast of the movie, this is my favourite director ). 2.4
At the end of the Sentence Filtering a set of potential candidate sentences is obtained. Such a set is used to feed a text summarization algorithm whose goal is to generate a unique summary to be used as justification of the recommendation. Such a summary is supposed to highlight the main contents discussed in the reviews of the item and to maximize both the coverage and diversity of the justification. This is done by selecting sentences which cover enough amount of topics discussed in the original reviews and by avoiding the redundancy as well.
To run text summarization we adapted the method described in [
Our approach combines centroid-based text summarization [
Formally, given a set of reviews R and its vocabulary V with size N = jV j, we first define a matrix M 2 RN;k, so-called lookup table, where the i-th row is a word embedding of size k, k << N , of the i-th word in V . The values of the word embeddings matrix M are learned by using Word2Vec. When the lookup table is learned, the summarization method consists of three phases:
Centroid Vector Building. The centroid vector that represents the meaning of the reviews is built in two steps. First, the most meaningful words occurring in the reviews (that is to say, those having their tf idf weight greater than a topic threshold) are selected. Next, the embedding of the centroid is computed as the sum of the embeddings of the top ranked words in the reviews using the lookup table M .
C =
X w2R;tfidf(w)>t
M [idx(w)] (3) In the eq. (3) we denote with C the centroid embedding related to the set of reviews R and with idx(w) a function that returns the index of the word w in the vocabulary.
Sentence Scoring. Given the centroid vector, we need to identify the sentences to be included in the summary. For each sentence in the set of the reviews, we create an embedding representation by summing the vectors for each word in the sentence stored in the lookup table M .
Sj =
X M [idx(w)] w2Sj sim(C; Sj ) =
CT
Sj jjCjj jjSj jj In the eq. (4) we denote with Sj the j-th sentence in the set of reviews R. Then, the sentence score is computed as the cosine similarity between the embedding of the sentence Sj and that of the centroid C of the set R.
Sentence Selection. The sentences are sorted in descending order of their similarity scores. The top ranked sentences are iteratively selected and added to the summary until the limit, in terms of the number of words in the summary, is reached. In order to minimize the redundancy of the information included in the summary, during the iteration we compute the cosine similarity between the next sentence and each one already in the summary. We discard the incoming sentence if the similarity value is greater than a threshold.
At the end of this process a summary is produced. An example of the summaries generated for three movies is reported in Table 2. These summaries were used in our experiments as justifications. 3
The goal of the experimental session was twofold: to evaluate the effectiveness of different configurations of our justifications based on text summarization (Experiment 1), and to compare our methodology to a baseline that exploits users’ reviews without text summarization (Experiment 2).
To this end, we designed a user study in the movie domain involving 141
subjects. To run the experiment, we deployed a web application5. As regards
Experiment 1, we run the experiment in a between-subject fashion, that is to
say, each user was randomly assigned to a configuration of our pipeline, and
he evaluated the justifications. Clearly, the user was not aware of the specific
configuration he was interacting with. Conversely, for Experiment 2, we run a
within-subject experiment, that is to say, all the users were provided with two
different explanation styles (i.e., our justifications exploiting text summarization
and a review-based baseline presented in [
To run the experiment we built a dataset by mapping MovieLens data with Amazon reviews discussing the movies. The resulting dataset contained 307 movies and 153,566 reviews. The average length of each review was 138.38 words. 5 http://193.204.187.192:8080/WebLodrecsys_AES (4) (5) Movie
Justification
In order to evaluate different configurations of our pipeline, we compared six different different alternatives of the algorithm, obtained by varying number of aspects and justification length. As for the length, we compared long and short justifications, depending on the overall length of the justification (50 words for short justifications, 100 words for long justifications). As for the number of aspects, we compared the justifications built by including all the aspects mentioned in the reviews, the top-10 aspects and the top-30 aspects.
Implementation details. Recommendations were generated by using
Personalized PageRank (PR) [
In a nutshell, each user involved in the experiment carried out the following steps:
(1) Preference Elicitation and Generation of the Justifications. We asked users to explicitly rate at least three items, chosen among a randomly generated subset of 20 movies. Next, recommendations were generated.
(2) Between-subject Evaluation through Questionnaires. We asked the users to fill in a questionnaire to evaluate the quality of the justification.
Each user was asked to evaluate the previously presented metrics through a five-point scale (1=strongly disagree, 5=strongly agree).
(3) Within-subject Evaluation through Questionnaires. We asked each user to evaluate the explanation style they preferred between our methodology and the baseline. As shown in Figure 1, we provided the user with both the explanations in the same screen, and we asked them to select the best one.
Finally, as for the evaluation metrics we used transparency, persuasiveness,
engagement and trust of the recommendation as the average score collected
through the user questionnaires, as in [
The results we obtained for Experiment 1 are reported in Table 3. The values represent the average scores given by the users for that specific metric. The higher the better, with the exception of the effectiveness where the best configuration is the one closer to zero.
Configuration
Metrics Aspects Length TRA PER ENG TRU EFF
All Short Top-10 Short Top-30 Short 2.89 2.83 3.16 2.74 3.06 3.06 3.26 3.06 2.69 2.93 2.83 3.19 0.48 0.89 0.94
The first result emerging from the experiment is that longer justifications tend to beat their shorter counterpart. This means that very short justifications cannot convey the information which is needed by the final users to better understand the quality of the suggestions they received. Such a low quality is also confirmed by analyzing the scores we obtained for short justifications : they are often below 3.00 (equivalent to ’partially agree’ as answer), which is considered as the minimum score that makes a justification as acceptable. Conversely, when longer justifications are produced by our technique, the results we obtained are generally higher. This means that the addition of more sentences can make the justifications and the recommendation process more satisfying, engaging, transparent and trustful for the target users.
As regards the number of aspects to be included, the experiment showed that the best results are obtained by exploiting the top-10 aspects identified by our aspect ranking module. This is confirmed for all the metrics except of the effectiveness. This means that by selecting a subset of relevant aspects we can provide the user with a summary containing the most relevant and useful information to evaluate the quality of the suggestion. Conversely, when a higher number of aspects is exploited (or even when all the aspects are took into account), some noise is probably introduced in the summarization process, thus non-relevant or non-interesting sentences are put in the final justification, and this leads to a decrease in the overall results.
Next, in Experiment 2 we compared our justification based on automatic
text summarization to the justifications generated by exploiting a review-based
baseline in a within-subject experiment. Due to space reasons, we only report the
results of the comparison between the best-performing configurations emerging
from Experiment 1 and the baseline. Specifically, we used long justifications based
on top-10 aspects. In a nutshell, our baseline identifies the most relevant aspects
by using our Aspect Extraction and Aspect Ranking strategies and randomly
selects a sentence expressing a positive sentiment that contains that aspects. For
more details on the baseline we suggest to refer to our previous work [
Movies
Aspects+Summar. Aspects Indifferent
As shown in Table 4, most of the users indicated that they preferred our methodology based on automatic text summarization. It is worth to note that we obtained the higher results for persuasion and engagement. This is a very encouraging outcome that confirmed the intuition behind this work, since the exploitation of text summarization can help to make interesting information about the recommended item emerge, and this can persuade the user in enjoying the item or can let the user discover new information about the item itself. 4
Overall, we can state the results emerging from these experiments confirmed the effectiveness of the approach, and showed that automatic text summarization can be useful to identify the most relevant aspects of the items and support the suggestions generated by a recommendation algorithm.
As future work we will introduce some strategy to automatically tune the
parameters of the model, in order to dynamically select the optimal number
of aspects and sentences to be included. Moreover, we will also evaluate more
sophisticated techniques for natural language processing, in order to include
also entities and bigrams in our justifications, and we will investigate how the
effectiveness of the justifications changes on varying of the different algorithms
used to generate the recommendation, as those based on Linked Open Data [