We present explanation rules, which provide explanations of user-based collaborative recommendations but in a form that is familiar from item-based collaborative recommendations; for example, \People who liked Toy Story also like Finding Nemo". We present an algorithm for computing explanation rules. We report the results of a web-based user trial that gives a preliminary evaluation of the perceived effectiveness of explanation rules. In particular, we nd that nearly 50% of participants found this style of explanation to be helpful, and nearly 80% of participants who expressed a preference found explanation rules to be more helpful than similar rules that were closely-related but partly-random.
An explanation of a recommendation is any content,
additional to the recommendation itself, that is presented to
the user with one or more of the following goals: to reveal
how the system works (transparency), to reveal the data it
has used (scrutability), to increase con dence in the system
(trust), to convince the user to accept the recommendation
(persuasion), to help the user make a good decision (e
ectiveness), to help the user make a decision more quickly
(e ciency), or to increase enjoyment in use of the system
(satisfaction) [
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee.
IntRS Workshop: Interfaces and Human Decision Making for Recommender Systems at RecSys’14 October 6–10, Foster City, CA, USA Copyright 20XX ACM X-XXXXX-XX-X/XX/XX ...$15.00.
The problem that we examine in this paper is how to
produce e ective explanations of user-based collaborative
recommendations. It is relatively easy to explain the
recommendations of content-based recommenders, e.g. by
displaying meta-descriptions (such as features or tags) that the
active user's pro le and the recommended item have in
common [
It is possible to explain a recommendation using data
other than that which the recommender used to generate
the recommendation [
We propose an algorithm for making item-based
explanations, also referred to as in uence-style explanations [
Fargo 4 5
We use a conventional user-based collaborative
recommender of the kind described in [
Before presenting the explanation algorithm, we de ne some terms: Explanation partner: The explanation partner is the member of the set of nearest neighbours who is most similar to the active user and who likes the recommended item. Often this will be the user who is most similar to the active user | but not always. In some cases, the most similar user may not have liked the recommended item: the recommendation may be due to the votes of other neighbours. In these cases, one of these other neighbours will be the explanation partner. It may appear that recommendations exploit the opinions of a set of neighbours (for accuracy), but explanations exploit the opinions of just one of these neighbours, the explanation partner. But this is not completely true. As we will explain below, the items included in the explanation are always members of the explanation partner's pro le, but they are also validated by looking at the opinions of all other users (see the notions of coverage and accuracy below).
Candidate explanation conditions: Let u be the active user and v be the explanation partner; let j be a corated item; and let ruj and rvj be their ratings for j. We de ne candidate explanation conditions as co-rated items j on which the two users agree.
In the case of numeric ratings, we do not insist on rating equality for there to be agreement. Rather, we de ne agreement in terms of liking, indi erence and disliking. For a 5-point rating scale, the candidate explanation conditions would be de ned as follows: candidates(u; v) = flikes(j) : ruj > 3 ^ rvj > 3g [ findi (j) : ruj = 3 ^ rvj = 3g [ fdislikes(j) : ruj < 3 ^ rvj < 3g For example, the candidate explanation conditions for users Ann and Bob in Table 1 are
flikes(Brazil ); dislikes(Dumbo); likes(Fargo)g Alien does not appear in a candidate condition because Ann's and Bob's ratings for it disagree; Crash and E.T. do not appear in candidate conditions because neither of them is co-rated by Ann and Bob. Input: user pro les U , recommended item i, active user u, explanation partner v Output: an explanation rule for i R if then i; Cs candidates(u; v); while accuracy(R) < 100 ^ Cs 6= f g do
Rs the set of all new rules formed by adding singly each candidate condition in Cs to the antecedent of R; R most accurate rule in Rs, using rule coverage to break ties between equally accurate rules; if accuracy(R ) accuracy(R) then
return R; R R ; Remove from Cs the candidate condition that was used to create R; return R;
Rule coverage: A rule covers a user if and only if the rule antecedent is satis ed by the user's pro le. For example, the rule in Figure 1 covers any user u whose pro le contains ratings ru;TheShining > 3 and ru;Frequency > 3, irrespective of what else it contains. Rule coverage is then the percentage of users that the rule covers. Rule accuracy: A rule is accurate for a user if and only if the rule covers the user and the rule consequent is also satis ed by the user's pro le. For example, the rule in Figure 1 is accurate for any user u whose pro le additionally contains ru;TheSilenceoftheLambs > 3. Rule accuracy is then the percentage of covered users other than the active user for whom the rule is accurate.
The algorithm for building an explanation rule works incrementally and in a greedy fashion; see Algorithm 1 for pseudocode. Initially, the rule has an empty antecedent, and a consequent that contains the recommended item i, written as `if then i' in Algorithm 1. On each iteration, the antecedent is re ned by conjoining one of the candidate explanation conditions, speci cally the one that leads to the most accurate new rule, resolving ties in favour of coverage. This continues until either the rule is 100% accurate or no candidate explanation conditions remain. 3.
We tested three hypotheses, the rst using an o ine experiment, the other two using a web-based user trial. 3.1
The number of candidate explanation conditions can be quite large. If explanation rules are to be practicable, then the number of conditions that the algorithm includes in the antecedent of each explanation rule needs to be quite small. Hypothesis 1: that explanation rules will be short enough to be practicable.
We ran the user-based collaborative recommender that we described at the start of the previous section on the MovieLens 100k dataset, and obtained its top recommendation for each user in the dataset. We then ran the explanation algorithm to produce an explanation rule that would explain the recommended item to that user. In Figure 2, we plot the number of candidate explanation conditions (vertical axis) against the number of these conditions that the algorithm includes in the rule (horizontal axis).
From the Figure, we see that the longest rules contained only three items in their antecedents. Not only that, but actually only 4% of the rules had three items in their antecedents; the other 96% were split nearly evenly between those having one and those having two items. We also see that the more candidates there are, the shorter the explanation rule tends to be. We have not investigated the exact reasons for this.
We repeated this experiment using a dataset with unary ratings to see what di erence this might make. We took a LastFM dataset that contains artist play counts for 360 thousand users and 190 thousand artists.1. We converted play counts to unary ratings, i.e. recording 1 if and only if a user has played something by an artist. The results were very similar to those in Figure 2 (which is why we do not show them here), again with no rule having more than three items in its antecedent.
These are encouraging results for the practicability of explanation rules. 3.2
We designed a web-based user trial, partly inspired by the
experiment reported in [
if there is a movie worth seeing. You use an online movie recommender to help you decide. The movie recommender recommends one movie and provides an explanation.
First, we sought to elicit the perceived e ectiveness of this style of explanation with the following hypothesis: Hypothesis 2: that users would nd explanation rules to be an e ective style of explanation.
We showed participants an explanation rule for a
recommendation and we asked them to rate its helpfulness on a
5-point scale. Speci cally, we asked \Would this style of
explanation help you make a decision?" with options Very
unhelpful, Unhelpful, Neutral, Helpful, and Very helpful. Our
1mtg.upf.edu/node/1671
wording di ers from that used by [
To encourage participants to focus on explanation style,
we followed [
To obtain a `yardstick', we also showed participants
another explanation and asked them whether it too was
helpful. For this purpose, we used the most persuasive
explanation style from [
In the experiment, the software randomly decides the order in which it shows the two explanation styles. Approximately 50% of participants see and rate the explanation rule before seeing and rating the histogram, and the remainder see and rate them in the opposite order.
Prior to asking them to rate either style of explanation, users saw a web page that told them that we had obscured the movie titles, and we showed them an explicit example of a redacted movie title. We conducted a pilot run of the experiment with a handful of users before launching the real experiment. Participants in the pilot run did not report and di culty in understanding the redacted movie titles or the redacted explanation rules.
We had 264 participants who completed all parts of the experiment. We did not collect demographic data about the participants but, since they were reached through our own contact lists, the majority will be undergraduate and postgraduate students in Irish universities.
Figure 5 shows how the participants rated explanation rules for helpfulness. Encouragingly, nearly 50% of participants found explanation rules to be a helpful or very helpful style of explanation (100 and 16 participants out of the 264, 2This is an observation made by Joseph A. Konstan in lecture 4-4 of the Coursera course Introduction to Recommender Systems, www.coursera.org. resp.); but about a quarter of participants found them neutral (69 participants), and a quarter found them unhelpful or very unhelpful (52 and 17, resp.). Figure 6 shows the same for the other style of explanation. Just over 70% of participants found this style of explanation to be helpful or very helpful (158 and 31 participants, resp.).
Note that we did not ask participants to compare the two styles of explanation. They are not in competition. It is conceivable that a real recommender would use both, either side-by-side or showing one of the two explanations by default and only showing the other to users who click through to a more detailed explanation page.
Furthermore, as the reader can judge by comparing
Figures 3 and 4, any direct comparison of the results is unfair
to the explanation rules since they have two levels of
redaction (the recommended movie and the antecedents in the
rules) whereas the histogram has just one (the recommended
movie). As far as we can tell, there is no explanation style
in [
For some readers, this may raise the question of why we showed participants the redacted histograms at all. The reason is to give a `yardstick'. If we simply reported that nearly 50% of participants found explanation rules to be helpful or very helpful, readers would not know whether this was a good outcome or not.
From the results, we cannot con dently conclude that the hypothesis holds: results are not in the same ball-park as the `yardstick'.3 But we can conclude that explanation rules are 3For readers who insist on a comparison: using Very Unhelpful = 1, Unhelpful = 2, etc., the mean rating for the a promising style of explanation: many users perceive them to be a helpful style of explanation, and they are therefore deserving of further study in a more realistic setting.
We note as a nal comment in this subsection that the
experiment reported in [
Next, we sought to elicit the perceived e ectiveness of our algorithm's way of building explanation rules: Hypothesis 3: that users would nd the algorithm's selection of conditions in the antecedents of the rules (based on accuracy and coverage) to be better than random.
In the same web-based user trial, we showed the participants two rules side-by-side (the ordering again being determined at random). One rule was constructed by Algorithm 1. The other rule was constructed so as to have the same number of conditions in its antecedent, but these were selected at random from among the candidate explanation conditions. Note they are not wholly random: they are still candidate explanation conditions (hence they are co-rated items on which the user and explanation partner agree) but they are not selected using accuracy and coverage.
We asked participants to compare the two rules. They selected one of four options: the rst rule was more helpful than the second; the second was more helpful than the rst; the two rules were equally helpful; and they were unable to tell which was the more helpful (\don't know").
There was no redaction in this part of the experiment. It
was important that participants judged whether the movie
preferences described in the antecedents of the rules did
support the recommended movie. Prior to asking users to rate
the two explanation rules, users saw a web page that told
them: that they would see a recommendation; that they
should pretend that the recommended movie was one that
they would like; that they would see two explanations; that
movie titles would no longer be obscured; and that they
should compare the two explanations for helpfulness. There
are, of course, the risks that measuring e ectiveness before
consumption like this may result in judgements that overlap
with persuasiveness, and that measuring perceived e
ectiveness is not as reliable as measuring something more objective
[
Figure 7 shows the outcomes of this part of the experiment. We see that 32% found the explanation rule to be more helpful (85 participants) and only 10% (27 participants) found the partly-random rules to be more helpful. This means that, of those who expressed a preference (85 plus 27 participants), 76% preferred the explanation rules and only 24% preferred the partly-random rules. Furthermore, a two-tailed z-test shows the di erence to be signi cant at the 0.01 level. This suggests that the algorithm does select candidate explanation conditions in a meaningful way.
However, 36% of participants found the rules to be equally helpful and 22% could not make a decision (95 and 57 participants resp.). This means, for example, that (again using redacted explanation rules is 3.21 (st.dev. 1.03), the mean rating for the redacted histograms is 3.66 (st.dev. 0.94); and, using Welch's t-test, we reject at the 0.01 level the null hypothesis that there is no di erence in the means. a two-tailed z-test), there is no signi cant di erence between the proportion who found explanation rules to be more helpful and the proportion who found the two rules to be equally helpful.
There are at least two reasons for this. The rst is that the participant is required to put herself `in the shoes' of another user. The recommendation and the rules are computed for a user in the MovieLens dataset, not for the person who is completing the experiment, who must pretend that she likes the recommendation. The person who completes the experiment may not know much, if anything, about the movies mentioned in the rules. This may be why the \don't know" option was selected so often.4 The alternative was to require participants in the experiment to register with the recommender and to rate enough movies that it would be able to make genuine recommendations and build realistic explanation rules. We felt that this placed too great a burden on the participants, and would likely result in an experiment skewed towards users with relatively few ratings.
The second reason is that the partly-random rules are still quite good rules: they are considerably more meaningful than wholly-random rules. As Table 2 shows, one of the partly-random rules used in the experiment is nearly as accurate as its corresponding explanation rule. The partlyrandom rules also have high coverage because randomly selected movies are often popular movies. In our pilot run of the experiment, we had tried wholly-random rules, but they were so egregiously worse than their corresponding explanation rules that we felt that using them would prejudice the results of the real experiment. Ironically, the partlyrandom rules that we use instead perhaps include too many movies that are reasonable substitutes for the ones in their 4An on-screen note told the participant that she was able to click on any title to get some information about the movie. If she did, we fetched and displayed IMDb genres and a oneline synopsis for the movie. But we did not record how many users exploited this feature. corresponding explanation rules, thus giving us much more equivocal results. 4.
We have presented an algorithm for building explanation rules, which are item-based explanations for user-based collaborative recommendations. We ran an o ine experiment and web-based user trial to test three hypotheses. We conclude that explanation rules are a practicable form of explanation: on two datasets no rule antecedent ever contained more than three conditions. We conclude that explanation rules o er a promising style of explanation: nearly 50% of participants found them to be helpful or very helpful, but the amount of redaction used in the experiment makes it hard to make rm conclusions about their e ectiveness. Finally, we conclude that users do nd the algorithm's selection of conditions for the rule antecedent to be better than random: just under 80% of participants who expressed a preference preferred the explanation rule to a partly-random variant. But results here are also partly confounded by the conditions of the experiment, where a participant has to put herself `in the shoes' of another user.
Given the caveats about the limitations of the
experiments, our main conclusion is that explanation rules are
promising enough that we should evaluate them further,
perhaps in a comparative experiment such as the one reported
in [
This publication has emanated from research supported in part by a research grant from Science Foundation Ireland (SFI) under Grant Number SFI/12/RC/2289. We are grateful to Barry Smyth for discussions about the design of our experiment. 6.