Recommender Systems (RS) help users to orientate themselves in large product assortments and provide decision support. Explanations help recommender systems to enhance their impact on users by, for instance, justifying made recommendations. Arguments provide reason in a more structured way, by denoting a conclusion that follows from one or more premises. While expert systems' explanation have a long tradition in using argumentative patterns, argumentative explanations for recommendations have not yet been systematically researched. This paper compares therefore the persuasion potential of different explanation styles (sentences, facts or argument style) by comparing the robustness of subjects' preferences when employing an additive utility model from conjoint analysis.
Recommender Systems (RS) support online customers in
their decision making and should help them to avoid poor
decisions [
Workshop IntRS at RecSys ’14 Foster City, CA USA
Copyright 20XX ACM X-XXXXX-XX-X/XX/XX ...$15.00.
recommendation list and promote objectives such as users’
trust in the system and confidence in decision making. In
the domain of expert systems explanations have already a
long tradition, where formal argumentation traces can serve
as explanations that justify the output of a system [
We researched the question if the introduction of an argumentbased writing style, i.e. use of the keyword therefore to denote the conclusion of the preceding premises, has an impact on the robustness of users’ preferences in face of additional explanations. As already mentioned we asked users to disclose their preferences for three different item domains (hiking routes, mobile phone plans and energy plans) in a supervised offline questionnaire. Figures 1 and 2 depict two exemplary items from the hiking domain. Subjects were invited to participate in a seminar room, where they had to answer a paper & pencil survey with two parts. The first part included for each of the three domains exactly 6 items, that are described by either 4 or 5 characteristics. Table 1 depicts the three item domains and the artificial design space of the item portfolios. To avoid confusion the semantics of the domain attributes were defined in a sidebar (e.g. Smartphone: denotes a device in the range of HTC Desire X or Nokia Lumia 625). Participants had to rank the 6 options according to their general preference with respect to the particular item domain. After disclosing their preferences in the first part of the questionnaire (see Figure 1 for a translated excerpt of the questionnaire) users had to solve a picture puzzle, where 10 different errors were hidden. The purpose of this task is twofold: first, it distracts users from their thoughts on the ranking tasks and, second, we could use the numerical measure of correctly marked errors to assess how concentrated participants followed the questionnaire. Once participants had finished the first part they handed it in and received the second part of the survey. This way we were able to avoid that participants could have taken a look on their first-round ranking when answering the second part. In the second part participants had again to rank sets of five items from the three item domains. However, in addition to the item characteristics already used in the first-round, additional explanations were given for each item. The explanation style acts as the manipulated variable (solely fact-based, argumentative facts and argumentative sentences). Explanation style is permuted within subjects, i.e. participants are confronted with all three explanation styles for a different item domain and in different orders, while the combination of item domain and explanation style is varied between subjects. For each item exactly two arguments, each with two premises and one conclusion, are added as additional information (see examples in Table 2). See Figure 2 for a depiction of two exemplary items from the hiking domain with explanations following the style of argumentative facts.
Finally, the questionnaire controlled for demographic characteristics and checked if participants noticed the intervention, i.e. one question asked what was relevant for ranking the items with multiple answering options. For analysis we selected only participants that considered the additional explanations provided in the second part in their ranking decision.
In Figure 3 we sketch the big picture of the study design.
Thus, participants rank sets of items from three different
domains twice, where item sets in the first and second part
of the questionnaire do not overlap. Due to measuring user
preferences twice for each domain (without and with
intervention of a specific explanation style), we can control for
the participants’ preferences on item sets and their
presentation. We employ an additive model from conjoint analysis,
that allows us to estimate the utilities for each item
characteristic [
low altitude easy distance very family-friendly low altitude easy distance therefore very family-friendly This route is of low altitude and easy distance, therefore it is very family-friendly. 100% renewable energy low environmental impact high sustainability 100% renewable energy low environmental impact therefore high sustainability This energy plan offers 100% renewable energy with a low environmental impact, therefore its sustainability is high. low basic fee many anytime minutes ideal for heavy use low monthly basic fee many anytime minutes therefore ideal for heavy use This mobile phone plan offers a low monthly basic fee with many anytime minutes, therefore it is ideal for heavy use.
Z denotes the positive or negative utility contributed by a specific item characteristic Z (for instance, the possibility to have your meal on route in the hiking domain). Having estimated the individual utilities of each item characteristic we computed an a priori ranking for the unseen item sets in the survey’s second part that is then compared with the observed ranks for each user.
In total 136 subjects, mostly students from Alpen-AdriaUniversta¨t Klagenfurt, participated in our survey. From each participant we received three rankings in the second part of the survey (one for each domain), i.e. a total of 408 computed rank correlations before cleaning. More than 80% of all participants were young people aged between 18 and 25. Two thirds of our participants were females. All respondents had a high-school degree and a few of them had already a graduation degree from a university. Before analysis we rigorously excluded participants whose answers might be unreliable due to the following criteria: 1. Only respondents who demonstrated a thorough attitude by identifying at least 50% of all hidden errors in the picture puzzle. 2. We asked participants what they considered to be relevant for making their decisions on the rankings. Based on the answers to this multiple choice question we included only respondents who had noticed the additional information (explanations) and excluded all respondents who answered that they relied on their gut feelings. 3. We also asked participants how they experienced this survey with the answering options interesting, challenging, boring, unclear and useless. For further consideration we only kept respondents that answered challenging and were thus captivated by the ranking tasks. We assumed that the option interesting is a polite way of saying boring or useless. 4. Finally we cleaned records from the dataset, where the estimation of individual utilities for product characteristics was not reliable, i.e. rank correlation between the a priori rankings based on estimated utility weights and the actual a priori ranking of participants had to be above 0.7.
After applying this extremely restrictive selection procedure we derived at the following size of the dataset (see Table 3). In order to check for the robustness of preferences af
styled facts that included the keyword therefore to denote a conclusion reduced the robustness of participants’ preferences more than the pure fact-based explanations, i.e. supporting our hypothesis that an argumentative explanation style would influence users more. Argumentative sentences preserved user preferences more than the fact-based explanation styles. Obviously, sentences need more cognitive effort from users to be understood and the effect of the keyword therefore was seemingly lost in the sentence structure. The difference between Spearman’s in all three categories is statistically significant according to Kruskall-Wallis test (p = 0.037).
In addition we checked for interaction effects between explanation style and product domain. As can be seen from Table 5 fact-based explanation styles lead to less robust preferences than sentence-based explanation styles. Furthermore, argumentative facts seem to reduce participant’s robustness of preferences even more than a pure facts based explanation style. The only exception is the hiking domain, where the order between facts and argumentative facts is inverted. However, in this product domain preference robustness is generally lower and it might have been harder for respondents to determine own preferences in the hiking domain than in the other two domains. able features of recommender systems. Limitations or possible lines of future research include varying the complexity of arguments (i.e the number of premises) or its number as well as additional item domains.
Authors acknowledge the financial support from the European Union (EU), the European Regional Development Fund (ERDF), the Austrian Federal Government and the State of Carinthia in the Interreg IV Italien- O¨sterreich programme (project acronym O-STAR). 5.
Argumentative facts Argumentative sentences 0.27 0.38 0.58
per domain and expl. style
This study therefore showed, that fact-based explanations
and an argumentative explanation style impacted
participants’ preferences stronger than full sentence explanations.
Objections against these conclusions might be the lack of
a control group and the paper & pencil design without a
real recommendation situation. A control group would
allow us to estimate the natural stability of preferences
between both rounds and without any intervention. However,
in this study we were not interested in absolute rank
correlation measures, but only in the comparison of robustness
of respondents’ preferences between different conditions and
assumed that some natural instability would affect all
explanation styles the same way. In order to assess the impact
of an argumentative explanation style we wanted to control
for other effects and biases as good as possible. The
supervised paper & pencil approach allowed us to control for user
preferences, the item portfolio and the persuasiveness of the
explanation content itself as well as insisting on a high
reliability of the measurements by excluding participants, who
made arbitrary rankings or did not notice the additional
explanations. In a previous study [
This short paper presented an innovative study design for measuring the impact of different explanation styles on participants’ robustness of preferences in face of additional explanations. The results indicate that fact-based explanations have a stronger impact on participants preference stability than sentence-based explanations. Furthermore, the use of the keyword therefore indicating a conclusion drawn from premises and an argumentative explanation style had already a measurable impact on participants. Thus arguments and fact-based explanations make users change their minds about the item portfolio and can therefore be valu