We describe the methodological considerations that arose over a series of experiments evaluating the e ectiveness of explanations for recommendations. In particular, we look at issues relating to: criteria, metrics, product domain used, choice of materials, possible confounding factors, and approximation of experience versus real experience. We generalize the problems we found and the solutions that we applied to adaptive systems. We illustrate the learned lessons with examples from our previous work on adaptive systems (ranging from adaptive learning to persuasive technologies).
The evaluation of adaptive systems is not easy, and several researchers have
pointed out potential pitfalls when evaluating adaptive systems. Examples of
pitfalls mentioned in [
The di culty in evaluating adaptive systems has led to a series of workshops
on this topic such as [
Recommender systems such as Amazon o er users recommendations, or suggestions of items to try or buy. These recommendations can then be explained to the user, e.g. \You might (not) like this item because...". In experiments, our system generates (given a simple user model) explanations for items using data retrieved from the Amazon website. In a between-subject design, we compared three degrees of personalization in a series of experiments. The example below illustrates the three conditions for one experiment in the movie domain: 1. Baseline: The explanation is neither personalized, nor describes item features: e.g. \This movie is one of the top 250 in the Internet Movie Database". 2. Non-personalized, feature based: e.g. \This movie belongs to the genre(s): Drama. Kasi Lemmons directed this movie." The feature `director' was not particularly important to this participant. 3. Personalized, feature based: e.g. \Unfortunately this movie belongs to at least one genre you do not want to see: Action & Adventure. Also it belongs to the genre(s): Comedy, Crime, Mystery and Thriller. This movie stars Jo Marr, Gary Hershberger and Robert Redford." For this user, the most important feature is leading actors, and the explanation considers that the user does not like action and adventure movies. 3
This section discusses some of the problems we encountered in our experiments evaluating explanations, and the solutions we adopted. It also elaborates on what other researchers can learn from our experiences for the evaluation of adaptive systems in general. We illustrate how the lessons can be generalized with example evaluations where the problem appears, and where solutions were applied. Although these problems are common to many evaluations of adaptive systems, it seemed fairer to select examples from our own work, as we have also been a ected by these problems in our previous research. 3.1
Problem. The rst issue we had to resolve was what we meant by a good explanation.
Solution. We surveyed the literature and discovered that explanations can serve
multiple aims, such as increasing transparency, trust, satisfaction, e ciency,
persuasiveness, and e ectiveness [
There is some evidence to suggest that personalization may increase
persuasion, or acceptance of recommendations [
Generalization of solution. In order to achieve a goal of optimization, the
optimum or the main evaluation criterion needs to be explicitly formulated prior to
evaluation. As optimization in one criterion may damage another, it is also
important to understand how the criteria relate to one other. For example, a study
of computer generated reports of babies in intensive care found that doctors
preferred graphical reports (satisfaction), but made better decisions with textual
reports (e ectiveness) [
Problem. The measured impact of explanations on e ectiveness may be confounded with the impact of the accuracy of the recommender system. If recommended items are meant to be liked by the user (i.e. we do not give predictions for items the user may not like), and the recommender system has poor accuracy, it would be hard to distinguish between the e ects of poor explanations and poor accuracy. Most likely, we would not be able to tell which factor was the main contributor to a large change in valuation of an item. This is a problem because in a real recommender system, it would be hard to guarantee comparable recommendation accuracy between participants and between conditions. Also, to obtain reasonable accuracy, participants would need to use the system for a non-trivial amount of time prior to evaluation, in order to provide their 3 Initially we used approximations of real experience, see Section 3.4 for a justi cation and further discussion. preferences.
Solution. We decided not to use a recommender system, and used random item selection instead. This meant that we did not require the experiments to include a training period. Instead, participants' preferences were explicitly requested in order to personalize the explanations. Also, our metric for e ectiveness (brie y described in Section 3.1) can be used regardless of whether participants liked or disliked the items; as long as they were able to make an initial assessment based on the explanation, we were able to study their change of opinion after experiencing the item. It was not a problem for us to o er explanations for items participants might not like as participants were told that the explanations were aimed at helping them make decisions (to try or not to try) rather than make (only positive) recommendations. This also meant that we did not have to control for recommendation accuracy.
Generalization of problem. As mentioned in the introduction (Section 1),
difculty in attributing cause has been previously discussed as a problem in the
evaluation of adaptive systems [
Generalization of solution. Layered evaluation has been mentioned as a way
to help overcome this problem [
Another solution for an adaptation taking time would be to run an
experiment in several installments - rst training the system on participants, and then
using the adapted system for further evaluations, such as in [
Problem. The e ectiveness of explanations and degree of personalization may well depend on the domain used.
Solution. We surveyed the literature and found that there has been a great deal
of debate about classi cation of products into di erent categories in economics
(see [
Generalization of solution. The solution of surveying domains, and evaluating in
multiple domains is applicable to adaptive systems in general. For example, in an
adaptive e-learning system, we evaluated an adaptive item sequencing strategy
for two learning tasks: a paired-associate learning task and a concept learning
task [
Problem. It can be very di cult and time consuming for participants to really
experience the recommended items. For example, it may take too long (or be
too expensive) for participants to read a recommended book or go on a
recommended holiday. Participants may also require time to fully experience a product
(for example, a real experience of a camera may involve using it over a couple of
days, so that initial technical di culties do not overly in uence the participants'
nal evaluation).
Solution. Previous work has approximated experience of the recommended item,
for example by letting participants read online reviews [
Generalization of problem. It can be very time consuming for participants to fully
experience adaptive systems. For example, a realistic experience of an adaptive
learning system involves learners using it over multiple sessions, learning
something they would normally be learning in another setting. Instead, we often
evaluate over one or two sessions, sometimes using a controlled arti cial
learning domain. For example, when evaluating adaptive navigation in a learning
system, we have used the arti cial domain of square dancing, with participants
learning to operate dancers on the screen using multiple computer-based lessons,
but all within a one-hour session [
Generalization of solution. Approximation of experience is often a reasonable
thing to do in early evaluations, as it requires less time, allows better
experimental control, and is sometimes better for ethical reasons. This can then be
followed by an evaluation in more realistic settings, often of a longitudinal
nature. We have used approximation in many of our studies. For example, we
measured whether an emphatic embodied agent in uenced participants' mood
after inducing a negative mood in an arti cial test (rather than say a real course
assessment) [
Problem. For the real experience valuation we needed items that participants had not yet experienced, that had enough interesting features to produce an explanation, would not take too long to evaluate, and that were ethically ok to use. Solution. To reduce time needed per item, we used short movies instead of full movies. To avoid movies that participants had already seen, an I-have-alreadyseen-this-movie button that skipped to another movie was added. To avoid exposing participants to sensitive material such as (extreme) violence or sex, only movies suitable for 15 years and over (PG-15) were used. By choosing nono ensive movies, there was a distinct risk that participants' ratings of movies would not be as well spread over possible rating values as they could be. We used a pilot study to con rm that while the distribution may not make full use of the possible values, participants still indicate values that di ered su ciently from the mid-point to warrant an interesting analysis. We also considered the presences of relevant features when selecting the movies. We included movies with actors (e.g. Rowan Atkinson) or directors (e.g. Tim Burton) that were likely to be known. As certi cation rating (e.g. PG - parental guidance advised) was used as a feature for explanation generation, we also selected movies that had an international certi cation, which is otherwise often missing for short movies.
We also found that most short movies use less famous actors/directors, and
therefore, despite our best e orts, there was a higher frequency of unknown
actor and director names than in our previous experiments [
Generalization of problem. Fields such as psychology have a common practice of carefully selecting the materials they use in experiments. This is done either because the material may a ect the outcome (we also discuss avoiding confounding factors in Section 3.2) or for other reasons such as ethical ones. In the evaluation of adaptive systems, we are often told what materials were used, but not on the basis of which criteria they have been chosen, and whether pilot studies have been done to validate their appropriateness.
Generalization of solution. The criteria on the basis of which materials are
chosen need to be clearly de ned and stated. In addition, pilot studies need to be
performed to test the suitability of materials. For example, when studying the
e ect of adding a doctor's photo on website credibility, we needed the photo to
contain an image of a doctor that would be considered credible in this domain.
We found such a photo by running pilot studies in which participants judged
the profession of the person depicted, and rated their domain credibility using
validated metrics [
Problem. To measure true e ectiveness, we needed to distinguish between participants not having formed an opinion of the item, and participants believing the item is kind of average (middle of the scale). Solution. We decided to add a separate opting-out option as an alternative for rating, for participants who were not able to formed an opinion. In our analyses of e ectiveness, we excluded ratings of participants who had opted-out. One thing we noticed in all our experiments was that while baseline explanations did surprisingly well for e ectiveness, they also led to a very high opt-out frequency: participants were unable to provide a rating. This means that only considering the change between the before and after rating for those people who opt-in is not a true re ection of e ectiveness.
Despite the opportunity to opt-out, we still found that some participants seemed to use the middle of the scale to indicate that they had no real opinion about a movie. This could be seen in the higher frequency of middle of the scale ratings for baseline explanations, or when participants were asked to give a rating based on only a movie title (without the explanation).
Problem. The e ectiveness metric from [
Generalization of problems. In the evaluation of all systems, including adaptive ones, one has to take care that the metric used is really measuring what you want to nd out. For example, when assessing personalisation in interactive TV often the time spent watching a programme is used as an indication of user interest. However, longer viewing times may well have been caused by other factors such as the viewer having a co ee or even being so bored that they fell asleep. Learners spending more time on a lesson may mean that they are more motivated or that they nd the lesson harder to understand.
Generalization of solutions. A critical analysis is required of all metrics used,
asking whether there are situations when the value given by the metric is not
accurate. For a metric to be good, the same value should have the same meaning
independent of the circumstances. For example, our experience shows that the
e ectiveness metric of [
This paper illustrates some of the problems we have encountered when investigating the e ectiveness of explanations in a recommender system. Our investigations consisted of a series of experiments, and in each experiment we improved our understanding of the evaluation design. As we have discussed, many of these issues are generalizable to other types of adaptive systems. In any evaluation, it is important to: { Decide which criteria to use { Avoid confounding factors { Take into account domain e ects { Build up the experiment gradually, and consider limited resources { Take into account the e ects of the material you select { Consider if a metric really measures what you want
From this paper it would perhaps be easy to conclude that it is hard (and probably impossible) to design the perfect evaluation. In retrospect, there is always something else that could impact the results. Clearly de ning your goals, metrics and re ning your design through a sequence of experiments, and gradually investigating di erent aspects, will however help you avoid the most common pitfalls - may your next evaluation be a successful one!