quire them to rely on memory. This can lead to unreliable feedback [2]. One way to reduce the e ort can be found in choice-based preference elicitation. Where most recommender systems ask users to provide a number of ratings on items (explicit feedback), recommender systems applying choice-based preference elicitation ask the user to make a number of choices (implicit feedback). Using implicit feedback to produce personalized ranking has been shown to provide better tting prediction models than using explicit feedback [8]. In recent user studies users of collaborative ltering systems were proCCS Concepts vided with choice-based preference elicitation[4, 6]. Where in the more standard rating-based preference elicitation peoInformation systems ! Recommender systems; Human-ple are asked to rate the items they know, in choice-based centered computing ! Human computer interaction preference elicitation they are asked to choose the item that (HCI); User studies; User models; best matches there preference from a list. In our own work, this alternative has been shown to require less e ort than
1.2
Previous research showed that choice-based preference elicitation can be successfully used to reduce e ort during user cold start, resulting in an improved user satisfaction with the recommender system. However, it has also been shown to result in highly popular recommendations. In the present study we investigate if trailers reduce this bias to popular recommendations by informing the user and enticing her to choose less popular movies. In a user study we show that users that watched trailers chose relatively less popular movies and how trailers a ected the overall user experience with the recommender system.
New user cold start is one of the central problems in recommender systems. It occurs when a user starts using a recommender system. As there is no information for this user to base recommendations on, the recommender system requires her to provide feedback in order to receive recommendations. This requires quite often signi cant e ort of the user.
In addition, as users watch only a certain amount of movies over any time period, asking users to provide a set amount of feedback may require them to provide feedback on items that they have experienced a longer time ago which will re
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from permissions@acm.org.
IntRS 2016, September 16, 2016, Boston, MA, USA. c 2016 ACM. ISBN 123-4567-24-567/08/06. . . $15.00 DOI: 10.475/123 4 1.3
Memory e ects could be a possible explanation for this
bias towards popular movies that results in people
receiving overly popular recommendations. In rating-based
recommender systems memory e ects have been shown to
inuence how users provide feedback. Bollen et al.[
The current study tries to investigate if this bias towards picking popular movies can be alleviated by giving users additional information to make more informed choices.
In order to both minimize the e ort required and maximize the reliability of the input given during the new user cold start situation, we propose to use choice-based preference elicitation and provide the user with additional information to give her the means to make more informed choices.
In most recommender systems users can already rely on meta-information like for example genre, cast and a synopsis. A possible additional source of information about a movie can be found in trailers. Trailers may help the user in two ways. Firstly, trailers can help a user in refreshing the memory to provide reliable feedback, alleviating potential memory problems described in the previous section. Secondly, even for movies that a user has not seen yet, a trailer can be used to evaluate whether or not a movie is worth watching. This is an advantage of choice-based preference elicitation over rating-based preference elicitation, because in rating-based users typically only rate (and provide information on) movies they have actually watched. 1.5
The present research aims to investigate how providing additional information in the form of trailers during choicebased preference elicitation a ects the interaction in terms of both objective behavior and subjective user experience.
In terms of objective behavior we hypothesize that trailers allow users to make more informed choices and rely less on popularity when making these choices. In other words, we expect the possibility to watch trailers to reduce the popularity of the items a user chooses.
In terms of user experience we expect trailers to provide the user with more information, which is expected to be reected in the perceived informativeness of the system. As we expect trailers to motivate users to select less popular movies, we expect perceived recommendation novelty (the opposite of popularity) and diversity to increase. Both novelty and diversity may a ect system and choice satisfaction.
It is hard to formulate expectations about the direction of
the e ect of trailers on user satisfaction. We expect user
satisfaction in this setting to consist of system satisfaction (i.e.
\how well does this system help me") and choice satisfaction
(\how happy am I with the item that I choose based on this
system"). In previous research novelty and system
satisfaction were shown to be negatively correlated [
The expected e ects are shown in Fig. 2 below, with where possible the directions of the hypothesized e ect.
A system was developed to address the research questions through an online study. Participants were invited to browse to a website where they could access our recommender system. Upon entering the website participants were assigned randomly to one of two experimental conditions: the trailer condition, where participants were given the possibility to watch trailers and the non-trailer condition, where participants could not watch those trailers. They were subsequently shown an introduction page with an informed consent form and a brief explanation about the task at hand.
After the explanation, the preference elicitation phase started
(see Fig 1 for a screenshot), where the experimental
manipulation came into e ect. Participants in the trailer condition
were able to see trailers, where participants in the non-trailer
condition were not. Applying the same methodology as in
[
After choosing, the system would incorporate the choice
and provide the participant with a new set of
recommendations. Participants would be assigned a null vector upon
entering. After which each choice was incorporated by the
recommender system in four steps, described in more detail
in [
After 9 such choices, the user would see an explanation about how the choices they made would be used to calculate the nal set of recommendations. The screen with nal recommendations was identical to the previous screens except for the explanation. The nal recommendations consisted of the Top-10 movies based on the last calculated user vector. People were asked to make the nal choice from this list after which they were invited to complete a survey designed to measure the user experience.
The interface allowed users to watch trailers in the trailer condition by hovering over the presented movie covers. The trailers were retrieved through The Movie Database1. After hovering for 2 seconds, a video player would appear in allocated space in the interface. Each trailer for which a user 1https://www.themoviedb.org/ pressed the play button was stored as a view. 2.1
The recommendations were predicted through a matrix factorization model trained on ratings for the 2500 most rated movies in the 10M MovieLens dataset. The nal dataset consisted of 69k users, 2500 items and 8.82M ratings. The performance metrics of the used model were up to standards (MAE: 0.61358, RMSE: 0.79643, measured through 5-fold cross-validation). 2.2
In total 89 participants made at least one choice in the system. Participants were recruited from di erent courses in the department and were entered in a ra e for one of 5 gift cards. No demographic information was asked. Out of the 89 participants 50 were in the condition where no trailers could be watched, 39 were able to watch trailers. The people who were able to do so, watched on average 10.38 trailers (median = 10; SD = 9:69).
In total 74 participants completed the survey. After inspection, data from 3 participants was removed because they completed the survey unrealistically fast. A total of 71 (40 of which did not have the possibility to watch trailers, 31 did) responses was thus used to study the e ects on user experience. 2.3
In order to test our hypothesis we measured aspects of behavior and developed a survey to measure user experience. In terms of behavior we measured the popularity of the movies people chose and whether or not they watched any trailers. Popularity is de ned as the ranked order based on the number of ratings in the original MovieLens dataset. The movies are ranked from the most rated (1) to the least rated (2500).
We investigate the user experience following the
evaluation framework from Knijnenburg et al. [
The results section will rst describe how trailers a ect the choices users make. After that the analysis of the survey data will provide insight in how trailers a ect the user experience. 3.1
The e ects on user behavior are expected to be two-fold. Firstly, as trailers allow the user to make more informed choices, we expect the individual chosen items to be less popular for people watching trailers. In other words, movies chosen by users who watch trailers are expected to have 1000
0 e v iltr a e −1000 −2000 watched_trailers
FALSE TRUE 2.5 5.0 choiceset 7.5 10.0 lower popularity ranks. Secondly, when people make less popular choices throughout the interaction with the system, we expect the individual choice sets to be less popular as a whole. For users that watch trailers we expect the average popularity rank of choice sets is expected to be lower.
An alternative way to study this e ect is by looking at the relative popularity of the choices users make, instead of the absolute popularity. To do this we calculated for each choice the di erence between the popularity rank of the chosen item and the average popularity rank of the items in the set. If this number is positive, the chosen item is above average in terms of popularity, if it is negative, the chosen item is below average in terms of popularity.
Although there was no di erence across experimental conditions, the plot in Figure 3 shows that for participants that actually watched trailers (i.e. people in the trailer conditions that watched at least one trailer) the relative popularity of the chosen item decreases after around 5 choices compared to participants that did not watch trailers (i.e. people in the non-trailer condition or in the trailer condition that did not watch any trailers). In a repeated measures ANOVA this e ect proves to be signi cantly lower (F (1; 87) = 6:992; p < 0:01) for users that watched trailers. Watching trailers thus made users choose relatively less popular movies.
In order to understand what the results are for the user experience we investigate the survey data. 3.2
The subjective constructs from the CFA were organized into a path model using Structural Equation Modeling. The resulting model had good model t ( 2(66) = 1052:974; p < 0:001; CF I = :997; T LI = :996; RM SEA = :029; 90%CI : [0:000; 0:084]). The corresponding path model is shown in Figure 4.
Di erent from earlier studies[
Choice Satisfaction task from the nal list of recommendations might not have been perceived as much di erent from the choice tasks during the preference elicitation task. System Satisfaction in turn is positively in uenced by Informativeness. In addition, the more people experience Informativeness, the less they perceive Diversity. Opposed to previous studies, we nd that higher diversity results in a lower Choice Satisfaction.
In order to investigate the overall e ects of the trailers we in addition consider the marginal means. The trailers a ect the user experience in a number of ways. Firstly, providing trailers is experienced as an increase in informativeness of the system (statistically signi cant: = 0:664; t(69) = 3:142; p < 0:01), as can be seen in the path model (Figure 4) and the marginal means (Figure 5).
It also results in an increased perceived diversity, but this e ect is counteracted by the decrease as a result of the increased informativeness. This indirect e ect of the manipulation on diversity through perceived informativeness results in the non-signi cant e ect we observer in Figure 5. As far as system satisfaction is concerned, trailers actually decrease the system satisfaction. But similar to perceived diversity, this direct e ect of trailers on system satisfaction is counteracted by the positive e ect of the increased informativeness on system satisfaction. 4.
The present study aimed to decrease the tendency to use popularity as a heuristic in a choice-based preference elicitation task by providing users with means to make informed choices.
The analysis on user behavior showed that people watching trailers are more inclined to pick relatively less popular items. By investigating the user experience we found that aside from the impact on the decisions users make, the user experience was in uenced. Informativeness of the system increased with the possibility of watching trailers. While no signi cant di erences were found on the other aspects of user experience, the path model provides insight in the positive and negative consequences of providing trailers, consisting of an increased informativeness and diversity, but decreased system satisfaction. 4.1
One of the limitations is that the e ect of trailers on user
experience with a recommender system is not tested against
a more standard approach of preference elicitation. As users
expressed rating items costs more e ort than choosing[
One aspect of behavior worth investigating based on the ndings in this study is information regarding in what stage of the preference elicitation task users watched trailers. The way data was stored in the current dataset does not allow us to investigate for example if people watch more trailers in the beginning, or towards the end of the study, which could provide more ne grained insight in how trailers in uence the choices people make. Future research should incorporate not only whether or not people watch trailers, but also when they do so. It would be particularly interesting to see if users use trailers di erently in the choice of the nal recommendations compared to the choices during the preference elicitation task.
The e ect of popularity on choice satisfaction needs to be investigated in more detail. Previous studies have shown that popularity of recommendations has a positive in uence on choice satisfaction in lab settings, but whether or not this e ect remains in the long run needs to be investigated. It is possible that popularity can be used as a heuristic when evaluating a recommender system, but that longer term interaction is actually harmed by high popularity.
We thank Olivier van Duuren, Niek van Sleeuwen, Bianca Ligt, Danielle Niestadt, Shivam Rastogi and Suraj Iyer for programming the user inferface and performing the experiment as part of their student research project. 5.