We describe an observational study of a recommender system that provides users with direct control over their personalization. Specifically, we allow users to tune a movie recommender towards more or less popular content. We report on 14 months of usage, which includes 6,846 users who visited the interface at least once. We find, surprisingly, that the popularity of items a user has interacted with historically is a poor predictor of their use of this interface.
In prior work [
Our platform, MovieLens (http://movielens.org), allows users to
choose among multiple recommendation algorithms [
The popularity tuner interface is shown in Figure 1. On each more/less popular click action, we use a binary search to locate new weights that will replace 4 of the user’s top 20 recommendations. For example, a user clicking “more popular” triggers a binary search that might yield a new blend of 92.1% predicted rating with 7.9% popularity. We emphasize the changes visually in the interface to help users understand the result of their actions. To encourage use of the feature, we added a “configure” button next to the user’s list of top recommendations at the top of the home page.
We deployed the feature on April 11, 2016, and collected data through June 15, 2017. We consider two groups for analysis. First, for examining overall use of the popularity tuner, we look at all users who visited the interface one or more times (visited_sample, N=6846, median logins=11, median ratings=168). Second, for examining users’ preferred popularity settings, we look at users who took at least one action (interacted_sample, N=4349, median logins=14, median ratings=205). These samples are skewed toward power users as compared with all users that logged in during this period (N=31371, median logins=3, median ratings=39).
Over the span of this observational study (or even the span of a single session), users might change settings multiple times. In fact, users in the visited_sample opened the tuning interface 24,657 times (3.6 times per user); users in the interacted_sample visited the page 19,465 times (4.5 times per user). Unless otherwise noted, user-level analysis examines the user’s final configuration at the time of analysis. For example, a user might try the tuner interface three times and explore the system in between each change; our user-level analysis considers only the most recent configuration. 2.1
The distribution of per-user activity in online systems is often
modeled by a power law distribution [
Within the interacted_sample, the most common popularity setting is the default one (1,390 users, 32.0% of sample). That is, after exploring one or more new settings, many users return to the default, either by clicking the “reset” button, or by sending the percentile # actions # sessions session length (sec.) 25th 0 1 3 50th (median) 1 2 5 80th 5 5 67 95th 14 10 251 99th 35 20 1428 Table 1: Descriptive statistics of activity levels by percentile in the observational study across users in the visited_sample. # actions and # sessions are user-level variables, while session length is a session-level variable with possibly multiple observations per user.
1400 rs1200 e su1000 fo 800 r 600 be 400 um 200 n 0 -5 0 5 10
variable estimate intercept -1.160000 *** # logins 0.000012 # ratings 0.000062 *** predictor==SVD 0.068710 *** mean pp1y, rated 0.818100 ** mean pp1y, wishlisted 0.064480 mean pp1y, clicked 0.401600 *** Table 2: Summary of our regression model to predict users’ choices, R2 = 0.04. Statistically significant p-values indicated as *** (p < 0.001) or ** (p < 0.01). same number of “more popular” and “less popular” actions. More users (1,714, 39.4%) end with a setting more popular than default, as compared with the number that end less popular than default (1,245, 28.6%). See Figure 2 for a visualization of this data, represented as number of steps from the default.
It is possible that we can predict users’ preferred popularity settings from their behavioral data. To explore this idea, we construct regression models using popularity-related predictor variables. In particular, we include a metric — “one-year popularity percentile” (pp1y) — that exactly mirrors the variable used in the popularity blending algorithm. We measure pp1y of rated, wishlisted, and clicked items to capture a user’s propensity to interact with more or less popular content. In addition, we use variables that capture the user’s number of logins, number of ratings, and chosen prediction algorithm. 0.30 0.25 lue 0.20 a v ing 0.15 d len 0.10 b itry 0.05 a lpuo 0.00 p 0.05 0.100.80 0.85 0.90 0.95 1.00 0.80 0.85 0.90 0.95 1.00 0 200 400 600 800 1000 mean pp1y, rated mean pp1y, clicked # ratings
Our regression model has several statistically significant variables, but overall, it is a poor fit with users’ popularity blending choices (adjusted R-squared=0.04); see Table 2 for a summary and Figure 3 for a visualization of several variables. The model indicates that the popularity of a user’s rated and clicked-on movies is positively correlated with the user’s final configuration value. Also, the model shows a relationship between a user’s choice of prediction algorithm and their chosen popularity blending weight (means: item-item=0.09, funkSVD=0.15). The model shows that number of ratings is positively correlated with the popularity setting, but with a tiny efect size; number of logins and popularity of wishlisted content are not statistically significant.
Thousands of users have tried the popularity tuner feature to conifgure their movie recommender. They have used the feature in very diferent ways, and it proves dificult to predict their choices based on behavioral data. The poor overall fit of our regression model underscores the potential helpfulness of giving control to users. While it is possible that our model is too preliminary, or that our interface does not suficiently elicit users’ ideal preferences, it is also possible that users have fickle preferences for their recommenders that are dificult to get right through modeling alone.