We present the results of two controlled behavioral studies on the effects of online recommendations on consumers' economic behavior. In the first study, we found strong evidence that participants' willingness to pay was significantly affected by randomly assigned song recommendations, even when controlling for participants' preferences and demographics. In the second study, we presented participants with actual system-generated recommendations that were intentionally perturbed (i.e., significant error was introduced) and observed similar effects on willingness to pay. The results have significant implications for the design and application of recommender systems as well as for e-commerce practice.
Recommender systems have become commonplace in online
purchasing environments. Much research in information systems
and computer science has focused on algorithmic design and
improving recommender systems’ performance
In this paper, we investigate the relationship between recommender systems and consumers’ economic behavior. Drawing on theory from behavioral economics, judgment and decision-making, and marketing, we hypothesize that online recommendations1 significantly pull a consumer’s willingness to pay in the direction of the recommendation. We test our hypotheses using two controlled behavioral experiments on the recommendation and sale of digital songs. In the first study, we find strong evidence that randomly generated recommendations (i.e., not based on user preferences) significantly impact consumers’ willingness to pay, even when we control for user preferences for the song, demographic and consumption-related factors, and individual level heterogeneity. In the second study, 1 In this paper, for ease of exposition, we use the term “recommendations” in a broad sense. Any rating that the consumer receives purportedly from a recommendation system, even if negative (e.g., 1 star on a five-star scale), is termed a recommendation of the system.
Paper presented at the 2012 Decisions@RecSys workshop in conjunction with the 6th ACM conference on Recommender Systems. Copyright © 2012 for the individual papers by the papers' authors. Copying permitted for private and academic purposes. This volume is published and copyrighted by its editors. 2.
Behavioral research has indicated that judgments can be
constructed upon request and, consequently, are often influenced
by elements of the environment. One such influence arises from
the use of an anchoring-and-adjustment heuristic
Past studies have largely been performed using tasks for which a
verifiable outcome is being judged, leading to a bias measured
against an objective performance standard
One of the few papers identified in the mainstream anchoring literature that has looked directly at anchoring effects in preference construction is that of Schkade and Johnson (1989). However, their work studied preferences between abstract, stylized, simple (two-outcome) lotteries. This preference situation is far removed from the more realistic situation that we address in this work. More similar to our setting, Ariely et al. (2003) observed anchoring in bids provided by students participating in auctions of consumer products (e.g., wine, books, chocolates) in a classroom setting. However, participants were not allowed to sample the goods, an issue we address in this study.
Predicted Ratings (expressing recommendations for unknown items)
(Consumer preference estimation)
(Preference formation / purchasing behavior / consumption)
Actual Ratings (expressing preferences for consumed items) Very little research has explored how the cues provided by recommender systems influence online consumer behavior. Cosley et al. (2003) dealt with a related but significantly different anchoring phenomenon in the context of recommender systems. They explored the effects of system-generated recommendations on user re-ratings of movies. They found that users showed high test-retest consistency when being asked to re-rate a movie with no prediction provided. However, when users were asked to rerate a movie while being shown a “predicted” rating that was altered upward or downward from their original rating by a single fixed amount of one rating point (providing a high or a low anchor), users tended to give higher or lower ratings, respectively (compared to a control group receiving accurate original ratings). This showed that anchoring could affect consumers’ ratings based on preference recall, for movies seen in the past and now being evaluated.
Adomavicius et al. (2011) looked at a similar effect in an even more controlled setting, in which the consumer preference ratings for items were elicited at the time of item consumption. Even without a delay between consumption and elicited preference, anchoring effects were observed. The predicted ratings, when perturbed to be higher or lower, affected the consumer ratings to move in the same direction. The effects on consumer ratings are potentially important for a number of reasons, e.g., as identified by Cosley et al. (2003): (1) Biases can contaminate the inputs of the recommender system, reducing its effectiveness. (2) Biases can artificially improve the resulting accuracy, providing a distorted view of the system’s performance. (3) Biases might allow agents to manipulate the system so that it operates in their favor. Therefore, it is an important and open research question as to the direct effects of recommendations on consumer behavior. However, in addition to the preference formation and consumption issues, there is also the purchasing decision of the consumer, as mentioned in Figure 1. Aside from the effects on ratings, there is the important question of the possibility of anchoring effects on economic behavior. Hence, the primary focus of this research is to determine how anchoring effects created by online recommendations impact consumers’ economic behavior as measured by their willingness to pay. Based on the prior research, we expect there to be similar effects on economic behavior as observed with consumer ratings and preferences. Specifically, we first hypothesize that recommendations will significantly impact consumers’ economic behavior by pulling their willingness to pay in the direction of the recommendation, regardless of the accuracy of the recommendation.
Hypothesis 1: Participants exposed to randomly generated
artificially high (low) recommendations for a product will
exhibit a higher (lower) willingness to pay for that product.
A common issue for recommender systems is error (often
measured by RMSE) in predicted ratings. This is evidenced by
Netflix’s recent competition for a better recommendation
algorithm with the goal of reducing prediction error by 10%
Hypothesis 2: Participants exposed to a recommendation that contains significant error in an upward (downward) direction will exhibit a higher (lower) willingness to pay for the product.
We test these hypotheses with two controlled behavioral studies, discussed next.
Study 1 was designed to test Hypothesis 1 and establish whether or not randomly generated recommendations could significantly impact a consumer’s willingness to pay. 3.1.
Both studies presented in this paper were conducted using the same behavioral research lab at a large public North American university, and participants were recruited from the university’s research participant pool. Participants were paid a $10 fee plus a $5 endowment that was used in the experimental procedure (discussed below). Summary statistics on the participant pool for both Study 1 and Study 2 are presented in Table 1. Seven participants were dropped from Study 1 because of response issues, leaving data on 42 participants for analysis.
The experimental procedure for Study 1 consisted of three main tasks, all of which were conducted on a web-based application using personal computers with headphones and dividers between participants. In the first task, participants were asked to provide ratings for at least 50 popular music songs on a scale from one to five stars with half-star increments. The songs presented for the initial rating task were randomly selected from a pool of 200 popular songs, which was generated by taking the songs ranked in the bottom half of the year-end Billboard 100 charts from 2006 and 2009.2 For each song, the artist name(s), song title, duration, album name, and a 30-second sample were provided. The objective of the song-rating task was to capture music preferences from the participants so that recommendations could later be generated using a recommendation algorithm (in Study 2 and post-hoc analysis of Study 1, as discussed later). In the second task, a different list of songs was presented (with the same information for each song as in the first task) from the same set of 200 songs. For each song, the participant was asked whether or not they owned the song. Songs that were owned were excluded from the third task, in which willingness-to-pay judgments were obtained. When the participants identified at least 40 songs that they did not own, the third task was initiated. In the third main task of Study 1, participants completed a withinsubjects experiment where the treatment was the star rating of the song recommendation and the dependent variable was willingness to pay for the songs. In the experiment, participants were presented with 40 songs that they did not own, which included a star rating recommendation, artist name(s), song title, duration, album name, and a 30 second sample for each song. Ten of the 40 songs were presented with a randomly generated low recommendation between one and two stars (drawn from a uniform distribution; all recommendations were presented with a one decimal place precision, e.g., 1.3 stars), ten were presented with a randomly generated high recommendation between four and five stars, ten were presented with a randomly generated midrange recommendation between 2.5 and 3.5 stars, and ten were presented with no recommendation to act as a control. The 30 songs presented with recommendations were randomly ordered, and the 10 control songs were presented last. 2 The Billboard 100 provides a list of popular songs released in each year. The top half of each year’s list was not used to reduce the number of songs in our database that participants would already own.
To capture willingness to pay, we employed the incentivecompatible Becker-DeGroot-Marschack method (BDM) commonly used in experimental economics (Becker et al. 1984). For each song presented during the third task of the study, participants were asked to declare a price they were willing to pay between zero and 99 cents. Participants were informed that five songs selected at random at the end of the study would be assigned random prices, based on a uniform distribution, between one and 99 cents. For each of these five songs, the participant was required to purchase the song using money from their $5 endowment at the randomly assigned price if it was equal to or below their declared willingness to pay. Participants were presented with a detailed explanation of the BDM method so that they understood that the procedure incentivizes accurate reporting of their prices, and were required to take a short quiz on the method and endowment distribution before starting the study. At the conclusion of the study, they completed a short survey collecting demographic and other individual information for use in the analyses. The participation fee and the endowment minus fees paid for the required purchases were distributed to participants in cash. MP3 versions of the songs purchased by participants were “gifted” to them through Amazon.com approximately within 12 hours after the study was concluded. 3.2.
We start by presenting a plot of the aggregate means of willingness to pay for each of the treatment groups in Figure 2. Note that, although there were three treatment groups, the actual ratings shown to the participants were randomly assigned star ratings from within the corresponding treatment group range (low: 1.0-2.0 stars, mid: 2.5-3.5 stars, high: 4.0-5.0 stars).
As an initial analysis, we performed a repeated measure ANOVA, as shown in Table 2, demonstrating a statistically significant effect of the shown rating on willingness to pay. Since the overall treatment effect was significant, we followed with pair-wise contrasts using t-tests across treatment levels and against the control group as shown in Table 3. All three treatment conditions significantly differed, showing a clear, positive effect of the treatment on economic behavior.
To provide additional depth for our analysis, we used a panel data
regression model to explore the relationship between the shown
star rating (continuous variable) and willingness to pay, while
controlling for participant level factors. A Hausman test was
conducted, and a random effects model was deemed appropriate,
which also allowed us to account for participant level covariates
in the analysis. The dependent variable, i.e., willingness to pay,
was measured on an integer scale between 0 and 99 and skewed
toward the lower end of the scale. This is representative of typical
count data; therefore, a Poisson regression was used
(overdispersion was not an issue). The main independent variable
was the shown star rating of the recommendation, which was
continuous between one and five stars. Control variables for
several demographic and consumer-related factors were included,
which were captured in the survey at the end of the study.
Additionally, we controlled for the participants’ preferences by
calculating an actual predicted star rating recommendation for
each song (on a 5 star scale with one decimal precision), post hoc,
using the popular and widely-used item-based collaborative
filtering algorithm (IBCF)
Mid 1.723** (interval five point scale) for the music genres country, rock, hip hop, and pop, the number of songs owned (interval five point scale), frequency of music purchases (interval five point scale), whether they thought recommendations in the study were accurate (interval five point scale), and whether they thought the recommendations were useful (interval five point scale). The composite error term (ui + εij) includes the individual participant effect ui and the standard disturbance term εij. The results of the regression are shown in Table 4. Note that the control observations were not included, since they had null values for the main dependent variable ShownRating.
The results of our analysis for Study 1 provide strong support for Hypothesis 1 and demonstrate clearly that there is a significant effect of recommendations on consumers’ economic behavior. Specifically, we have shown that even randomly generated recommendations with no basis on user preferences can impact consumers’ perceptions of a product and, thus, their willingness to pay. The regression analysis goes further and controls for participant level factors and, most importantly, the participant’s predicted preferences for the product being recommended. As can be seen in Table 4, after controlling for all these factors, a one unit change in the shown rating results in a 0.168 change (in the same direction) in the log of the expected willingness to pay (in cents). As an example, assuming a consumer has a willingness to pay of $0.50 for a specific song and is given a recommendation, increasing the recommendation star rating by one star would increase the consumer’s willingness to pay to $0.59.
The goal of Study 2 was to extend the results of Study 1 by testing Hypothesis 2 and exploring the impact of significant error in true recommendations on consumers’ willingness to pay. As discussed below, the design of this study is intended to test for similar effects as Study 1, but in a more realistic setting with recommender-system-generated, real-time recommendations..
Participants in Study 2 used the same facilities and were recruited from the same pool as in Study 1; however, there was no overlap in participants across the two studies. The same participation fee and endowment used in Study 1 was provided to participants in Study 2. 15 participants were removed from the analysis in Study 2 because of issues in their responses, leaving data on 55 participants for analysis.
Study 2 was also a within-subjects design with perturbation of the
recommendation star rating as the treatment and willingness to
pay as the dependent variable. The main tasks for Study 2 were
virtually identical to those in Study 1. The only differences
between the studies were the treatments and the process for
assigning stimuli to the participants in the recommendation task of
the study. In Study 2, all participants completed the initial
songrating and song ownership tasks as in Study 1. Next, real song
recommendations were calculated based on the participants’
preferences, which were then perturbed (i.e., excess error was
introduced to each recommendation) to generate the shown
recommendation ratings. In other words, unlike Study 1 in which
random recommendations were presented to participants, in Study
2 participants were presented with perturbed versions of their
actual personalized recommendations. Perturbations of -1.5 stars,
-1 star, -0.5 stars, 0 stars, +0.5 stars, +1 star, and +1.5 stars were
added to the actual recommendations, representing seven
treatment levels. The perturbed recommendation shown to the
participant was constrained to be between one and five stars,
therefore perturbations were pseudo-randomly assigned to ensure
that the sum of the actual recommendation and the perturbation
would fit within the allowed rating scale. The recommendations
were calculated using the item-based collaborative filtering
(IBCF) algorithm
Each participant was presented with 35 perturbed, personalized song recommendations, five from each of the seven treatment levels. The song recommendations were presented in a random order. Participants were asked to provide their willingness to pay for each song, which was captured using the same BDM technique as in Study 1. The final survey, payouts, and song distribution were also conducted in the same manner as in Study 1. 4.2.
For Study 2, we focus on the regression analysis to determine the relationship between error in a recommendation and willingness to pay. We follow a similar approach as in Study 1 and model this relationship using a Poisson random effects regression model. The distribution of willingness to pay data in Study 2 was similar to that of Study 1, overdispersion was not an issue, and the results of a Hausman test for fixed versus random effects suggested that a random effects model was appropriate. We control for the participants’ preferences using the predicted rating for each song in the study (i.e., the recommendation rating prior to perturbation), which was calculated using the IBCF algorithm. Furthermore, the same set of control variables used in Study 1 was included in our regression model for Study 2. The resulting regression model is presented below, where the main difference from the model used in Study 1 is the inclusion of Perturbationij (i.e., the error introduced for the recommendation of song j to participant i) as the main independent variable. The results are presented in Table 5. log(WTPij)= b0 + b1(Perturbationij)+ b2(PredictedRatingij) + b3(Controlsi) + ui + εij The results of Study 2 provide strong support for Hypothesis 2 and extend the results of Study 1 in two important ways. First, Study 2 provides more realism to the analysis, since it utilizes real recommendations generated using an actual real-time recommender system. Second, rather than randomly assigning recommendations as in Study 1, in Study 2 the recommendations presented to participants were calculated based on their preferences and then perturbed to introduce realistic levels of system error. Thus, considering the fact that all recommender systems have some level of error in their recommendations, Study 2 contributes by demonstrating the potential impact of these errors. As seen in Table 5, while controlling for preferences and other factors, a one unit perturbation in the actual rating results in a 0.115 change in the log of the participant’s willingness to pay. As an example, assuming a consumer has a willingness to pay of $0.50 for a given song, perturbing the system’s recommendation positively by one star would increase the consumer’s willingness to pay to $0.56.
Study 1 provided strong evidence that willingness to pay can be affected by online recommendations through a randomized trial design. Participants presented with random recommendations were influenced even when controlling for demographic factors and general preferences. Study 2 extended these results to demonstrate that the same effects exist for real recommendations that contain errors, which were calculated using the state-of-theart recommendation algorithms used in practice.
There are significant implications of the results presented. First, the results raise new issues on the design of recommender systems. If recommender systems can generate biases in consumer decision-making, do the algorithms need to be adjusted to compensate for such biases? Furthermore, since recommender systems use a feedback loop based on consumer purchase decisions, do recommender systems need to be calibrated to handle biased input? Second, biases in decision-making based on online recommendations can potentially be used to the advantage of e-commerce companies, and retailers can potentially become more strategic in their use of recommender systems as a means of increasing profit and marketing to consumers. Third, consumers may need to become more cognizant of the potential decision making biases introduced through online recommendations. Just as savvy consumers understand the impacts of advertising, discounting, and pricing strategies, they may also need to consider the potential impact of recommendations on their purchasing decisions.
This work is supported in part by the National Science Foundation grant IIS-0546443.