=Paper=
{{Paper
|id=Vol-1679/paper6
|storemode=property
|title=Complements and Substitutes in Product Recommendations: The Differential Effects on Consumers' Willingness-to-pay
|pdfUrl=https://ceur-ws.org/Vol-1679/paper6.pdf
|volume=Vol-1679
|authors=Mingyue Zhang,Jesse Bockstedt
|dblpUrl=https://dblp.org/rec/conf/recsys/ZhangB16
}}
==Complements and Substitutes in Product Recommendations: The Differential Effects on Consumers' Willingness-to-pay==
https://ceur-ws.org/Vol-1679/paper6.pdf
Complements and Substitutes in Product
Recommendations: The Differential Effects on
Consumers’ Willingness-to-pay
Mingyue Zhang Jesse Bockstedt
Tsinghua University Emory University
Beijing, China Atlanta, GA
zhangmy.12@sem.tsinghua.edu.cn bockstedt@emory.edu
ABSTRACT product on its own webpage, consumers are often exposed to
Product recommendations have been shown to influence additional relevant products as recommendations, such as on
consumers’ preferences and purchasing behavior. However, Amazon.com. It is an open research question whether consumer’s
empirical evidence has yet to be found illustrating whether and how purchase decisions such as willingness-to-pay for the focal product
the recommendations of other products affect a consumers’ would be affected by the display of ‘other products’ and the type of
economic behavior for the focal product. In many e-commerce information presented with these recommendations. Some work
websites, a product is presented with co-purchase and co-view has studied this in offline settings [31], but little work addresses this
recommendations which potentially contain complement and issue in the online recommendation context.
substitute products, respectively. Very little research has explored Particularly, the types of ‘other products’ in a recommendation set
the differential effects of complementary and substitutable may vary, but they can be generally categorized into substitutes and
recommendations. In this study, we are interested in how the type complements [19]. Substitutes are products that can be purchased
of recommendations of other products impact the consumers’ instead of each other, while complements are products that
willingness-to-pay for the focal product, and additionally how the experience joint demand. For example, when a user is evaluating a
recommendations’ price and the consumers’ decision stage cellphone, it’s reasonable to recommend other phones to better
moderate this effect. We conducted a 2x2x2 randomized match his/her needs, but it also makes sense to recommend batteries,
experiment to examine how the consumers’ willingness-to-pay is chargers, or cases, which commonly make up a bundle
affected by these factors. Experimental results provide evidence recommendation [34]. Research has shown that consumers factor
that there is no significant main effect difference between into consideration the source or type of information when making
complementary and substitutable recommendations. But we their purchase decisions [26]. Economic theory suggests that
observed a significant interaction effect between recommendation complements increase demand for the focal product because of
type and decision stage, which highlights the importance of timing increasing the possibility of users finding added value for the focal
in recommender systems. Other findings include that consumers product [31]. With the increased demand, the market price will
are willing to pay more for a specific product when the price of a increase accordingly, leading to higher individual willingness-to-
recommended product is high, as well as when they are in later pay. Whereas, substitutes decrease demand for the focal product
decision stages. These findings have significant implications for the due to competition, which leads to a lower market price and
design and applications of recommender systems. individual willingness-to-pay. Despite the extensive literature
about complements and substitutes in economics, little research
Keywords studied their differential effects in online recommendation settings.
recommender systems; complements; substitutes; willingness-to-
pay; price; decision stage In this study, we are interested in how the type of recommendations
of other products impact the focal product, and additionally how
1. INTRODUCTION the recommendations’ price and consumers’ decision stage
Recommender systems (RS) are becoming integral to how moderate this effect. The price of a recommended product can be
consumers discover new products and have a strong influence on perceived as a contextual reference point. Along with the nature of
what consumers buy and view. For instance, 60% of Netflix content cross-price elasticity of demands between complementary/
consumption originates from recommendations, and 35% of substitutable goods [20], the research question of how the price
Amazon sales are attributed to recommendations [11]. With their interacts with recommendation type is of great value. Additionally,
utmost importance for retailers, some studies have been conducted when shopping online consumers tend to have a two-stage decision
to explore the behavioral effects of recommender systems on making process [9]: first, screening a large set of available products
consumers [1][2]. Specifically, prior studies found that consumers’ to identify a subset of the most promising alternatives; and second,
preferences and willingness-to-pay for a product can be influenced evaluating the latter in more depth to make a final purchase
by the values of personalized recommendations. This provides decision. Zheng et al. (2009) [33] argue that consumers prefer
evidence that consumers’ behavior is vulnerable toward different recommendations in each stage because they are driven by
recommendation agents. However, there is still space for different goals in each stage, i.e., in stage 1 they are comparing
researching the behavioral effects of detailed recommendation alternatives, whereas in stage 2 they are reviewing candidates. For
features. For example, when evaluating the information of a focal this reason, we are also interested in the moderation role of decision
stages, which has not been previously studied.
IntRS 2016, September 16, 2016, Boston, MA, USA.
Copyright remains with the authors and/or original copyright holders.
In the following sections, we firstly introduce the theoretical protector or other accessories. Figure 1 and Figure 2 provide an
underpinnings of this research, based on which five hypotheses are example to this extent with the co-purchase and co-view
proposed. Then we discuss the design of a randomized experiment, recommendations on Amazon.com when the focal product is ‘Dell
which measures consumers’ willingness-to-pay across different Inspiron 15 i5558-5718SLV’.
recommendation scenarios. We present the results of our analysis
and discuss the implications for online retail practices and research
involving recommender systems. The investigation provides a new
angle for understanding the behavioral aspect of recommender
systems, as well as guidelines to further improve the design of
recommendation agents.
2. THEORETICAL FRAMEWORK
In this section, we discuss the relevant theoretical foundations for
our research questions in terms of three dimensions: Figure 1. Amazon co-purchase product recommendation
recommendation type, price, and consumers’ decision stage. Based
on our primary goal of uncovering the differential effects between
different recommendation types, we firstly review research about
complementary/substitutable goods in the marketing literature, as
well as related empirical studies in the recommender system
context. Second, since the basic relationship between
complements/substitutes is their cross-price elasticity of demands,
we discuss theories describing how one product’s price might
influence consumers’ willingness-to-pay for another product.
Finally, we discuss the related literature on consumer decision
making processes and how this interacts with the recommender
systems. Figure 2. Amazon co-view product recommendation
2.1 Complements and Substitutes in By definition in microeconomics, if product A and B are
complements, increased demand for product A should be
Recommendation associated with increased demand for product B [33]. This
The study of complements and substitutes has long been a central complementary product effect leads to the co-purchase network.
subject in the marketing literature. Generally, products are On the other hand, substitute products have an inverse demand
considered complements (substitutes) if lowering (raising) the price relationship. This leads to the co-view network because consumers
of one product leads to an increase in sales of another [31]. tend to view and compare substitutes before making final purchase
Research shows that consumer choice is easily influenced by decisions. Given that recommendations of both complements and
context and the set of alternatives available at the time of decision substitutes are often presented along with the focal product, it is of
[24], thus, there is significant demand interrelationship among significant practical interest to understand their differential effects.
substitutable and complementary goods [20]. Generally, two Based on economic theory, complements increase demand for the
moderate or strong substitutes should be offered separately, focal product, and the market price will increase accordingly,
whereas two complements should be offered as a bundle [32], in leading to higher individual willingness-to-pay. Whereas,
order to maximize the profit. This is because the introduction of a substitutes decrease demand for the focal product due to
complement may increase the possibility of buyers finding new competition, which leads to a lower market price and individual
uses or added value for existing products [31], whereas the willingness-to-pay. Thus, we put forth the following hypothesis:
substitutes can be consumed or used in place of one another.
H1: Consumers tend to have higher willingness-to-pay for the focal
In the online recommendation scenario, a focal product is often product when it is displayed together with a complementary
presented with several related items as the recommendations. Take recommendation as compared to being displayed with a
Amazon.com as an example, each product is featured on its own substitutable recommendation.
designated webpage, along with additional relevant products as
recommendations. Hence, a visible directed product network is Note that this initial hypothesis is intended to test a main effect of
created whereby products are explicitly connected by hyperlinks recommendation type and is price agnostic. We address the
[16]. Some studies [22][23][5] have examined the behavioral moderating effects of price in the next section.
impacts of recommendation networks, with these studies primarily
focusing on the co-purchase recommendation network. Many e- 2.2 Pricing
commerce websites provide recommendations from two product Researchers in marketing and economics have long recognized that
networks: co-view and co-purchase product networks. Only pricing decisions sometimes incorporate more than one product.
recently have their differential effects been considered [16]. More This is because consumers tend to respond to price relative to some
interestingly, co-purchase and co-view networks can be used to reference price [25], such as the other prices in the store at the point
implicitly represent two recommendation strategies, that is, of purchase. Both prospect theory and mental accounting suggest
recommending complementary and substitutable products, that consumers make decisions based on losses or gains relative to
respectively. Although not always the case, it is common that co- a reference point. When consumers compare the actual price of the
purchased items contain complementary products while co-viewed focal product with other reference prices, incidental price learning
products contain substitutable products. For example, a consumer [21] occurs without any explicit intention to memorize them. In
buy a laptop computer may view several laptops, but purchase only offline physical stores, retailers can attempt to influence positively
one laptop along with a complementary mouse, software, screen the degree to which the sales of one item affect sales of other items
through in-store product locations and shelf space allocations, for shopping goals in the first stage of a shopping process. Thus, their
example, locating two complements together. This is very similar thinking is more abstract when in the first stage. Shopping goals
to the online recommendations where other products are co- become concrete when they are closer to the final purchase point in
displayed with prices along with the focal product on the webpage. the second stage. Therefore, marketing promotions for similar
When consumers evaluate the focal product, the prices of products (i.e., substitutes) are more effective in influencing
recommended products are expected to affect their purchase- consumer’s spending when their goals are less concrete [10][14].
related decisions for the focal product. Contrast effect theory [30] Researchers have also studied the behavioral effects of
suggests that the perceived value of the focal product’s price is recommendations beyond standard substitute recommendations.
decreased (increased) when the recommendations presented along For instance, Zheng et al. (2009) [33] argued that customers prefer
with have relatively higher (lower) price. That is, consumers are different types of recommendations in different purchase stages. In
willing to pay more (less) when the product seems cheaper (more the first stage of an online purchase process, customers are
expensive) relative to other products. Thus, we have the following navigating webpages to compare a large set of similar products.
hypothesis about the main effect of price on willingness-to-pay: Whereas in the second stage, customers already have a clear
candidate set through which to make a purchase decision. In the
H2: The price of a recommended product has a positive influence second stage, substitutive recommendations likely have little
on consumers’ willingness-to-pay for the focal product, that impact and recommendations of complement products may be
is, consumers are willing to pay more when the price of the preferred since they introduce items that can add value to the
recommended product is higher than the price of focal purchase of the focal product. Hence, we hypothesize the following
product. interaction effect:
There are also significant cross-relationships among the sales of H3A: There is an interaction effect between the stage and type of
substitutable and complementary products [20]. In particular, lower recommendations, such that the stage has a positive effect on
price or promotion of one product can stimulate sales of a
willingness-to-pay for the focal product when the
complement, whereas supplant sales of other substitutes. That is to recommendation type is complement and it has a negative
say, when the prices of complementary goods go up, the purchase effect on willingness-to-pay for the focal product when
likelihood for the other complementary good may go down, while recommendation type is substitute.
if the price of one of the substitute goods goes up, the purchase
likelihood for others will go up [12]. Furthermore, by influencing 3. EXPERIMENTS
the demand of a complementary/ substitutive product through its Recommendations on Amazon.com and other platforms generally
price, the consumers’ willingness-to-pay for that product will be fall into the complement and substitute product types through the
influenced as well [18]. This cross-relationship nature of substitutes co-purchase and co-view lists. However, this is not always the case,
and complements indicates that the effect of price is stronger when and other contextual factors and user self-selection can impact the
the competition between two products is high, which is common effect and content of these recommendations. Therefore, to
among substitutes. Therefore, we hypothesize the following: eliminate these confounding factors and conditions that naturally
H2A: There is an interaction effect between the price and type of occur in the field, we designed a randomized controlled experiment
recommendations, such that the positive influence of so that the recommendation type, recommendation price, and
recommendation price on willingness-to-pay for the focal decision stage can be cleanly manipulated. This controlled and
product is stronger when the recommendation type is randomized treatment approach allowed us to test our hypotheses
substitute as compared to complement. and make causal inferences.
2.3 Consumers’ Two-Stage Decision Making 3.1 Experiment Design and Participants
As illustrated in the previous literature [9][17][29], consumers are Our hypotheses express the main effects of each of three main
often not capable of evaluating all available alternatives in great factors (recommendation type, price of recommended product, and
depth, and this results in a two-stage decision making process. In decision stage) as well as two two-way interaction effects (price x
the first stage, consumers usually browse a large set of available type and stage x type) on willingness-to-pay for a focal product.
options and identify a small subset of candidates for further Since the focus of the study is on the effects of complementary
consideration. In the second stage, they tend to thoroughly evaluate versus substitutable recommendations, we did not hypothesize the
the candidates and make a final purchase decision. In the second interaction between recommendation price and decision stage.
decision stage consumers’ motivation and determination to make Additionally, since the three-way interaction among these factors
purchases are increased, thus having higher willingness-to-pay for is complex and no prior theory provides insights to this regard, this
selected alternatives. Hence, we hypothesize the following main interaction was also not hypothesized.
effect of decision stage: A factorial experiment was used to test our hypotheses efficiently.
H3: The decision stage has a positive influence on consumers’ Specifically, a 2 (types of recommendation: complements vs.
willingness-to-pay for the focal product, that is, consumers substitutes) x 2 (recommended products’ price relative to the focal
are willing to pay more when they are in the second stage. product’s price: low vs. high) x 2 (decision stage: stage 1 vs. stage
2) full-factorial design was used, which results in 8 treatment
Given this multistage mental process, Ge et al. (2012) [8] argued conditions. Although, the full factorial provides the opportunity to
that the manner in which information is processed differs test the three-way interaction and the price x stage interaction, our
systematically between the two decision stages. Their experimental analysis focuses only on the main effects and interactions identified
results reveal that the timing of the presentation of specific pieces in our hypotheses. The advantage of factorial experiment designs
of information about an alternative across shopping stages has a over randomized controlled trails (RCTs) is that they provide more
great impact on consumers’ choice. This difference can be statistical power with fewer participants. Generally, the objective
attributed to the shopping goals theory [14] and construal level of RCT is to compare the individual experimental conditions to
theory [15]. Specifically, consumers are less certain of their each other directly, while in a factorial experiment the
combinations of experimental conditions are compared, i.e., the (i.e., consider-then-choose) adapted directly from [8], which the
main effects and interactions. participants followed prior to measuring dependent variables.
Participants were randomly assigned to one of two decision stage
We manipulated the three factors between subjects, who were manipulations: complete the first and second stage of the shopping
undergraduates from a business school in a large public university procedure before being shown the focal product page or complete
in North America. Subjects received extra credit for their
only the first stage of the shopping procedure before being shown
participation in the experiment. We performed a power analysis the focal product page. In all treatment groups, the focal product
with the assumption that the effect size of our model will be and its posted price and description remained the same.
medium, i.e., 𝐶𝑜ℎ𝑒𝑛
𝑓 ( = 0.15 (Cohen, 1988). To achieve power
(1-𝛽) of 0.80 and a medium effect size, as well as maintain a 3.2 Stimuli and Procedures
significance level (α) at 0.05, the minimum sample size for a model Participants were first instructed with a cover story that they were
with three main effects and two interactions is 92 (the calculation participating in research focusing on consumers’ preferences and
was made by using the package ‘pwr’ in R). purchase behavior. They were also told that there were no right or
We published a web link for our online experiment to a large wrong answers. Following these initial instructions, participants
undergraduate class containing approximately 400 students. 261 were randomly assigned to one of the eight treatment groups.
students clicked on the link to initiate the study. Participants were Before the main task of the experiment, participants were asked to
randomly assigned to one of the eight treatment conditions. The answer some basic questions about their opinions on electronic
median time of completion is 12 minutes. We dropped observations products and were asked to rate several different electronic product
for 126 participants for the following reasons: not completing the categories. Participants were told that their answers to these
study, completing the experiments in an extremely short time (e.g., questions would be used later by our system to predict their
less than 4 minutes), completing the study in very long time (e.g., preferences and make personalized recommendations for them.
more than 4 hours indicating the study was started, stopped, and This pre-experiment task was used to eliminate their doubt about
started again later), and not passing manipulation checks. the basis of recommendations in later steps.
Participants were informed that multiple manipulation checks
would be used to determine if they took the study seriously, which In the main task of the experiment, subjects were asked to shop for
would then impact whether they received extra credit for their a computer mouse and make a purchase decision. We implemented
participation. Since we collected the data as an online survey the two-stage shopping decision process based on the methodology
instead of bringing students to a laboratory, it is a common used in the marketing literature, (e.g., [8]). In the first stage process,
phenomenon that response and completion rates are relatively low participants were presented with descriptions of 12 computer mice
[3]. As a result, 135 valid observations were left. The distribution as search results on the e-commerce store. They were asked to
of the valid observations across treatment groups is shown in Table browse and evaluate all the product information presented. The
1. As can be seen in Table 1, the randomly assigned treatments are ‘Next Page’ button appeared only after 30 seconds had elapsed, as
evenly distributed among participants. means of preventing participants from moving ahead too quickly,
without reviewing the stage 1 products. Manipulation check
Table 1. Experimental design and sample sizes per group questions were also asked to check their impressions about these
complements substitutes initial 12 mice. In the second stage, we narrowed down the choice
set to 2 mice and participants were asked to evaluate the two items
Stage 1 Low 16 17 and pick one of them as their final purchase choice. Participants
Stage 1 High 17 18 who were randomly assigned to the groups with condition ‘stage 1’
Stage 2 Low 17 16 would only go through the first stage (i.e., browsing information).
Stage 2 High 17 17 Comparably, those who were randomly assigned to groups with
condition ‘stage 2’ would go through both the first and the second
Experiment participants were put in the scenario of purchasing a stage.
new computer mouse on an e-commerce site like Amazon.com. We
chose a mouse because it is very common for consumers to
purchase electronic products online, and a computer mouse has a
large number of potential complements and substitutes with both
low and high prices.
For the first manipulation factor (i.e., type of recommendation),
participants were randomly assigned to one of two different
shopping interfaces: the focal product page with recommendations
of complementary products, or the focal product page with
recommendations of substitute products. The pages included
product descriptions directly from Amazon.com. We omitted brand
information in the descriptions to eliminate any brand bias. Note
that both the complementary and substitutable goods derived from
real recommendations from the website. For the second
manipulation factor, (i.e., price of the recommended products),
participants were randomly assigned to either a high or low price
condition. In the high price condition, the recommended products Figure 3. Example Screenshot for the experimental interface
were higher in price than the focal product and in the low price (i.e., substitutes recommendation with low price)
condition the opposite was true. For the third manipulation factor
(i.e., decision stage), we designed a two-stage shopping procedure
After the stage manipulation, participants viewed a specific focal (1) Do you think the products in the section titled ‘We think you
mouse product page. Along with the focal mouse, may also like these items’ are complements to the mouse you
recommendations were presented according to the random evaluated? and (2) Do you think the products in the section titled
treatment condition. Figure 3 provides example screenshots of the ‘We think you may also like these items’ are substitutes to the
recommendation interface. In the groups with ‘low (high) price’ mouse you evaluated?. In terms of the decision stage manipulation
condition, all the recommendations’ prices were slightly lower check, we did not directly ask subjects’ perceptions about decision
(higher) than the price of the focal mouse. Subjects could click on stage because this may be an incomprehensible terminology.
the recommendation to view a detailed description. The number of Instead, we asked them ‘In the previous task you just finished,
clicks and duration on each webpage were also recorded. which procedure(s) have you been through?’, and provided the
following possible responses: (1) Evaluating a large set of
After viewing the focal product page based on their treatment alternative products as if you were gathering information in early
condition, participants were asked to provide their willingness-to- stages of shopping and (2) Evaluating a small set of alternative
pay for the focal product. Upon completing the shopping task, products as if you were trying to choose a final one to purchase.
participants responded to a set of manipulation check questions and
An additional question was used to check participants’ perception
completed a short survey with demographic questions that we use about the relative price: ‘What do you think of the price level of the
as control variables in our analysis (age, gender, level of education, mouse you just evaluated?’.
computer experience, web experience, e-commerce experience,
familiarity with and attitudes toward recommender systems). First, to check if participants consciously distinguished between
complement and substitute recommendations, we compared their
3.3 Dependent Measure responses toward the two manipulation check questions about
Willingness-to-pay is the maximum amount an individual is willing recommendation type. They responded with the following 5 claims:
to sacrifice to procure a product. Here we adopted the method used “Definitely yes” (coded as 5), “Probably yes” (coded as 4),
by Rucker & Galinsky (2008), Rucker et al. (2014) and Kim & Gal “Maybe” (coded as 3), “Probably not” (coded as 2), and “Definitely
(2014) [13][27][28] to measure willingness-to-pay. Participants not” (coded as 1). As expected, participants in the complements
indicated their willingness-to-pay using a sliding scale where they group perceived the recommendations as complements
could choose from 0% to 120% of the retail price. The interval (i.e., ( 𝑀01234525678 = 4.15, 𝑀8;<8=7;758 = 1.61, 𝑡 133 = 16.17, 𝑝 < 0.001 ),
0%-120%) is used to reduce the amount of response variance and and not as substitutes ( 𝑀01234525678 = 1.42, 𝑀8;<8=7;758 =
to guard against outliers. We are interested in relative changes in 4.68, 𝑡 133 = −28.87, 𝑝 < 0.001). The extremely low p-values of
willingness-to-pay due to treatment effects and not in estimating these tests help guard against any potential multiple comparison
point estimates of willingness-to-pay for specific products, thus the issues. These results support the validity of our manipulation for
interval willingness-to-pay metric is sufficient. Furthermore, recommendation types. Further, for the stage check question,
because the market price for the focal product was given, it’s not participants in different stage conditions correctly perceived their
realistic to use the Becker–DeGroot–Marschak approach [7] or decision stages ( 𝑀87GH5I = 1.19, 𝑆𝐷 = 0.39, 𝑀87GH5( = 1.97, 𝑆𝐷 =
second-price auctions to elicit willingness-to-pay. Figure 4 shows 0.17, 𝑡 133 = −14.81, 𝑝 < 0.001 ). Finally, a successful
the interface for entering willingness-to-pay: manipulation check was observed for the price. Due to the contrast
effect, people in the high price recommendation condition felt the
price of focal product is lower than those assigned in the low price
recommendation condition ( 𝑀M=HM = 2.96, 𝑆𝐷 = 0.57, 𝑀41N =
3.33, 𝑆𝐷 = 0.68, 𝑡 133 = −3.45, 𝑝 < 0.001).
4.2 Main Results
Table 3 shows the mean and standard deviation of the willingness-
Figure 4. Entering willingness-to-pay to-pay, measured as a percentage (0%-120%) of the focal product’s
original price, in each of the eight treatment groups.
4. RESULTS
Table 2 provides summary statistics on the demographic items Table 3. Mean (SD) Willingness-to-pay (%) in Each Group
collected in our post-experiment survey. Decision stage Price complements substitutes
Table 2. Demographic Summary Statistics Stage 1 Low 68.19 (15.86) 79.29 (17.66)
Control variables Summary Stage 1 High 87.65 (13.50) 93.78 (21.81)
Age Mean: 21.6, SD: 3.35 Stage 2 Low 88.29 (14.29) 81.63 (16.04)
Gender 50.37% --- female Stage 2 High 93.94 (10.87) 89.53 (13.14)
Primary language 88.89% --- native English speaker To test the proposed hypotheses, we need to make comparisons
Experience with 88.15% --- spend more than 4 hours per between combinations of groups – the main and interaction effects.
Internet day on the Internet Since there are three manipulated factors in the experiment, we
Experience with e- 77.04% --- browse e-commerce websites started by conducting a three-factor Analysis of Variance
commerce more frequently than once a week (ANOVA) and the results are presented in Table 4. The results
Familiarity with RS 85.19% --- familiar with RS reveal that the main effect of stage and price, as well as the
Attitude toward RS 82.22% --- RS is helpful for finding interaction between recommendation type and stage are significant.
relevant items We also conducted orthogonal contrast analysis, which provided
consistent results and is omitted due to space constraints. Details
4.1 Manipulation Checks can be obtained by contacting the authors directly.
In order to check the saliency of our recommendation type, two
questions were asked of participants in the post-experiment survey:
Table 4. Results of Three-factor ANOVA
Df SSE MSE F value Pr(>F)
Type 1 73 73 0.279 0.5981
Price 1 4670 4670 17.786 4.66e-05 ***
Stage 1 1200 1200 4.570 0.0345 *
Type : Price 1 9 9 0.036 0.8505
Type : Stage 1 1688 1688 6.430 0.0124 *
Price : Stage 1 872 872 3.321 0.0708 +
Residuals 127 33343 263
Figure 6. Interaction effects (Left: type × price; Right: type × stage)
Significance levels: + 𝑝 ≤ 0.1, * 𝑝 ≤ 0.05, ** 𝑝 ≤ 0.01, *** 𝑝 ≤ 0.001.
Furthermore, to get the effect size of our model as well as
Figure 5 displays the average willingness-to-pay under the coefficients of each factor, we estimated a sequential linear model.
combined conditions. Specifically, there is no significant difference Firstly, we regressed the willingness-to-pay on a set of control
between groups with complement and groups with substitute variables, including gender, preference to the focal product,
recommendations ( 𝑀01234525678 = 84.76, 𝑆𝐷 = 16.67, 𝑀8;<87=7;758 = experience with e-commerce, familiarity with and attitude toward
86.24, 𝑆𝐷 = 18.85, 𝐹 = 0.279, 𝑝 = 0.598). However, when the prices recommender systems. After that, we included the five independent
of recommended products are relatively high, participants’ variables of interests to the model. The type factor has two levels,
willingness-to-pay is much higher than that in relatively low prices either complements (0) or substitutes (1), the price factor is either
condition( 𝑀41N = 79.48, 𝑆𝐷 = 17.48, 𝑀M=HM = 91.26, 𝑆𝐷 = 15.75, 𝐹 = low (0) or high (1), and the stage factor is either stage 1 (0) or stage
17.786, 𝑝 < 0.001), which supported H2. Similarly, the difference 2 (1). The regression results of these two models are shown in Table
between conditions in stage 1 and stage 2 was in the expected 5, and the R-square increased 0.149 after including these five
directions ( 𝑀87GH5I = 82.60, 𝑆𝐷 = 19.99, 𝑀M=HM = 88.45, 𝑆𝐷 = 14.26, variables. Consistent with the ANOVA results, we got significant
𝐹 = 4.570, 𝑝 < 0.05), thus supporting the hypothesis that consumers positive coefficients for price and stage, indicating that consumers
are willing to pay more when they are in the second decision stage have higher willingness-to-pay in the high price condition
(H3). Further, we calculated Cohen’s d to capture the effect size. (compared to low price) and stage 2 condition (compared to stage
Cohen’s d is known as the difference of two population means and 1). In addition, the interaction effect between type and stage is also
divided by the standard deviation from the data. The effect size for marginally significant at level α = 0.1. The coefficients in the table
price and stage factors are 0.669 and 0.372, which indicate suggest that consumers shown a recommended product with high
medium-to-large and small-to-medium effect, respectively. price reported 10.353% higher willingness-to-pay in terms of the
retail price. Similarly, consumers in the second stage reported
10.832% higher willingness-to-pay in terms of the retail price than
those in the first stage.
Table 5. Results of the Linear Regression Models
Dependent variable: Willingness-to-pay (%)
Model 1: Control Model 2: Full model
(𝑅 ( = 0.1699) (𝑅 ( = 0.3189)
Coefficient T P- Coefficient T P-
(SE) statistic value (SE) statistic value
Intercept 53.166 5.088 1.25e- 44.589 4.521 1.42e-
(10.450) 06*** (9.862) 05***
Type 5.572 1.154 0.251
(4.828)
Price 10.353 2.747 0.007
(3.769) **
Figure 5. Average Willingness-to-pay in Combined Conditions Stage 10.832 2.703 0.008
(4.008) **
For the interaction effects, Figure 6 demonstrates the difference of Type * 2.279 0.427 0.670
mean values for complements and substitutes groups under Price (5.336)
different price levels and decision stages, respectively. The left Type * -10.729 -1.885 0.062
Stage (5.691) +
figure shows no interaction between type and price, since both
Preference 7.141 4.640 8.45e- 6.232 4.178 5.50e-
complements and substitutes groups have higher willingness-to- (1.539) 06*** (1.492) 05***
pay in high relative price conditions ( 𝐹 = 0.036, 𝑝 = 0.851 ). The Gender 3.255 1.056 0.292 3.803 1.309 0.193
crossing lines in right figure indicate significant interaction effect (3.081) (2.905)
between type and stage ( 𝐹 = 6.430, 𝑝 < 0.05 ). Particularly, the Experience -1.126 -0.697 0.487 -1.054 -0.690 0.492
effect of decision stage is much stronger when the (1.616) (1.527)
Familiarity 0.600 0.388 0.699 1.412 0.947 0.345
recommendations are complements ( 𝑀87GH5I = 78.21, 𝑆𝐷 =
(1.548) (1.491)
17.62, 𝑀87GH5( = 91.12, 𝑆𝐷 = 12.28, 𝑡 65 = −3.29, 𝑝 < 0.001 ), while Attitude -6.362 -2.098 0.0378 -6.924 -2.467 0.015
it’s not significant under substitute conditions ( 𝑀87GH5I = (3.032) * (2.807) *
86.74, 𝑆𝐷 = 21.18, 𝑀87GH5( = 85.70, 𝑆𝐷 = 15.14, 𝑡 66 = 0.23, 𝑝 = Significance levels: + 𝑝 ≤ 0.1, * 𝑝 ≤ 0.05, ** 𝑝 ≤ 0.01, *** 𝑝 ≤ 0.001.
0.41 ). Therefore, our hypothesis of interaction effect H3A is
partially supported, and H2A is not supported. The effect size of our sequential multiple regression model is
V
STU WSTV
calculated by Cohen’s 𝑓 ( . It is defined as 𝑓 ( = V , where 𝑅X(
IWSTU
(
is the variance accounted for by a set of control variables 𝐴 and 𝑅XZ
is the combined variance accounted by 𝐴 and another set of recommendations against different purchase stages, as well as
independent variables of interest 𝐵. Here in our model, we have highlighting the importance of timing in recommender systems.
𝑅X( = 0.1699 and 𝑅XZ (
= 0.3189 , resulting in a medium-to-large The price of recommended products was also found to have
effect size of 𝑓Z( = 0.219 . We also conducted a post-hoc power significant effects on willingness-to-pay. Serving as a reference
analysis and with the 135 observations and the calculated effect point, the prices of recommendations may be compared with the
size, the power of our model is 0.993 while maintaining the retail price of the focal product, which could cause consumers to
significance level at 0.05. This provides evidence that the null adjust their willingness-to-pay through incidental price learning.
effects are true and not the result of a lack of power. Under the condition with high recommendation prices, consumers
tend to have higher willingness-to-pay for the focal product and
Since we measured the willingness-to-pay by restricting vice versa. This positive effect is significant no matter the type of
participants’ choice from 0% to 120% of the stated retail price, it recommendation for other products.
may result in censored and non-normal data. Therefore, we
performed robustness check that removed the normality 5.2 Theoretical Contributions
assumption. We conducted two non-parametric tests and ran a Our research offers important theoretical contributions in the
Tobit regression model using both dummy coding (0,1) and effect following ways. First, studies on product recommendations have
coding (-1,1). The coefficients and significance levels are focused on the consumers’ different preferences and behaviors for
consistent with our baseline analysis. Due to space constraints, one products in the presence of recommendations [1][2]. This paper
details of the robustness check are omitted. extends the behavioral research on recommender systems by
studying the question whether recommending ‘other products’ on
5. DISCUSSION AND CONCLUSIONS the same webpage had an effect on consumers’ willingness-to-pay
5.1 Summary of Findings for the focal product. Second, prior research has not paid much
In this paper, we conducted a randomized experiment to examine attention to different types of recommendations. Deriving from
the impact of complement and substitute recommendations on economics literature, two typical relationships between products
consumers’ willingness-to-pay for the focal product. Table 6 are examined, that is, complements and substitutes. Third, our
summarizes our findings which corresponds to the proposed research is one of the few studies that examine the detailed
hypotheses. recommendation features, i.e., price of recommended products as
well as consumers’ decision stage. Integrating the consumers’
Table 6. Hypotheses and Results
decision process, we have a better understanding of the behavioral
Hypotheses Results aspects of recommendations in online purchases.
H1: Consumers tend to have higher willingness-to-pay Not
about the focal product when it is displayed together Supported 5.3 Implications for Practice
with a complementary recommendation as compared to Beyond contributing to advancing the academic literature, our
being displayed with a substitutable recommendation.
findings also have significant practical implications and may guide
H2: The price of a recommended product has a positive Supported the platform’s recommendation strategy. The vulnerability of
influence on consumers’ willingness-to-pay for the
focal product, that is, consumers are willing to pay more
consumers’ willingness-to-pay indicates the importance of ‘other
when the price of the recommended product is higher products’ in recommender systems. This suggests new possibilities
than the price of focal product. for influencing product sales by manipulating the contextual
H2A: There is an interaction effect between the price Not information of recommendations. Another important implication is
and type of recommendations, such that the positive supported about the timing of recommendations, i.e., complementary
influence of recommendation price on willingness-to- recommendations should be delayed to the second decision stage.
pay for the focal product is stronger when the
recommendation type is substitute as compared to 5.4 Future Work
complement. The main limitation of this study is that we are not observing real
H3: The decision stage has a positive influence on Supported world purchases. In contrast, however, an advantage is that our
consumers’ willingness-to-pay for the focal product,
controlled randomized experiment allows us to make causal
that is, consumers are willing to pay more when they are
in the second stage.
inferences, and thus trading external validity for identification.
H3A: There is an interaction effect between the stage Partially Future research can be developed by exploring other factors
and type of recommendations, such that the stage has a Supported associated with recommended products, such as average ratings,
positive effect on willingness-to-pay for the focal quality, pictures of complements/substitutes and so on.
product when the recommendation type is complement Additionally, we can use observational data to empirically validate
and it has a negative effect on willingness-to-pay for the the findings of our experiment. By examining the relationships
focal product when recommendation type is substitute. between recommendation network properties and products’ sales,
Experimental results provide evidence that there is no significant we will have additional support for the influence of complementary
main effect difference between complementary and substitutable and substitutable product recommendations on consumers’
recommendations on consumers’ willingness-to-pay for the focal economic behavior from an aggregate level.
product. We further investigated two factors that commonly present
with recommendations: decision stage and the price of
6. REFERENCES
[1] Adomavicius, G., Bockstedt, J.C., Curley, S.P. and Zhang, J.
recommended products. We found that consumers are willing to
2014. Suggest or Sway? Effects of Online Recommendations
pay more for a specific product as decision stage increases. An
on Consumers’ Willingness to Pay. (2014). Working paper.
interesting finding is the interaction between recommendation type
and decision stage. The positive effect of stage vanished when the [2] Adomavicius, G., Bockstedt, J.C., Curley, S.P. and Zhang, J.
recommendation is substitute to the focal product, while it is very 2013. Do recommender systems manipulate consumer
significant with complementary recommendations. This is preferences? A study of anchoring effects. Information
consistent with previous findings that customers prefer different Systems Research. 24, 4 (2013), 956–975.
[3] Baruch, Y. and Holtom, B.C. 2008. Survey response rate Conference on Knowledge Discovery and Data Mining
levels and trends in organizational research. Human (2015).
Relations. 61, 8 (2008), 1139–1160. [20] Mulhern, F.J. and Leone, R.P. 1991. Implicit Price Bundling
[4] Cohen, J. 1988. Statistical Power Analysis for the Behavioral of Retail Products: A Multiproduct Approach to Maximizing
Sciences (second ed.). Lawrence Erlbaum Associates. Store Profitability. Journal of Marketing. 55, 4 (1991), 63–
[5] Dhar, V., Geva, T., Oestreicher-singer, G. and Sundararajan, 76.
A. 2014. Prediction in Economic Networks. Information [21] Nunes, J.C. and Boatwright, P. 2004. Incidental Prices and
Systems Research. 25, 2 (2014), 264–284. Their Effect on Willingness to Pay. Journal of Marketing
[6] Donaldson, C., Jones, A.M., Mapp, T.J. and Olson, J.A. Research. 41, 4 (2004), 457–466.
1998. Limited dependent variables in willingness to pay [22] Oestreicher-Singer, G. and Sundararajan, A. 2012a.
studies: applications in health care. Applied Economics. 30, 5 Recommendation networks and the long tail of electronic
(1998), 667–677. commerce. MIS Quarterly. 36, 1 (2012), 65–83.
[7] Frederick, S. 2012. Overestimating Others’ Willingness to [23] Oestreicher-Singer, G. and Sundararajan, A. 2012b. The
Pay. Journal of Consumer Research. 39, 1 (2012), 1–21. visible hand? Demand effects of recommendation networks
[8] Ge, X., Häubl, G. and Elrod, T. 2012. What to Say When: in electronic markets. Management Science. 58, 11 (2012),
Influencing Consumer Choice by Delaying the Presentation 1963–1981.
of Favorable Information. Journal of Consumer Research. [24] Payne, J.W., Bettman, J.R. and Johnson, E.J. 1992.
38, 6 (2012), 1004–1021. Behavioral decision research: a constructive processing
[9] Häubl, G. and Trifts, V. 2000. Consumer decision making in perspective. Annual Reviews of Psychology. 43, 1 (1992),
online shopping environments: The effects of interactive 87–131.
decision aids. Marketing Science. 19, 1 (2000), 4–21. [25] Rajendran, K.N. and Tellis, G.J. 1994. Contextual and
[10] Ho, S.Y., Bodoff, D. and Tam, K.Y. 2011. Timing of temporal components of reference price. Journal of
Adaptive Web Personalization and Its Effects on Online Marketing. 58, 1 (1994), 22–34.
Consumer Behavior. Information Systems Research. 22, 3 [26] Rao, A.R. and Sieben, W. a. 1992. The Effect of Prior
(2011), 660–679. Knowledge on Price Acceptability and the Type of
[11] Hosanagar, K., Fleder, D., Lee, D. and Buja, A. 2014. Will Information Examined. Journal of Consumer Research. 19, 2
the Global Village Fracture Into Tribes? Recommender (1992), 256–270.
Systems and Their Effects on Consumer Fragmentation. [27] Rucker, D.D. and Galinsky, A.D. 2008. Desire to Acquire:
Management Science. 60, 4 (2014), 805–823. Powerlessness and Compensatory Consumption. Journal of
[12] Jin, R.K.-X., Parkes, D.C. and Wolfe, P.J. 2007. Analysis of Consumer Research. 35, 2 (2008), 257–267.
bidding networks in eBay: Aggregate preference [28] Rucker, D.D., Hu, M. and Galinsky, A.D. 2014. The
identification through community detection. AAAI Workshop Experience versus the Expectations of Power: A Recipe for
- Technical Report. WS-07-09, (2007), 66–73. Altering the Effects of Power on Behavior. Journal of
[13] Kim, S. and Gal, D. 2014. From Compensatory Consumption Consumer Research. 41, August (2014), 381–396.
to Adaptive Consumption: The Role of Self- Acceptance in [29] Russo, J.E. and Leclerc, F. 1994. An eye-fixation analysis of
Resolving Self-Deficits. Journal of Consumer Research. 41, choice processes for consumer nondurables. Journal of
August (2014), 526–542. Consumer Research. 21, September (1994), 274–290.
[14] Lee, L. and Ariely, D. 2006. Shopping goals, goal [30] Sherif, M. and Hovland, C.I. 1961. Social judgment:
concreteness, and Conditional Promotions. Journal of Assimilation and contrast effects in communication and
Consumer Research. 33, June (2006), 60–71. attitude change. (1961).
[15] Liberman, N., Trope, Y. and Wakslak, C. 2007. Construal [31] Shocker, A., Bayus, B. and Kim, N. 2004. Product
level theory and consumer behavior. Journal of Consumer Complements and Substitutes in the Real World: The
Psychology. 17, 2 (2007), 113–117. Relevance of “Other Products.” Journal of Marketing. 68, 1
[16] Lin, Z., Goh, K.-Y. and Heng, C.-S. 2015. The demand (2004), 28–40.
effects of product recommendation networks: an empirical [32] Venkatesh, R. and Kamakura, W. 2003. Optimal Bundling
analysis of network diversity and stability. MIS Quarterly. and Pricing under a Monopoly: Contrasting Complements
(2015). Forthcoming. and Substitutes from Independently Valued Products. The
[17] Liu, Q. and Arora, N. 2011. Efficient Choice Designs for a Journal of Business. 76, 2 (2003), 211–231.
Consider-Then-Choose Model. Marketing Science. 30, 2 [33] Zheng, J., Wu, X., Niu, J. and Bolivar, A. 2009. Substitutes
(2011), 321–338. or Complements: Another Step Forward in
[18] Loomis, J., Gonzalez-Caban, A. and Gregory, R. 1994. Do Recommendations. Proceedings of the 10th ACM conference
Reminders of Substitutes Influence Contingent Valuation on Electronic commerce. (2009), 139–145.
Estimates? Land Economics. 70, 4 (1994), 499–506. [34] Zhu, T., Harrington, P., Li, J. and Tang, L. 2014. Bundle
[19] McAuley, J., Pandey, R. and Leskovec, J. 2015. Inferring recommendation in ecommerce. Proceedings of the 37th
Networks of Substitutable and Complementary Products. international ACM SIGIR conference on Research &
Proceedings of the 21th ACM SIGKDD International development in information retrieval - SIGIR ’14. (2014),
657–666.