Recommendations on e-commerce websites help users discover their interests and assist them in deciding on items to purchase; however, users are prone to bias when comparing and selecting items due to cognitive limitations. The decoy effect, a common user bias phenomenon, has been confirmed in previous studies to induce user selection of items by adding one other item when comparing two items. Although previous studies have confirmed the difference in item selection with and without decoy items in controlled experiments, the mechanism of decoy effect in e-commerce websites has not been elucidated. This study is the first to propose a method for evaluating the decoy effect on real e-commerce websites. We proposed a rowbased decoy effect detection method inspired by users' tendency to compare items in the same row when browsing recommended items on e-commerce websites. In addition, a new metric, called intra-row decoy effect rate, is proposed to evaluate the degree of decoy effect. Our month-long study of the recommended order of items on three e-commerce sites reveals that e-commerce sites influence users' item choices regardless of whether they intentionally generate a decoy effect.
On e-commerce websites, users tend to struggle with selecting an item to buy because of the massive
amount of available information related to the items. Recommendation systems filter such information
based on the user–item interaction history, thereby alleviating the problem of information overload.
However, even if a recommendation system outputs a set of items that fit the user, the user does not
select the first ranked item without consideration; instead, the user compares items before selecting the
best one. Users are vulnerable to human cognitive biases during comparison and selection [
The decoy effect [
To explore the decoy effect, previous studies [
To the best of our knowledge, our study is the first to propose a methodology to evaluating the decoy effect on real e-commerce recommendation systems. The main contributions of this study are as follows.
1) We obtained real recommendation results from e-commerce sites and confirmed the existence of the decoy effect in real e-commerce recommendation systems, which has never been revealed.
2) We proposed a method for extracting row-based decoy items and confirmed the effect of a decoy on a real dataset, which was inspired by our experimental analysis where users have a high tendency to compare items in the same row of the web page in which the recommended items are displayed.
3) A new metric, called intra-row decoy effect rate (IRDE Rate), was proposed to evaluate the degree of the row-based decoy effect.
The remainder of this paper is organized as follows. The preliminary knowledge is introduced in Section 2, followed by a review of related works in Section 3. Details of the proposed method are described in Sections 4 and 5. Section 6 introduces the user study experiment conducted in this study on a one-month dataset using the IRDE Rate. Finally, the conclusions of the paper are presented in Section 7.
This section introduces how the decoy effect, which is classified into the asymmetric dominance
effect [
Bateman et al. [
The compromise effect (CE) [
This section presents a review of previous research on the decoy effect in recommendation systems.
The decoy effect tends to cause cognitive bias and induces users. Teppan et al. [
Teppan et al. [
The closest work to this study is that by Rafai et al. [
The studies mentioned above proposed a method for detecting and mitigating the decoy effect; however, the common limitation is that the mechanism and degree of decoy effect in actual recommendation systems are not known. 3.2.
The way in which the attributes are expressed also affects the decoy effect. Cui et al. [
Before evaluating the decoy effect in actual recommendation systems, we need to emphasize the
importance of understanding the users’ habits when comparing items while browsing e-commerce
websites, which has been neglected in previous studies. Although Teppan et al. [
To confirm the hypothesize, we adopted a method for determining users’ habits when comparing items, which reveals how the users' consecutive clicks are concentrated. We assume that two consecutive clicks are estimated as a comparison of items because consecutive clicks occur in a relatively short time interval compared with the overall time spent browsing the website. Assume that user clicks ( ≥ 2) items with the sequence = ( 1, 2, … , ), where indicates the k-th item clicked by user . We define the intra-row comparison ratio (IRC ratio) as the probability that two consecutive item clicks are in the same row. For instance, if a user clicks four items 1, 2, 3, and 4 sequentially, where 1and 4 are displayed in row1, whereas 2 and 3 are displayed in row2, the IRC ratio is calculated as 1/3 because the consecutive click pair ( 2, 3) is in the same row, whereas the other pairs ( 1, 2) and ( 3, 4) are in different rows.
This definition can be used to calculate the IRC ratio to verify our hypothesis. Assume that there are rows of items and the user’s clicks are randomly distributed among the rows, the IRC ratio is 1 calculated as
∗ ( − 1)/( − 1) = 1, i.e., (the number of consecutive item clicks in the same row)/ (the total number of consecutive item clicks). If the observed IRC ratio is much higher than that in the random case, it is reasonable to conclude that users have a higher probability of comparing items displayed in the same row. 4.2.
We invited 27 university students with experience in online shopping and who understand Japanese, English, and Chinese. We then asked them to participate in 2-h experiments on three real e-commerce sites, including amazon.au, amazon.jp, and jd.com (hereafter referred to as AmazonAU, AmazonJP, and JD, respectively). Note that we used three different e-commerce sites in different countries to ensure the generalization of the experimental results.
Initially, the participants imagined their real shopping process based on the assumption of buying a monitor. Then, each participant was invited to participate in the experiment with the three e-commerce sites separately. After entering the keyword “monitor” in the search form, the e-commerce site returned the relevant items on multiple web pages, each of which has multiple rows of items. AmazonJP and AmazonAU output 4 items per row, and each page has 15 rows, whereas JD outputs 5 items per row, and each page has 12 rows. Table 1 presents a summary of the statistics of the three websites. Note that the participants entered Japanese “monitor” and Chinese “monitor” on AmazonJP and JD, respectively to ensure that the search results were relevant. Finally, we asked the participants to explore the top three webpage results to compare the items to buy, followed by recording the items they were interested in clicking on while browsing the recommended items.
The participants’ tendency to compare items using the IRC ratio is presented in Table 1. Table 1 shows that the consecutive clicks tend to occur in the same row at a higher probability than random comparison (4.95 times on AmazonJP, 6.23 times on AmazonAU, 4.64 times on JD), which demonstrates the large gap between the observed IRC ratio and that with random comparison. Note that the random comparison assumed that clicks occur randomly in the result items across the three pages. Besides, even if we assume the users’ all clicks stayed within the same page, i.e., the users did not click the items in the other pages except for the first page, the observed IRC ratio is still higher at least a factor of 1.55 comparing the IRC ratio when all the clicks occur randomly within the same page (the baseline IRC ratio with random comparison is tripled). Therefore, we can conclude that users tend to compare items within the same row.
In this section, we propose a method for evaluating the decoy effect on real e-commence recommendation systems, including 1) a detection method for row-based decoy effect and 2) a new metric called IRDE Rate to evaluate the degree of the row-based decoy effect. The notations used for the decoy effect in real e-commerce websites are summarized in Table 2.
0,
Definiotin = { 1, 2 … , }; Set of rows of items returned by an e-commerce website = { ,1, ,2, … , ,| |}; Set of items in row
*The calculation method is mentioned in section 4.1 5.1.
Assume that the recommended results returned by an e-commerce website contain rows denoted as
= { 1, 2 … , }, where items in multiple pages are inlined. For instance, the website returns 3 pages
and 15 rows of items per page; in total, there are 45 rows. Each row = { ,1, ,2, … , ,| |} contains
multiple items, where , is the -th item in row . As described in Section 4, we extend the original
definition of the decoy effect [
∃ , , ∃ , , ∃ , ∈ , , ≠ , , , ≠ , , , ≠ , , (1) ℎ , ∈
denote the areas dominated by items , and , , respectively [
, , and , are items in row . We extract a set of rows in which the decoy item is included, thereby forming a set we generate a decoy item set , as given in Eq. 2.
= { | _
= 1}. For each row ∈ , = { , | , ∈ , ∃ , ∈ , ,
From Eq. 2, we generate the target item set , as given in Eq. 3. = { , | , ∈ , ∃ , ∈ , , ∉ , , The rest of the items in row are categorized into a competitor item set , as given in Eq.4. = { , | , ∈ , , ∉ , , ∉ }
Each item belongs to and only belongs to one set (i.e., ∩ ∩ = ∅). That is, we set the priority as > > .
(3) (4) (5) (6)
In this study, we propose a method for detecting decoy items in each row of recommendation results,
as described in this sub-section. Previous studies [
The concept of detecting decoy items in a row is based on the fact that the decoy item has low attribute values and a low selection rate compared with other items in the row because the purpose of the decoy effect is to boost the selection rate of the target item. We can detect decoy items based on the above idea to confirm that 1) the decoy-suspected item has a low selection rate compared with other items in the same row and 2) the difference in the selection rate in the row is reduced if the decoysuspected item is replaced with a decoy-free item. Details of the steps are presented in Algorithm 1. Note that we used a common item as a decoy-free item, where the common item is a randomly selected item from the same e-commerce recommended items and is not a decoy item of any items in the same row. 5.3. in Eq. 5. 1 3∗1 3 = ∗
( − 1, 2) , where
is the number of combinations in which the decoy effect exists in row , calculated using Eq. 6, is the the number of items displayed in row , is the combination function, and ( − 1, 2) denotes the number of combinations to select two out of − 1. For instance, there are three items ,1, ,2, ,3 displayed in row , ,1 is the decoy of ,2. The = 1. A higher IRDE rate indicates a higher risk of the decoy effect. is calculated as
We further propose a metric called the intra-row decoy effect rate (IRDE Rate) for calculating the degree of the row-based decoy. The IRDE rate indicates the risk of the decoy effect by pairing items ( and ) in the same row to detect whether other items in the same row are decoys of or , as given , | , ∈ , ∀ , , ∀ , ∈ , , ≠ , ≠ , , ( ,
In this section, we describe our user study experiment of the row-based decoy effect on real ecommerce websites targeting amazon.au, amazon.jp, and jd.com (hereafter shown as AmazonJP, AmazonAU, and JD).
We constructed three pseudo-e-commerce sites with the real recommended items returned by each e-commerce website, namely, Amazon JP, Amazon AU, and JD. Although a real e-commerce site can be used in the experiment, the returned recommended items may differ over time, resulting in an unfair outcome when comparing the decoy effect. To reproduce the same recommended item pages, we continuously crawled the recommended items for “monitor” from the three e-commerce websites for search form in English, Japanese, and Chinese (depending on the e-commerce site), we collected the recommended items from each e-commerce website. We collected the top 100 items from each ecommerce website daily. Note that we set the number of gathered items to 100 because we assumed that users usually click items within the top 100 items. The crawler also remembered each item's ranking (i.e., displayed position) on the webpage to reproduce the same recommended item list. The collected information for each item includes the item name and detailed attribute page URL. Subsequently, we collected detailed attributes for each item, including price, resolution, refresh rate, and size. Finally, we reproduced the three pseudo-e-commerce sites (AmazonJP, AmazonAU, and JD) using the collected items with their attributes.
Each pseudo-e-commerce site can display one row of items at a time in the same order as the collected order. Because we examined the row-based decoy effect, each pseudo e-commerce site shows only one row. The number of items per row is also the same as that on each real e-commerce site, that is, 4 items for AmazonAU and AmazonJP, and 5 items for JD. In addition, the pseudo-e-commerce site can switch between two different item placement patterns: 1) original page and 2) decoy-free page. The original page shows the same order of items collected from real e-commerce websites. The decoy-free page replaces the decoy item in the row with a common item, where the common item is a decoy-free item for any other items in the same row. By comparing the users' selection distributions of target and competitor items in the two different patterns, we can evaluate the existence of decoy effect, where the decoy items are detected by Algorithm 1.
Some examples of the original and decoy-free pages are shown in Fig. 6 and Fig. 7, where the images are dummy images, and we used the same images for all items to avoid any side effects from the images. In Fig. 6, item3 is the target item and item4 is the decoy of item3. Item3 has higher attributes and a
Input: Output:
Row of items in original page One row decoy-free items ′
= { ,1, ,2, … , ,| |} with attribute values = { ,1′, ,2′, … , ,| ′ |} 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 for ← 1
| |do for , in
end if end if end for end for while return ′ for ← 1 |
| do ( , , selected common item
(| |,2) do if ≠ and ≠ then if , is docoy of , or decoy of , then ← ⋃ { , } = 1 do judge the existence of decoy effect in row (Eq.1) _ ) replace , in with a randomly lower price compared with item4. Item1 and item2 are two competitors. In Fig. 7, item4 was changed to a common item because item4 in Fig. 6 was detected as a decoy of item3 (target item), whereas the other two items are two competitor items. In Fig. 6 and Fig. 7, “1222” indicates the day of data collection (12/22), whereas the text in red indicates the number of decoy rows on that day. Users can press the “TURN TO ROW” button to view different rows of items. 6.2.
A user study experiment was conducted on our constructed pseudo-e-commerce sites; we confirmed the efficiency of the decoy effect detection algorithm described in Section 5.2 and verified the existence of the decoy effect on real e-commerce websites using the row-based decoy effect detection method proposed in Section 5.3.
The number of decoy rows detected using the proposed algorithm for each e-commerce site per day is presented in Table 3. Table 3 shows that AmazonJP and AmazonAU have a similar average number of decoy rows, whereas JD has more than double the number of decoy rows for AmazonJP and AmazonAU.
item1 item2 item3
item4 item1 item2 item3
item4
We invited 20 university students as participants for the three-hour-long experiment; these participants were different from those in Section 4.2. At the beginning of the experiment, we asked the participants to imagine a real shopping process of buying a new monitor. The participants set their budgets according to their usual expenditure. Then, we introduced the experimental procedure on the pseudo-e-commerce sites as follows: 1) The experiments were conducted on three pseudo-e-commerce sites separately. 2) The participant was asked to access the decoy-free pages between Nov. 25th (1125) and Dec. 24th (1224). Then, multiple rows of items were shown individually.
3) For each row of items, the participant must select and record the item in which he or she has the most interest.
4) The participant must repeat steps 2) and 3) to complete the selection of items for all the rows. 5) The participant was asked to access the original pages between Nov. 25th (1125) and Dec. 24th (1224). Then, repeat steps 2) to 4).
6) The pseudo e-commerce site was changed to another one to repeat steps 1) to 5) until the experiments on the three e-commerce sites have been completed. Note that we asked the participant to input the duration for JD from Dec. 16th (1216) to Jan. 14th (0114) because the collected data duration is different from the other two e-commerce sites.
During the experiment, we limited the number of rows displayed to each participant in step 2) to three to ensure that the participants had sufficient time to compare and select items.
We analyzed the data recorded by the participants consisting of the selected items in each row. Specifically, we calculated the average selection rate for each item (target and competitor items) in each row on the original and decoy-free pages. For the original pages, we calculated the selection rate of the decoy items, whereas for the decoy-free pages, we calculated the selection rate of the newly added common items.
We hypothesize that replacing the decoy item with a common item decreases the selection rate for target and competitor items; that is, other items except for the common item. This is because when a decoy item exists, it has a lower selection rate than the other items, which increases the selection rate of the other items. However, when the decoy item is replaced with a common item, the selection rate of the other items decreases because the common item has a higher selection ratio than the decoy item. To verify this hypothesis, we investigated the following three perspectives.
1) Lower selection rate for the decoy item: We compared the selection rates of decoy items with the selection rates of the other items, that is, competitor and target items, in (a) of Figs. 8–10, and confirmed that the selection rates of decoy items are relatively small (0.062 for AmazonJP, 0.065 for AmazonAU, and 0.054 for JD), which satisfies one of the characteristics of the decoy item. Note that the number of selections confirmed the same trend.
2) Effect of replacing with a common item: We compared the reduction in the selection rates shown in (c) of Figs. 8–10 and confirmed that the selection rates of both competitor and target items decreassed after replacing the decoy item with a newly added common item, validating our hypothesis. For instance, for AmazonJP (Fig. 8), when newly added common items replace decoy items, the average selection rates for competitor and target items decrease by 0.115 and 0.249, respectively. The same trends were observed for AmazonAU (Fig. 9) and JD (Fig. 10). Note that the number of selections confirmed the same trend.
3) Difference in the selection rates of the target and competitor items: The reduction rate of the target items is higher than that of the competitor items by a factor of 2.17 (AmazonJP), 5.02 (AmazonAU), and 4.18 (JD), which shows that the bias that makes the target items more likely to be selected has been reduced. Note that the number of selections confirmed the same trend. In Fig. 10, we notice that the selection rate of competitor items is higher than target items. The presence of decoy items can influence the selection rate of target items, but it does not necessarily mean that the selection rate of target items will increase to higher than competitor items. The selection rate also depends on user's preference and the quality of the items.
The above three confirmed perspectives demonstrate that our method can detect the decoy effect and its existence on the three e-commerce websites.
Figs. 11–13 show the intra-row decoy effect rate (IRDE rate) transition for AamzonJP, AmazonAU, and JD, respectively depicting the degree of the row-based decoy effect. It can be observed from the figures that AmazonJP has the highest average IRDE rate of 0.1467 ranging from 0.1 to 0.2049 for the thirty days data. The peak IRDE rate was observed on 12/12/2021, whereas the IRDE rate decreased to (a)original page (b) decoy-free page
(c) Reduction Rate (a)original page (b) decoy-free page
(c) Reduction Rate (a)original page (b) decoy-free page Figure. 10: Item Selection Distribution on JD (c) Reduction Rate the minimum value on 12/18/2021. AmazonAU and JD also had different IRDE rates daily; they had average IRDE rates of 0.0627 and 0.0873, respectively. These IRDE rates do not indicate any intentional decoy effects; however, it should be noted that we encounter the decoy effect every day, where we may naturally gravitate toward some items in the recommended items in each row.
Unlike in previous studies on decoy effect that are based on virtual items, this study proposed a new method for evaluating the decoy effect on real e-commerce websites, including 1) a user study experiment to verify that users tend to compare items displayed in the same row, 2) a detection algorithm to extract the rows with a possible decoy effect by adopting a new metric called the IRDE Rate, and 3) a user study experiment to verify the decoy effect on real e-commerce recommendations, which affect users’ item selection decisions. We confirmed the existence of the decoy effect in real ecommerce recommendations by replacing decoy items with common items; subsequently, we confirmed a reduction in the IRDE rate by a factor of 2.17, 5.02, and 4.18 for amazon.jp, amazon.au, and jd.com, respectively.
In the future, we will extend the row-based decoy effect to page-based decoy effect; that is, confirming the decoy effect among items in different rows. In addition, we plan to confirm the decoy effect in more detail, including how the decoy effect is affected by the range of values of the attributes; for instance, does the decoy effect still exist if the price difference between the target and decoy items is large? 8. Reference