Analysis of User Behavior in Interfaces with Recommended Items: An Eye-tracking Study Peter Gaspar Michal Kompan Slovak University of Technology in Bratislava, Slovak University of Technology in Bratislava, Faculty of Informatics and Information Technologies Faculty of Informatics and Information Technologies Bratislava, Slovakia Bratislava, Slovakia name_surname@stuba.sk name.surname@stuba.sk Jakub Simko Maria Bielikova Slovak University of Technology in Bratislava, Slovak University of Technology in Bratislava, Faculty of Informatics and Information Technologies Faculty of Informatics and Information Technologies Bratislava, Slovakia Bratislava, Slovakia name.surname@stuba.sk name.surname@stuba.sk ABSTRACT visual attention can be reliably measured only by gaze-tracking, When analyzing user implicit feedback in recommender systems, which is unavailable in practical scenarios. Therefore, instead of several biases need to be taken into account. A user is influenced measuring visual attention, researchers try to model the typical by the position (i.e., position bias) or by the appeal of the items visual attention patterns and predict the gaze behavior according (i.e., visual bias). Since images have become an essential part of to them [26]. the Web, the study of their impact on user behavior during the The order of the gaze visits is influenced by attention bias – a ten- decision-making tasks is fundamental. This work contributes to dency to look on certain item(s) earlier than on others [13, 18]. This the understanding of attention bias in item lists interfaces of rec- may depend on many factors, such as user interface layout, visual ommenders. We present an eye-tracking user study that strives to style, or item content representation. For example, one-dimension analyze users’ behavior in the task of choosing a movie to watch. item lists may induce different behavior than two-dimension ones. Items are shown to users using two alternative representations: tex- Or, the user can scan through textual items differently than through tual and image. We found changes in the user’s behavior when the those represented by images or animations. Moreover, attention image type of interface is present. Based on our findings, the visual bias may also depend on user characteristics such as goals, skills or appeal of the images made users to change their gaze sequences cultural background. There are two main types of attention biases: more frequently. position bias (induced by the position of the items) and visual bias (induced by the visual appeal of the items). KEYWORDS In this work we studied the attention bias in recommended item lists, where we compared the user behavior in textual and image user feedback, visual bias, eye-tracking, recommendation representation of the items. We proposed and conducted an eye- ACM Reference Format: tracking user study, where participants had to choose a movie to Peter Gaspar, Michal Kompan, Jakub Simko, and Maria Bielikova. 2018. watch. Movies were presented either with their title or poster in Analysis of User Behavior in Interfaces with Recommended Items: An Eye- the circular layout (due to the eye-tracking methodological reasons, tracking Study. In Proceedings of Joint Workshop on Interfaces and Human as explained later). Our main goal was to investigate the possible Decision Making for Recommender Systems, October 2018, Vancouver, Canada (IntRS Workshop). 5 pages. differences in gaze paths between items represented either by text, or image. Our findings show that textual and graphical representations of 1 INTRODUCTION items induce different participants behavior in their gaze sequences. Interpretation of the feedback in recommender systems has been Users tend to skip more items when viewing interface with the an open problem for many years. Many approaches in this field images and make bigger transitions between the items as well. still suffer from the lack of satisfactory feedback interpretation on Moreover, when viewing content based on the preferred genres, recommended items. While gathering an implicit feedback is easy users tend to make smaller transitions than in case of random (e.g., clicks on items), distinction between the positive and negative content. Results of our study support our assumptions that the items one is tough. By clicking on an item, the user implicitly expresses a represented by images may change the users’ attention and there positive feedback, though his/her attitude may change after learn- is clearly a need to take them into account in better understanding ing more details about the item. On the other hand, not clicking on of user behavior in recommender systems. an item does not automatically mean that the user is not interested in it: he/she might not have seen it. To solve this ambiguity, we need the information on user’s visual attention. Unfortunately, a 2 USER FEEDBACK INTERPRETATION In recommender systems a user feedback is a fundamental part IntRS Workshop, October 2018, Vancouver, Canada of the process of user modelling. It is used to better understand IntRS Workshop, October 2018, Vancouver, Canada P. Gaspar et al. user’s preferences about the items and this information is further Shrestha and Lenz [20, 21], who studied the user’s gaze over image- used to train recommendation models. However, the user feedback heavy e-commerce website. Here, the participants were exposed can be also misinterpreted due to the various influences. Thus, it is to two types of interfaces – image-based and text-based and were beneficial to identify these influences and take them into account given two basic tasks: browsing and searching. They found out that while analyzing and interpreting users’ behavior. the users exposed to a page containing images focused mostly on There are several factors that may influence user’s behavior images themselves. Moreover, during both browsing and searching while browsing the items on the Web, such as personal charac- tasks, the participants did not follow the F-pattern. teristics (most notably demography, personality, emotions, and The visual bias in the domain of recommendation is not well mood [23]) or user’s short-term goal [14]. However, one of the understood. In the stereotypical case of vertical one-dimension most important aspects that may influence the user are the items textual item lists we can rather safely assume the sequences ac- themselves and the way they are presented. cording F-pattern [16, 19], where mainly the position bias plays a Recommended items are usually presented in a form of a list role [7, 12]. However, other setups, such as grid-based interfaces, or a grid [3]. The research in the recommendation lists has been although heavily used in practice, are investigated by few works already well mapped among researchers [8, 10] with grids gaining a only ([15, 26]). There is already a knowledge of the shift of user recent interest [26]. Here, the researchers analyze users’ behavior in behavior when facing the interfaces containing images, but there lists/grids and utilize it to predict, which item would user prefer [2, is an important open problem – how to utilize this knowledge in 6, 26]. These methods take into account users’ clicks in the ranked an evaluation of the recommender system. Several models have lists/grids and assume that the clicked item is also relevant for a been proposed ([6, 26]) that use gaze data in order to predict user user. behavior and recommend items. However, current state-of-the-art Chen et al. compared user behavior in three recommender inter- either cover the textual representation of the items, or a combina- faces [5]: list, grid, and pie (similar to circular interface). They found tion of textual and image representation. Thus, there is a need to that the users tended to click at the top and bottom of these three investigate textual and image representation separately to better un- interfaces. The users preferred pie and grid over list and their con- derstand visual bias and interpret users behavior when interacting fidence during decision-making was highest for the pie interface. with recommended items. In addition to clicks, users’ gaze can be used for behavior analysis over item lists. In another study Chen et al. [4] compared different 3 STUDY OF VISUAL BIAS layouts in which the recommended items can be presented: (a) a We identified three steps that need to be reflected during the study simple list, and organization interface with (b) vertical layout and of the visual bias: detection, quantification, and interpretation of the (c) quadrant layout. In organization layout items are grouped by visual bias (in recommendation process). In this paper, we focused the category (category is an explanation of representative proper- on the detection of visual bias. ties of the item). They found that users were more likely to buy For this purpose, we designed and conducted an eye-tracking items grouped by the category titles and also had more fixation on user study. We focused on the role of visual bias in changes of user products in this interface. behavior, thus we chose two types of interfaces where the items Users’ clicking activity can be biased towards the rank of the were represented either with text, or images. The study was done items, such that the probability of a click on an item depends on the in the domain of movies where the images are already a primary rank of the item – position bias [7, 12]. When position bias occurs, representation of the items. The movies were presented to users a click might be a result of the position and not of a relevance. in a circular layout, where each card in the circle represented one This poses an issue namely during the user feedback interpretation movie. where the raw user clicks cannot be considered to be representatives The research question of the study was as follows: How the user of the actual users’ preferences. behavior differs in the task of picking a movie in case if the movies In position bias research authors mainly focus on items repre- are represented as posters in comparison to text?. sented by their textual characteristics. There are several domains where images associated with items play an important role when 3.1 Study Scenario a user interacts with the interface (such as movies [25], or fash- The main task of the study participants was to select one movie ion [24, 27]). Since items are represented by images, they pose a from the presented on the screen. Movie lists were generated either valuable information for users about the items’ characteristics. randomly or based on the preferred genres that participants selected Visual bias occurs when users’ behavior is changed due to the at the beginning of the study. After viewing all of the 24 stimuli fact that they perceive the items that are represented by the im- (192 different movies), the participants were asked to indicate the ages. Similarly to the position bias problem, when the visual bias movies that they had previously watched (i.e., the movies they happens and we analyze users’ behavior, his/her activity can be watched before participating in the study). The list of the movies to misinterpreted and we cannot confidentially decide whether the check contained only those movies that were displayed at stimuli. user’s click on the item was invoked by the user’s preference (to Dataset. We used the MovieLens 20M [11] dataset containing the item) or by the appearance of the item. 27 278 movies. Additional metadata (actors, directors, plot) and Visual attractiveness and saliency of images on the Web have posters were obtained from The MovieDB API1 . We removed the been already well-studied. Nielsen et al. [16] revealed that, for a movies without the poster, the most popular movies (TOP 10% of majority of tasks, users follow an F-shaped pattern while reading the 1 https://www.themoviedb.org/documentation/api website’s textual content. Nielsen’s study was further extended by IntRS Workshop, October 2018, Vancouver, Canada the movies based on their average rating), and the movies released before 1990 (to reflect the age of the participants). Presentation of the items. To ensure that the initial probability of the first gaze would not be influenced by previous attention [9], items were positioned approximately into the circular shape (Fig- ure 1). Moreover, participants were asked to fixate on a small target in the middle of the screen for 4 seconds before each set of movies was displayed to them. The target was also used for the validation of calibration as described below. To compare the behavior of the participants when interacting with the texts and images, two types of items representations were used: (a) text-based that used only the title of the movie, (b) image- based that used only the poster of the movie (Figure 1). Title or Figure 1: Example of two stimuli containing the movie cards poster were placed into the rectangle with fixed dimensions (card). (left – text-based, right – image-based stimulus). Only one In text-based representation, the title of the movie was vertically type of representation was shown per stimulus. The circu- centered and there was a rectangle with gray-colored background lar layout was chosen from the eye-tracking methodologi- of a fixed height to ensure that the text would not cause any visual cal reasons to suppress a potential first fixation bias. After bias. If a participant chose to show the details of the movies (using a stimulus was displayed, users had to fixate at the middle the button Detail), they were displayed within the rectangle. cross. Stimuli generation. To cover various scenarios of movie selection, we opted for the several approaches of generating movie sets (Ta- University of Technology in Bratislava2 [1]. All the instructions ble 1). Firstly, we utilized three genres that the participant picked and parts of the study were presented as a website placed within as preferred at the beginning of the study. For each genre we gener- the Tobii Studio3 test suite. We collected user behavior data in the ated one text-based stimulus (row A) and three image-based stimuli form of clicking and eye-tracking activity. Eye-tracking data were (row B). In addition, we randomly selected three non-preferred collected using the Tobii X2-60 eye-trackers (60 Hz sampling freq.) genres and for each generated one image-based stimulus (row C) with the screen resolution 1920x1200px. Validations of eye-tracker to capture possible differences across various genres. We generated calibration were performed at the beginning, during the study, and 9 stimuli (D, E), where the movies were generated randomly from at the end of the study. Average precision (based on [22]) for all the the whole dataset. participants was 0.38 degree (median 0.30 degree), average accuracy 0.93 degree (median 0.71 degree). Table 1: Strategies for stimuli generation. We did not use An analysis of the results was based on the eye-tracking data. We equal number of the stimuli, since our goal was to gather converted raw gaze data to fixations using the I-VT filter [17]. To more data for image-based stimuli (to use them for future identify which items (i.e., movies) attracted the users, we considered studies). When directly comparing text-based and image- the movie cards to be the Areas of interest (AOIs). There were based strategies, we firstly normalized the results. exactly 8 AOIs corresponding to the 8 cards (with unique movies) for each stimulus. The main goal was to identify fixations that matched particular AOIs. We sequentially analyzed each fixation Source of the movies Presentation No. of stimuli and checked within which AOI it fitted. A Preferred genres Text 3 The fixations that matched the interaction buttons (Select, Detail) B Preferred genres Image 9 were omitted. To account for small calibration errors in AOI hit C Non-preferred genres Image 3 detection, we experimentally set the absolute error tolerance of D Random Text 5 ∆ = 5px (set based on our observations; larger value of ∆ tended E Random Image 4 to lower the detection accuracy). A fixation was considered as an AOI hit, when it fell within this 5px boundary around the AOI rectangle. To remove the users and stimuli with corrupted eye- Each movie was presented to a participant in the entire session tracking data, we calculated the median Euclidean distance between only once (to prevent any priming and memory effects). Moreover, the fixation and the validation point and removed users and stimuli we randomized the order of movies within stimuli and the order of with distances above 100px (based on the validation target that has the stimuli themselves. If two participants picked the same genre, size 100x100px). they both encountered the same set of movies at some point in the session (though in a different order). 4 STUDY RESULTS There were totally 64 participants in the study (45 males, 19 females); 3.2 Data Collection and Preprocessing varying in age 15-27 (σ = 3.42). Most of them were high school and university students, the rest adults of various occupations. The study was conducted in controlled, eye-tracking laboratory conditions in User eXperience and Interaction Research Center 2 https://uxi.sk/ at Faculty of Informatics and Information Technologies, Slovak 3 https://www.tobiipro.com/product-listing/tobii-pro-studio IntRS Workshop, October 2018, Vancouver, Canada P. Gaspar et al. Figure 3: Gaze transition matrices of text-based (left) and image-based (right) stimuli. Rows indices represent the size of the transition in the previous step in a sequence. Column indices represent the size of the transition in the following step in a sequence. Counts are a sum of occurrences over all the participants and all AOIs. Figure 2: AOIs transition graph of image-based stimuli. next AOI, regardless of the direction (clockwise / counterclockwise). Thickness of the edges corresponds to the number of transi- Hence the number of displayed items was 8, the values of gaze tions between AOIs (sum over all participants). Intervals are transition sizes fit into {1, 2, 3, 4}. Size of 1 reflects that the user based on quartiles and median of the number of transitions. visited the next AOI, sizes above 1 that the sequence of gaze was There is a strong circular pattern of gaze sequences (edges broken and a participant skipped some of the items (a possible on the perimeter have notably higher number of transitions indicator of the visual bias in the image-based stimuli). in comparison to other edges). Since the differences between Figure 3 shows the absolute counts of gaze transitions (a sum over the text-based and image-based AOI transitions were not no- all users and all stimuli). We may identify a difference in transition table, we omit the text-based version. of size 1, where the texts exceed the images. On a counterpart bigger transitions happened more frequently when the user was presented A pilot study was conducted with 3 participants to identify the the images. However, in this case the absolute counts may be a bit problems and to adjust the overall design of the experiments. After misleading, since the users tended to behave differently. the removal of error data, we were left with 56 participants and To verify whether the behavior varied on a per-user basis, for 1,169 stimuli instances to analyze. each user and each stimulus we calculated an average size of gaze transitions, calculated a per-user average, and performed the pair- 4.1 Position Bias wise t-test between the users (all the populations had normal dis- In a circular interface, we hypothesized that the position bias may tribution, based on D’Agostino-Pearson’s test, p > 0.05). It was re- have an influence on user’s behavior in two main ways: (1) there vealed that the average size of the gaze transition was significantly is an AOI that tends to have significantly higher frequency of the different (p-value < 0.01) when comparing text-based (mean=1.28, first fixations, (2) the order of the visits on the items is similar to σ = 0.13) and image-based (mean=1.36, σ = 0.14) stimuli. The total the circular order of the items. count of breaks (i.e., gaze transitions greater than 1) per stimulus Although we used the validation target (in the center of the was in average also higher for the images (text = 3.49, images = screen) at the beginning of each stimulus to reset participants’ gaze, 4.01, p < 0.01). it was revealed that in most cases they tended to start to fixate If we examine the image type of representation only, the average mostly in the same position, which is ascribed to the top of the size of the break was largest for the posters of random movie stimuli. screen. The behavior was not different regardless of the texts or The smaller breaks (size 2) were more frequent for the strategy of images (based on the Kolmogorov-Smirnov statistic test, stat=0.5, preferred genres, whereas the largest breaks (size 4) were more p-value=0.1877). common when the random movies were presented. AOIs transition graph for the images representation is illustrated in Figure 2. It is clear that users tended to follow the circular path 5 CONCLUSION while observing the items (visible from the edges on the perime- Attention bias poses a fundamental issue when analyzing implicit ter with highest number of occurrences). The similar pattern was behavior of the users on the Web. In this paper, we studied two types present in the text type of representation. On the other hand, from of attention biases – position bias and visual bias. We proposed the graph, we may reveal that there were also cases, where the and conducted an eye-tracking user study with 64 participants that users tended to skip some AOIs. strives to better understand these biases in the task of selecting an item from the presented recommended items. For the execution of 4.2 User Sequence Breaks the study, we picked the movie domain. To measure the difference in sequences of AOIs visits, the gaze We compared the users’ behavior when the item (movie) was transition sizes were calculated for each stimulus. We defined the represented either by a text, or by a poster (image) and identified gaze transition size as the number of AOI that users visited until the several differences. In both representations of the items a strong IntRS Workshop, October 2018, Vancouver, Canada position bias occurred. The items were presented in a circular layout the Fourth ACM Conference on Recommender Systems (RecSys ’10). ACM, New to eliminate positon bias. Participants tended to follow its shape York, NY, USA, 39–46. https://doi.org/10.1145/1864708.1864721 [9] Andrew T. Duchowski. 2007. Eye Tracking Methodology: Theory and Practice. while browsing. However, when the participants were exposed to Springer-Verlag, Berlin, Heidelberg. the movie posters, they were more likely to break such a sequence [10] Evgeny Frolov and Ivan Oseledets. 2016. Fifty Shades of Ratings: How to Benefit from a Negative Feedback in Top-N Recommendations Tasks. In Proceedings of and change their standard gaze path. This behavior was consistent the 10th ACM Conference on Recommender Systems (RecSys ’16). ACM, New York, across many participants. Another interesting finding was that NY, USA, 91–98. https://doi.org/10.1145/2959100.2959170 even though we instructed the participants to look at the center [11] F. Maxwell Harper and Joseph A. Konstan. 2015. The MovieLens Datasets: History and Context. ACM Trans. Interact. Intell. Syst. 5, 4, Article 19 (Dec. 2015), 19 pages. of the screen before each stimulus, they tended to start their gaze https://doi.org/10.1145/2827872 sequence at the AOIs that were placed at the top of the screen. [12] Thorsten Joachims, Laura Granka, Bing Pan, Helene Hembrooke, and Geri Gay. Our study provides a direct comparison of image-based interface 2005. Accurately Interpreting Clickthrough Data As Implicit Feedback. In Pro- ceedings of the 28th Annual International ACM SIGIR Conference on Research and with text-based interface and attempts to quantify the differences. Development in Information Retrieval (SIGIR ’05). ACM, New York, NY, USA, The analysis of user sequence breaks showed that images caused 154–161. https://doi.org/10.1145/1076034.1076063 [13] Tilke Judd, Krista Ehinger, Fredo Durand, and Antonio Torralba. 2009. Learning the changes in user behavior from several perspectives. We also to predict where humans look. In 2009 IEEE 12th International Conference on found that the user behavior varied based on the shown genre, or a Computer Vision. 2106–2113. https://doi.org/10.1109/ICCV.2009.5459462 degree of a preference to a genre when the images were presented. [14] Michal Kompan, Ondrej Kassak, and Maria Bielikova. 2017. Beyond User Prefer- ences: The Short-Term Behaviour Modelling. In Proceedings of the RecSys’2017 1st Thus, not only the representation might be important, but also the Workshop on Temporal Reasoning in Recommender Systems (CEUR Workshop Pro- choice of which image to show should be properly examined. We ceedings), Vol. 1922. CEUR-WS.org, 1–3. http://ceur-ws.org/Vol-1922/paper1.pdf believe that this choice should be also taken into account during [15] Wanxuan Lu and Yunde Jia. 2014. An Eye-Tracking Study of User Behavior in Web Image Search. In PRICAI 2014: Trends in Artificial Intelligence. Springer the evaluation of user behavior in results of recommendations. International Publishing, Cham, 170–182. We strive to further investigate user behavior in the recommen- [16] Jakob Nielsen. 2006. F-Shaped Pattern For Reading Web Content, Jakob Nielsen’s Alertbox. https://www.nngroup.com/articles/f-shaped-pattern-reading-web- dation item grids containing images. In our future work, we plan to content-discovered/. (2006). [Online; accessed 25 February 2018]. utilize the measured user behavior and use it in order to take into [17] Anneli Olsen. 2012. The Tobii I-VT fixation filter. Algorithm description. account both position and visual bias in feedback interpretation https://www.tobiipro.com/siteassets/tobii-pro/learn-and-support/analyze/how- do-we-classify-eye-movements/tobii-pro-i-vt-fixation-filter.pdf. (2012). task and user behavior modelling. [18] Claudia Roda. 2011. Human attention and its implications for human–computer interaction. Human attention in digital environments 1 (2011), 11–62. [19] Andrew Schall and Jennifer Romano Bergstrom. 2014. 1 - Introduction to Eye ACKNOWLEDGMENTS Tracking. In Eye Tracking in User Experience Design, Jennifer Romano Bergstrom and Andrew Jonathan Schall (Eds.). Morgan Kaufmann, Boston, 3 – 26. https: This work was partially supported by Human Information Behavior //doi.org/10.1016/B978-0-12-408138-3.00001-7 in the Digital Space, the Slovak Research and Development Agency [20] Sav Shrestha and Kelsi Lenz. 2007. Eye gaze patterns while searching vs. brows- under the contract No. APVV-15-0508, and grants No. VG 1/0646/15, ing a Website. Usability News 9, 1 (2007), 1–9. http://usabilitynews.org/ eye-gaze-patterns-while-searching-vs-browsing-a-website/ No. VG 1/0667/18, and No. KEGA 028STU-4/2017. The authors [21] Sav Shrestha, Kelsi Lenz, Barbara Chaparro, and Justin Owens. 2007. “F„ Pattern would also like to thank for financial contribution from the STU Scanning of Text and Images in Web Pages. Proceedings of the Human Factors Grant scheme for Support of Young Researchers. and Ergonomics Society Annual Meeting 51, 18 (2007), 1200–1204. https://doi.org/ 10.1177/154193120705101831 [22] Tobii Technology. 2011. Eye tracker accuracy and precision. (2011). REFERENCES https://www.tobiipro.com/learn-and-support/learn/eye-tracking-essentials/ what-affects-the-accuracy-and-precision-of-an-eye-tracker/ [Online; accessed [1] Maria Bielikova, Martin Konopka, Jakub Simko, Robert Moro, Jozef Tvarozek, 04 May 2018]. Patrik Hlavac, and Eduard Kuric. 2018. Eye-tracking en masse: Group user studies, [23] Licai Wang, Xiangwu Meng, Yujie Zhang, and Yancui Shi. 2010. New Approaches lab infrastructure, and practices. Journal of Eye Movement Research 11, 3 (2018). to Mood-based Hybrid Collaborative Filtering. In Proceedings of the Workshop on https://doi.org/10.16910/jemr.11.3.6 Context-Aware Movie Recommendation (CAMRa ’10). ACM, New York, NY, USA, [2] Rongwei Cen, Yiqun Liu, Min Zhang, Bo Zhou, Liyun Ru, and Shaoping Ma. 2009. 28–33. https://doi.org/10.1145/1869652.1869657 Exploring Relevance for Clicks. In Proceedings of the 18th ACM Conference on [24] Anna Wroblewska and Lukasz Raczkowski. 2016. Visual Recommendation Use Information and Knowledge Management (CIKM ’09). ACM, New York, NY, USA, Case for an Online Marketplace Platform: Allegro.Pl. In Proceedings of the 39th 1847–1850. https://doi.org/10.1145/1645953.1646246 International ACM SIGIR Conference on Research and Development in Information [3] Li Chen and Pearl Pu. 2008. A Cross-cultural User Evaluation of Product Retrieval (SIGIR ’16). ACM, New York, NY, USA, 591–594. https://doi.org/10. Recommender Interfaces. In Proceedings of the 2008 ACM Conference on Rec- 1145/2911451.2926722 ommender Systems (RecSys ’08). ACM, New York, NY, USA, 75–82. https: [25] Lili Zhao, Zhongqi Lu, Sinno Jialin Pan, and Qiang Yang. 2016. Matrix factor- //doi.org/10.1145/1454008.1454022 ization+ for movie recommendation. IJCAI International Joint Conference on [4] Li Chen and Pearl Pu. 2011. Users’ Eye Gaze Pattern in Organization-based Artificial Intelligence (2016), 3945–3951. Recommender Interfaces. In Proceedings of the 16th International Conference on [26] Qian Zhao, Shuo Chang, F. Maxwell Harper, and Joseph A. Konstan. 2016. Gaze Intelligent User Interfaces (IUI ’11). ACM, New York, NY, USA, 311–314. https: Prediction for Recommender Systems. In Proceedings of the 10th ACM Conference //doi.org/10.1145/1943403.1943453 on Recommender Systems (RecSys ’16). ACM, New York, NY, USA, 131–138. https: [5] Li Chen and Ho Keung Tsoi. 2011. Users’ decision behavior in recommender //doi.org/10.1145/2959100.2959150 interfaces: Impact of layout design. In RecSys’11 Workshop on Human Decision [27] Zhengzhong Zhou, Yifei Xu, Jingjin Zhou, and Liqing Zhang. 2016. Interactive Making in Recommender Systems. Image Search for Clothing Recommendation. In Proceedings of the 2016 ACM on [6] Shiwei Cheng, Xiaojian Liu, Pengyi Yan, Jianbo Zhou, and Shouqian Sun. 2010. Multimedia Conference (MM ’16). ACM, New York, NY, USA, 754–756. https: Adaptive User Interface of Product Recommendation Based on Eye-tracking. //doi.org/10.1145/2964284.2973834 In Proceedings of the 2010 Workshop on Eye Gaze in Intelligent Human Machine Interaction (EGIHMI ’10). ACM, New York, NY, USA, 94–101. https://doi.org/10. 1145/2002333.2002348 [7] Nick Craswell, Onno Zoeter, Michael Taylor, and Bill Ramsey. 2008. An Exper- imental Comparison of Click Position-bias Models. In Proceedings of the 2008 International Conference on Web Search and Data Mining (WSDM ’08). ACM, New York, NY, USA, 87–94. https://doi.org/10.1145/1341531.1341545 [8] Paolo Cremonesi, Yehuda Koren, and Roberto Turrin. 2010. Performance of Recommender Algorithms on Top-n Recommendation Tasks. In Proceedings of