Users’ Decision Behavior in Recommender Interfaces: Impact of Layout Design Li Chen and Ho Keung Tsoi Department of Computer Science Hong Kong Baptist University Hong Kong, China {lichen, hktsoi}@comp.hkbu.edu.hk ABSTRACT Unfortunately, little is known about the impact of recommender Recommender systems have been increasingly adopted in the interface’s layout on users’ decision-making behavior. There is current Web environment, to facilitate users in efficiently locating also lack of studies that examined whether users would perceive items in which they are interested. However, most studies so far differently, especially regarding their decision confidence and have emphasized the algorithm’s performance, rather than from perceived system’s competence, due to the change of layout. Thus, the user’s perspective to investigate her/his decision-making in this paper, we are particularly interested in exploring users’ behavior in the recommender interfaces. In this paper, we have behavior in the recommender interface when it is presented with performed a user study, with the aim to evaluate the role of layout three layout designs: list, grid and pie. As a matter of fact, most of designs in influencing users’ decision process. The compared current recommender systems follow the list structure, where layouts include three typical ones: list, grid and pie. The recommended items are listed one after another. The grid layout, experiment revealed significant differences among them, with a two-dimensional display with multiple rows and columns, has regard to users’ clicking behavior and subjective perceptions. In also been applied in some recommender sites to display the items. particular, pie has been demonstrated to significantly increase As the third alternative design, pie layout, though it has been users’ decision confidence, enjoyability, perceived recommender rarely used in recommender systems, has been proven as an competence, and usage intention. effective menu design for accelerating users’ selection process [2]. The comparison among them via user evaluation could hence Categories and Subject Descriptors tell us which layout would be most desirable to optimize the H5.m. Information interfaces and presentation (e.g., HCI): recommender’s benefits. That is, with the ideal layout design, Miscellaneous. users can be more active in clicking recommendations, be more confident in their choices, and be more likely to adopt the General Terms recommender system for repeated uses. Design, Experimentation, Human Factors. Concretely, we evaluated three layout designs from both objective and subjective aspects to measure users’ decision performance. Keywords The objective measures include users’ clicking behavior (e.g., the Users’ decision behavior, recommender system, interface layout, first clicked item’s position, the amount of clicked items, etc.), user study. and time consumption. Subjective measures include users’ decision confidence, perceived interface competence, 1. INTRODUCTION enjoyability, and usage intention. These measurements are mainly Although recommender systems have been popularly developed based on the user evaluation framework that we have established in recent years as personalized decision support in social media from prior series of user studies on recommenders [4,8,9]. We and e-commerce environments, more emphasis has been placed thus believe that they can be appropriately utilized as the standard on improving algorithm accuracy [10], and less on studying users’ to assess user behavior. Relative to our earlier work [5], this paper actual decision behavior in the recommender interfaces. On the was for the first time to investigate the effect of basic layouts of other hand, according to user studies conducted in other areas, recommender interfaces on users’ decision process, which is also users will likely adapt their behavior when being presented with new in the general domain of recommender systems, to the best of different information presentations. For instance, in a recent study our knowledge. done by Kammerer and Gerjets, the presentation of Web search engine results by means of a grid interface seems to prompt users 2. THREE LAYOUT DESIGNS to view all results at an equivalent level and to support their 2.1 List Layout selection of more trustworthy information sources [7]. Braganza et al. also investigated the difference between one-column and As mentioned above, most existing recommender systems employ multi-column layouts for presenting large textual documents in the standard one-dimensional ranked-order list style, where all web-browsers [1]. They indicated that users spent less time items are displayed one after the other. For instance, MovieLens scrolling and performed fewer scrolling actions with the multi- is a typical collaborative filtering (CF) based movie recommender column layout. system (www.movielens.org). In this system, items are ranked by their CF scores in the descending order and presented in the list format. The score represents the item’s matching degree with the current user’s interest. RecSys’11 Workshop on Human Decision Making in Recommender Systems, October 23-27, 2011, Chicago, IL, USA Figure 1.a shows the sample layout (where every position is for placing one item). The number of shown items varies among 21 existing systems. Some systems (e.g., Criticker.com) limit the 3. PROTOTYPE IMPLEMENTATION number to 10 or less, while some systems (like MovieLens) give a list of items as many as possible and divide them into pages (e.g., We implemented a movie recommender system with the three one page displays a fixed number of items). Each item is usually layout versions. The recommending mechanism is primarily based described with its basic info (e.g., thumbnail image, name, rating). on the hybrid of tag suggestions and tag-aware item When users click an item, more of its details will be displayed in recommendation [11]. Specifically, based on the user’s initial tag a separate page. profile, the system will first recommend a set of tags from other users as suggestions to enrich the new user’s profile. In the mean 2.2 Grid Layout time, a set of movie items with higher matching degree with the The grid layout design has also been applied in some existing user’s current tag profile is returned as item recommendations. If websites (e.g., hunch.com). In this interface, recommendations are the user modifies her/his profile, the set of recommendations will presented in multiple rows and columns, so several items are laid be updated accordingly. More concretely, the control flow of the out next to each other in one line. The regular presentation is to system works in the following four steps: align the items horizontally (line by line). For example, as shown Step 1. To begin, the new user is asked to specify a reference in Figure 1.b, the positions 1, 2, 3, …, 12 are respectively product (e.g., a favorite movie) as the starting point. The product allocated with items that are ranked 1st, 2nd, 3rd, …, 12st according and its associated tags (as annotated by other users) are then to their relevance scores. stored in the user’s profile. Alternatively, s/he can directly input Because users likely shift eyes to nearby objects [6], we were one or more tag(s) for building her/his initial profile. interested in verifying whether the grid format would stimulate Step 2. Profile-based Item Recommendation. Based on the profile, users to discover more items than in list. the system generates a set of item recommendations (i.e., movies in our prototype) to the user via the weighted combination of FolkRank and content-based filtering approaches. Specifically, Position 1 1 2 3 1 12 FolkRank transforms the tripartite graph found in the folksonomic Position 2 2 systems into the two-dimension hyper-graph. In parallel, the 11 3 content-based filtering approach rank items based on the 4 5 6 Position 3 4 correlation between the content of the items (i.e., title, keywords, 10 Position 4 and user-annotated tags) and the user’s current profile. A tuning 7 8 9 9 5 parameter is dynamically set to adjust the two approaches’ Position 5 8 6 relative weights in producing the top k recommendations. Position 6 10 11 12 7 Step 3. Tag recommendation. In the recommender interface, the system not only returns item recommendations, but also a set of Position 7 …… tags to help users further enrich their profile if they need. To generate the tag recommendation, we first deployed the Latent a. List b. Grid c. Pie Dirichlet Allocation (LDA), which is a dimensionality reduction Figure 1. The three layout designs for recommender interface technique, to extract common topics among all user tags in the (the number refers to the position of a recommendation). database. Each topic represents a cluster, and all the extracted clusters were then applied to match with the current user’s tag 2.3 Pie Layout profile. New tags from the best matching clutsers are then Another two-dimensional layout design is to place the items in the retrieved as recommended tags to the user. These tags’ associated compass format, i.e., pie layout. This idea originates from the items are also integrated into the process of generating item comparison of linear menu (i.e., the alphabetic ranked-order of recommendations in the next cycle if any of them were selected menu choices) and pie menu [2]. In the pie menu, items are placed by the user. Moreover, the tag recommendations were grouped along the circumference of a circle at equal radial distances from into three categories in the interface: factual tags (i.e., the tag the center. The distance to and size of the target can be seen as an describes a fact of the item, “rock”), subjective tags (the people’s effect on positioning time according to Fitts’ law [3]. Researchers opinion, “cool”) and personal tags (used to organize the user’s previously found that due to the decreased distance (i.e., the own collection, e.g., “my favorites”). The grouping is minimum distance needed to highlight the item as selected) and automatically performed. For example, if the tag is a common increased target size, users selected items slightly faster. The drift keyword in the item’s basic descriptions, it is treated as factual distance after target selection and error rates were also minimized. tags. General Inquirer1 , a content analysis program, is employed to determine whether a tag is subjective. The rest of the tags that We thus believe that the pie layout could offer a novel alternative do not belong to the first two categories are then considered to be and potentially more effective design to be studied. The reason is personal tags. that it would support users to have a quicker overview of all displayed items, as the interface consumes greater width but less Step 4. If the user has done any modifications on her tag profile, it height. In addition, it would allow users to click items faster, will be used to produce a finer-grained item recommendation in because the mean distance between items is reduced. the next interaction cycle (returning to step 2). When we concretely implemented this interface, we adhere to the The process from Step 2 to Step 4 continues till the user selects regular clockwise direction to display the items along the circle, item(s) as her/his final choice(s), or quit from the system without with the most relevant item placed at the first position (see Figure 1.c). 1 http://www.webuse.umd.edu:9090/ 22 selecting any recommendations. More details about the algorithm 4. EXPERIMENT SETUP steps can be referred to [11]. To build the prototype, we crawled 998 movies and their info 4.1 Measures (including posters, names, overall ratings, number of reviewers, Identifying the appropriate criteria for assessing a recommender directors, actors/actresses, plots, etc.) from IMDB (Internet Movie system from the user’s perspective has always been a challenging Database) site. These movies’ associated tags were extracted from issue. Accumulated from our previous experiences on this track MovieLens for building the tag base. [4,8,9], a set of measures have been established. The framework Concretely, the system returns 24 movie recommendations at a not only includes objective interaction effort that users have spent time. The 24 movies are sorted in the descending order by their with the system (e.g., time consumption), but also users’ relevance scores, and then divided into two pages (i.e., each page perceived confidence in choices that they made in the with 12 movies). The switching to the second page is through the recommender and their intention to repeatedly use the system. “More Movies” button. Such design could enable us to evaluate More specifically, in this experiment, in order to in depth identify user behavior not only in a single page, but also their switching the three layouts’ respective effects on user behavior, we assessed behavior across pages (i.e., whether they click the button to view the following aspects (see Figure 2). more items). The recommended movies are presented differently in the three 4.1.1 Objective Measures layout versions (see Figure 3). In the list layout, the 12 movies in The objective measures mainly include quantitative results from one page are displayed in the list style, where the ranked 1st is analyzing users’ actual behavior in using the interface. Concretely, positioned at the top, followed by the ranked 2nd one (the ranked they cover two major aspects. 1st one means that the movie has the highest score among the 12 movies). In the grid layout, three movies are displayed along one Clicking behavior. It has been broadly recognized that users’ row and four in one column. More specifically, the first row clicking decisions on the recommender interface (i.e., clicking an shows the ranked 1st, 2nd and 3rd movies from left to right, the item to view its detailed info) reflects their interest in the item. second row is with the ranked 4th, 5th, 6th movies, and so on. In the Therefore we recorded users’ clicking behavior and clicked items’ pie layout, the 12 movies (each with the same target size as in positions. The goal was to evaluate whether the clicking would be grid) are presented in a clockwise direction, with the ranked 1st influenced by the layout, and which interface could support users movie at the 12 clock’s position, 2nd at the 1 clock’s position, and to easily find interesting items. Specifically, the clicking behavior so forth. was analyzed via three variables: 1) the users’ first clicked item’s position, from which we could know whether users’ first click In all of the three interfaces, each movie has a poster image, name, falls on the most relevant item (as predicted by the system) or not. rating, number of reviews and a brief plot. More of the movie’s 2) All clicks on distinct items that a user has made throughout details can be accessed by clicking it. A separate detail page will her/his session of using the interface. This variable can expose the then show the movie’s director(s), actor/actress info, detailed plot, distribution of clicks over different areas on the interface. The and give links to IMDB and trailer, etc. If users like this movie, comparison among all users could further reveal their similar they could click the button “My Choice” at the detail page. clicking pattern. In addition, the total amount of clicked items There is also a profile area in the three interfaces, which allows could tell us how many items interested the user when s/he was users to modify their tag profile by selecting the system-suggested confronted with the whole set of recommendations in the ones or inputting their own. In list and grid, it is placed on the left respective layouts. 3) Frequency of clicking “more movies”. Such panel, and in pie, it is in the central part. action indicates that users switched to the next page to view more recommended items. If the frequency is higher, one possible Recommender Interfaces explanation is that users felt enjoyable while using the interface and were motivated to take the effort in viewing more items, or it is because users cannot find the interesting items at the first page. Thus, this number should be analyzed in combination with other Objective Subjective perceptions variables, especially users’ subjective opinions on the interface, behavior so that we could more fairly attribute it to the pros or cons of the Confidence in choices interface. Clicking Objective effort consumption. Besides above mentioned analyses behavior (e.g., Perceived interface on users’ clicking behavior, we also recorded the time a user first click, competence spent in completing the task on the specific interface. This value amount of can be used to represent the amount of objective effort that users clicked items and exerted while using the interface. In fact, it has been frequently Enjoyability adopted in related literatures to be an indicator of the system’s performance [10]. However, less time does not mean that users Objective effort Behavioral intentions would perceive less effort taken or have better decision quality [8]. (e.g., time spent) That is why we included various subjective constructs (see the next subsection) to better understand the interface’s true merits. Figure 2. Objective and subjective measures in the user study. 23 List layout Grid layout Pie layout Figure 3. A movie recommender interface with three layout versions. 4.1.2 Subjective Measures order was randomized in order to avoid any carryover effects (so there are six possible sequences of displaying the three layouts). Users’ decision confidence and perception of the interface were To evaluate each layout, a concrete task was assigned to the user. mainly obtained through the post-task survey. Actually, the Concretely, each layout interface was randomly associated with subjective measures can be quite useful to expose the competence one scenario for the user to play the role and perform the of the interface in assisting users’ decision-making and its ability situational task. For example, one scenario is “This is October, the in increasing users’ intention to use the system again. The festival Halloween is coming. John is a college student, and he variables that we have used in this experiment cover four would like to organize an event to watch movie with his friends at constructs: decision confidence, perceived interface competence, his home. After discussing with his friends, they would like to enjoyability, and behavioral intentions. The perceived interface watch a horror movie in this festival. John is responsible for competence was qualitatively measured through multiple choosing some movies as candidates. Please imagine yourself as dimensions: users’ perception of item/tag recommendation quality, John and use the interface to find three candidates that you would perceived ease of use of the interface in searching for info, and like to recommend to your friends.” The other two scenarios were perceived ease of use in modifying their profile. The behavioral respectively for Valentine’s Day, and the military subject. In each intention was assessed from users’ intention to use the interface scenario, the user was encouraged to freely use the interface to find again. three most suitable movies according to her/his own preferences. Table 1 lists all of the questions we used to measure these The experiment was setup as an online procedure. It contains the subjective variables. In the form of questionnaire, each question instructions, recommender interfaces and questionnaires, so that was required to respond on a 5-point Likert scale from “strongly users could easily follow and we could also automatically record disagree” (1) to “strongly agree” (5). all of their actions in a log file. The same administrator supervised the experiment for all participants. Table 1. Questions to measure users’ subjective perceptions A total of 24 volunteers (12 females) were recruited. 3 are with age Measured Question responded on a 5-point Likert scale from less than 20, 1 with age above 30, and the others are between 20 and variables “strongly disagree” to “strongly agree” 30. Most of them are students in the university, pursuing Bachelor, Decision Q1: I am confident that I found the best choices Master or PhD degrees, but their studying majors are diverse. All confidence through the interface. participants had visited movie recommender sites (e.g., Yahoo Q2: The interface helps me find some good movies; movie) before the experiment, and 58.3% have even visited the Q3: This interface provides some good “tag” indicated sites at least a few times every three months. The Perceived suggestions to help me specify criteria; recommender participants also specified the mean reasons that will motivate them Q4: I found it easy to use the interface to search for to repeatedly use such a site. Among the various reasons, the ease of interface’s movies; use of the site’s user interface was indicated as the most important competence Q5: I found it easy to modify my profiles in the factor (with the importance rate 3.83 in the range of 1 to 5). The interface. second important factor is the site’s ability in helping them find Enjoyability Q6: I felt enjoyable while using this interface. movies that they like (3.79), followed by the site’s reputation (3.5). Behavioral Q7: I am inclined to use this interface again. Intention 4.3 Results 4.2 Experiment Procedure and Participants 4.3.1 Objective Behavior For each layout version, we first counted the number of users’ The primary factor manipulated in the experiment is layout as we first clicks that fall on a particular position and then classified prepared with three versions in the prototype system: list, grid, pie. them into areas. Specifically, in one interface, each area contains To compare the three layouts, we applied the within-subjects three adjacent positions (e.g., 1-3 positions compose the first area, experiment design. That is, every participant was required to 4-6 form the second area, and so on). Areas 5 to 8 refers to the evaluate all of them one by one, but the interfaces’ appearance positions at the second recommendation page of the interface. 24 Figure 4 shows the actual distribution. In total, 8, 10, and 8 users pie. Though it took longer in list and pie, the differences are not have clicked item in the first area respectively in list, grid and pie significant (p > 0.1 by ANOVA and three pairs of t-test). interfaces. Then in the list and pie, there exists a linear drop from areas 1, to 2, then to 3. In area 4, the list’s curve returns to the 4.3.2 Subjective Perceptions same level of area 2, but in pie it goes much higher even beyond Besides measuring users’ objective behavior, we were driven to the level of area 1. In grid, a sharp drop appears from areas 1 to 2. further understand their subjective perceptions such as decision Then the curve rebounds and reaches to a level equivalent in areas confidence, perceived ease of use of the recommender interface, 3 & 4. Another interesting finding is that there are 3, 2, and 1 of and intention to use it again in the future, as described in Section users’ first clicks were at the second page respectively in list, grid 4.1.2. and pie (i.e., in areas 5 to 8). To rank these areas by the amounts Significant differences were found in respect of these subjective of first clicks, we can see that the hotter areas in list are 1, 2 & 4. measures (see Table 2). First of all, most of users were confident In grid, they are 1, 3 & 4, and in pie, they are 1 and 4. that they found the best choices through pie. The mean score is To further investigate the hot areas throughout a user’s whole 3.54 which is marginally significantly higher than the average in interaction session, we counted her/his total clicks made on each list (vs. 3.125, p = .076, t = -1.85). The grid’s score is in between interface. The average numbers of items clicked by a single user (3.33). Secondly, due to the change of layout, users perceived pie are 3.96, 3.875, and 4.84 in list, grid and pie respectively. The more competent in helping them find good movies (3.58 vs. 3.29 difference between grid and pie is even marginally significant (p in grid, p = .09, t = -1.77; list: 3.33), easier to use (3.5 in pie = 0.076, t = -1.86, by paired samples t-test). The exact distribution against 3 in list, t = -2.77, p = .01; the difference between grid and of the average user’s clicks among the eight areas is shown in list is also marginally significant: 3.375 vs. 3, p = .095), and Figure 5, from which we can see that above 50% of a user’s clicks easier to modify their profile (3.375 in pie vs. 3.04 in list, t = - on list were in areas 1 (28.42%) and 4 (27.37%), followed by 1.88, p = .07). Moreover, users rated higher on pie’s ability in areas 3 and 2. In grid and pie, the two hotter areas are also 1 and providing good tag suggestions (3.46 in pie vs. 3 in list, t = .2.41, 4, but the comparison regarding areas 2 and 3 shows that the p = .02; vs. 2.9 in grid, t = -2.25, p = .03). They also felt more clicks on them are more evenly distributed in pie (respectively enjoyable while using pie than list (3.42 against 2.875 in list, t = - 17.24% and 18.10%), which in fact also has higher total amount 2.72, p = .01; grid: 3.12). The median and mode values are also of clicks than in grid. reported in Table 2. Moreover, the clicking distribution across pages 1 and 2 is Table 2. Users’ subjective perceptions with the three layouts (L: significantly different among the three interfaces. More clicks List; G: Grid; P: Pie) appeared in grid’s second page (24.73% accumulated from areas 5 Mean (st.d) Median Mode to 8), and pie’s (19.83%), against 7.37% in list. This finding L G P L G P L G P suggests that grid and pie might more likely stimulate users to Q1 3.125 3.33 3.54*L 3 3.5 4 3 4 4 (.85) (.92) (.78) click the “More Movies” button for viewing more recommended Q2 3.33 3.29 3.58*G 3 3.5 4 4 4 4 items. In this regard, we further found that 50% of users have (.82) (1.04) (.72) actually gone to the second page while using grid, followed by Q3 3 3.375* L 3.5* L 3 3 4 3 3 4 41.7% users who did so in pie, and 25% in list (p = .056 between (.88) (.92) (.88) L,G Q4 3 2.92 3.46* 3 3 3.5 4 3 4 grid and list, t = -2.01). (.98) (1.02) (.88) Q5 3.04 3.17 3.375*L 3 3 4 3 3 4 (.91) (.92) (.92) Q6 2.875 3.17 3.42*L 3 3 3.5 3 4 4 (.74) (.96) (.93) L Q7 2.92 3.17 3.29* 3 3 3.5 3 3 4 (.83) (.92) (.95) Note: Asterisks denote highly or marginally significant differences to the respective abbreviated interfaces (by paired samples t-test). 4.3.3 User Comments At the end of the study, we also asked each user to give some free Figure 4. The distribution of users’ first clicks. comments on the interfaces. 9 users explicitly praised pie. As quoted from their words, “it is easy for me to see all without scrolling the page”, “easy, clear, more information”, “easy to use”, “no need to loop around as the movies are all in the middle”, etc. Similar preference was also given to grid: “I can get a glimpse of all movies within a page”, “the layout of displaying movie is good for browsing”, “it lists more movies”, “the item displayed clearly, and no need to scroll up or scroll down for watching the information”. Thus, the obvious advantage of pie and grid, as user perceived, is that they allow them to easily see many choices without scrolling and facilitate them to browse and seek info. On the other hand, the Figure 5. The distribution of an average user’s whole clicks comments to list were mainly negative (as stated by 14 users): “find during her interaction session with an interface. the movie difficultly”, “need to scroll down”, “not easy to use”, “I As for the total time spent on each interface, on average, it is can’t see all suggested movies at once”, “too long inefficient take 156.375 seconds in list, 109.875 seconds in grid, and 152.667 in effort to scroll”, etc. Therefore, the frequent reason behind users’ 25 disliking is that the list is not easy for them to see all suggested [8] Pu, P and Chen, L. Trust-Inspiring Interfaces for movies and demands more effort. Recommender Systems. Journal of Knowledge-Based Systems (KBS), vol. 20 (6), 542-556, 2007. 5. CONCLUSIONS AND FUTURE WORK [9] Pu, P and Chen, L. A user-centric evaluation framework of In conclusion, this paper reports our in-depth studying of users’ recommender systems. In Proc. RecSys’10 Workshop on decision behavior and attitudes in different recommender User-Centric Evaluation of Recommender Systems and Their interface layouts. Specifically, we compared three typical layout Interfaces (UCERSTI’10), 14-21, 2010. designs: list, grid and pie. The results revealed that in list and grid, users’ first clicks largely fall in the top three positions, but in [10] Ricci, F., Rokach, L., Shapira, B., and Kantor, P.B. (Eds.) pie they also came to other areas. The distribution of an average Recommender System Handbook. Springer, 2011. user’s whole set of clicks in an interface further showed that [11] Tsoi, H. K. and Chen, L. Incremental tag-aware user profile though the top three positions (i.e., the area 1) and the last three building to augment item recommendations. In 2nd positions (i.e., the area 4) are commonly popular in the three Workshop on Social Recommender Systems (SRS’11) in layouts, the clicks are more evenly distributed in pie among all CSCW’11, 2011. areas at its first page. Grid and pie are even more active in stimulating users to click items in the next recommendation page. From subjective measures and user comments, we found that users did prefer using pie and grid to list. Moreover, pie has been demonstrated with significant benefits in increasing users’ decision confidence, perceived interface competence, enjoyability, and usage intention. For our future work, we will conduct more user studies, including eye-tracking experiments, to track users’ eye-movement behavior in the recommender interfaces. Another interesting topic will be to investigate the interaction effect from items’ relevance ordering with the layout. That is, when the ordering was changed (i.e., reversed ascending order instead of regular descending order), would users’ behavior be influenced or not? In fact, with the varied ordering condition, we are able to identify whether users would spontaneously evaluate the item’s relevance, or their selection behavior would be largely influenced by the layout. For example, in the list interface, would they still select items at the top though they are least relevant? The relative role of layout against the relevance ordering could be hence revealed. 6. REFERENCES [1] Braganza, C., Marriott, K., Moulder, P., Wybrow, M. and Dwyer, T. Scrolling behaviour with single- and multi- column layout. In Proc. WWW 2009, 831-840. [2] Callahan, J., Hopkins, D., Weiser, M. and Shneiderman, B. An empirical comparison of pie vs. linear menus. In Proc. CHI 1988, ACM, 95-100. [3] Card, S. K., Newell, A. and Moran, T. P. The Psychology of Human-Computer Interaction. L. Erlbaum Assoc. Inc., Hillsdale, NJ, USA, 1983. [4] Chen, L. and Pu, P. Evaluating critiquing-based recommender agents. In Proc. AAAI 2006, 157-162, 2006. [5] Chen, L. and Pu, P. Eye-Tracking Study of User Behavior in Recommender Interfaces. In Proceedings of 2010 International Conference on User Modeling, Adaptation and Personalization (UMAP’10), 375-380, Big Island, Hawaii, USA, June 20-24, 2010. [6] Halverson, T. and Hornof, A. J. A minimal model for predicting visual search in human-computer interaction. In Proc. CHI 2007, 431-434. [7] Kammerer, Y. and Gerjets, P. How the interface design influences users' spontaneous trustworthiness evaluations of web search results: comparing a list and a grid interface. In Proc. ETRA 2010, 299-306. 26