Exploring User-Controlled Hybrid Recommendation in Conference Contexts Chun-Hua Tsai Behnam Rahdari Peter Brusilovsky University of Pittsburgh University of Pittsburgh University of Pittsburgh Pittsburgh, USA Pittsburgh, USA Pittsburgh, USA cht77@pitt.edu ber58@pitt.edu peterb@pitt.edu ABSTRACT Typically, paralleled hybrid recommender fuses multiple rele- A hybrid recommender system fuses multiple data sources to deliver vance sources by assigning static weights to different sources. The recommendations. One challenge of this approach is to match the optimal weights are trained or learned using ground truth data changing user preferences with a list of static recommendations. In (i.e., known ratings). The problem with this approach is that users this paper, we present two user-controllable hybrid recommender might seek recommendations for different reasons and in different interfaces, Relevance Tuner (for people recommendation) and Paper contexts. The individual sources in a hybrid recommender might Tuner (for paper recommendation), which offer a set of sliders to become more or less valuable depending on each case. As a result, tune the multiple relevance sources on the final recommendation while the “optimal” static fusion could provide the best ranking ranking on-the-fly. We deployed the user interfaces to a real-world with high algorithm accuracy, it might be sub-optimal for the users international academic conference with a field study. The result in some specific cases. of the log analysis showed the conference attendees did adopt the The problem of optimal source fusion has been originally ex- interface in exploring the hybrid recommendations. The finding plored in the domain of information retrieval where it was demon- provided evidence in supporting the proposed controllable interface strated that the user might be in a better position to decide which can be deployed to a broader set of conference context. weight should be assigned to each relevance source in each case [2]. The idea of user-controlled personalization has been further CCS CONCEPTS explored in recommender systems domain by O’Donovan et al., Schafer et al. [13, 15]. More recently, Bostandjiev et al. [4] intro- • Human-centered computing → Web-based interaction; User duced sliders as an approach to engage the user into tuning various interface design; Empirical studies in interaction design. parameters of a recommendation approach. Following that, the use of sliders as a way to support user-controlled fusion has been ex- KEYWORDS plored in the domain of recommender systems [14] and information Hybrid Recommendation; User-Driven Fusion; User Interface De- retrieval [7] brings additional evidence in favor of using sliders for sign; Conference Contexts user-controlled personalization. ACM Reference Format: In our past work, we explored sliders as a tool for the user- Chun-Hua Tsai, Behnam Rahdari, and Peter Brusilovsky. 2019. Exploring controlled hybrid recommendation in a research conference context User-Controlled Hybrid Recommendation in Conference Contexts. In Joint where it was applied to suggest meeting with the most relevant Proceedings of the ACM IUI 2019 Workshops, Los Angeles, USA, March 20, attendees [16]. A controlled user study demonstrated the benefits 2019, 6 pages. of this approach. However, it remains unclear whether conference attendees would adopt this approach outside of a controlled study 1 INTRODUCTION context where the use of sliders was strongly recommended. This Hybrid recommender systems [5] have been gradually becoming paper expands our study of user-controlled hybrid recommenda- more and more popular due to their ability to combine strong tion in the same conference context by adding two new aspects. features of different recommender approaches. One promising hy- First, we attempted to extend this approach by applying it to the bridization design is the paralleled hybrid recommender [5], which context of attendee, author, and paper recommendation. Second, fuse recommendation results produced by diverse types of existing instead of performing another controlled study, we assessed the recommender algorithms as well as multiple kinds of traces left by new implementation in an uncontrolled field study by releasing the modern internet users, i.e., browsing trails, bookmarks, ratings, cre- updated system to attendees of the EC-TEL 2018 conference. ated social links, etc. In this paper, we will refer to each contributing The results of our log analysis showed the conference atten- data source or approach that can generate a list of recommendation dees did adopt the proposed controllable interface in browsing the ranked by relevance to the target user as a relevance source. Each recommendations. The finding supported the effectiveness of the of these sources could be used to build a profile of user interests proposed user interface and its applicability in a broader set of and deliver valuable recommendations. conference context. In the following sections, we review a few like- minded research projects, explain the design of the user-controllable recommender interface and how it can be applied for recommend- IUI Workshops’19, March 20, 2019, Los Angeles, USA ing both papers and people, and review research evidence obtained Copyright © 2019 for the individual papers by the papers’ authors. Copying permitted for private and academic purposes. This volume is published and copyrighted by its from this field study. editors. IUI Workshops’19, March 20, 2019, Los Angeles, USA Chun-Hua Tsai, Behnam Rahdari, and Peter Brusilovsky 2 RELATED WORK 3 USER CONTROLLED HYBRID Nowadays, it became easier to leverage large amounts of user data RECOMMENDATIONS FOR ACADEMIC to enhance personalization in online applications. A recommender CONFERENCES system can create user models that utilize users’ web browsing In this paper, we discuss a visual interface design with user-driven trails, item ratings, demographic information or connected social control function and meaningful visual encoding. The design aims networks for providing personalized recommendations in differ- to help the users to inspect or tune the ranked recommendations ent contexts. An effective user model can predict the relevance in a hybrid recommender system with multiple relevance sources. of each recommendable item for the user [12]. In search of bet- Our proposed interface combines several features that have been ter performance (i.e., algorithm accuracy), multiple data sources found beneficial by the past work including slider control of source or recommendation techniques could be fused using different hy- importance [14, 16] and stackable bars for visualizing combined bridization strategies, e.g., paralleled hybrid recommender fuses relevance [7, 16]. The design can be applied to general relevance multiple relevance sources by training a classifier for determining exploration tasks or recommendation contexts. However, in this pa- the relevance sources’ weighting [5]. The approaches have been per, we particularly focus on two conferences-focused information widely adopted in many real-world online applications. However, needs, i.e., people and paper recommendations. user interests and information needs might not be constant, which We implemented the design as two recommender user interfaces: makes it is hard to predict user preferences in every situation. That Relevance Tuner (for people recommendation) and Paper Tuner is, it is difficult to find a “one-fit-for-all” weights for a paralleled (for paper recommendation). The two interfaces were served as hybrid recommender in all cases. To overcome this limitation, one components of conference support system Conference Navigator promising solution is to offer some form of user control so the users 3 (CN3), which equipped with the recommendation functions to can interact with the system based on their current situation. the conference attendees. CN3 has been used to support more Bringing user control to a hybrid recommender system allows than 45 conferences at the time of writing this paper and has data the users to have an immediate effect on the recommendations [12], on approximately 7,045 articles presented at these conferences; i.e., the users can further filter or re-sort the recommendation based 13,055 authors; 7,407 attendees; 32,461 bookmarks; and 1,565 social on their preference or information need. It usually requires an in- connections. The earlier version of the Relevance Tuner has been teractive visualization framework that combines recommendation explored in a controlled user study [16], however, in the past we with visualization techniques to support user interaction or inter- have not explored this approach in different contexts and have not vention into the recommendation process [11]. The idea of the user- assessed it in a field study. controllable interface of different recommendation approaches was originally presented in [15]. Bostandjiev et al. [4] suggested a slider- 3.1 People Recommendation: Relevance Tuner based interface that the user can adjust the weights of the items and the social connections. Verbert et al. [19] encouraged users to 3.1.1 Relevance Sources. To rank the recommended attendees by choose the most appropriate sources of relevance for each case and their relevance to the target user, the system uses five separate rec- provided a cluster-map interface to support user-driven exploration ommender engines that rank other attendees along five dimensions. and control of tags, agents and users. Parra and Brusilovsky [14] at- Text similarity of their academic publications, topic similarity of tempted to increase both controllability and transparency of hybrid research interests using topic modeling, social similarity through recommendation by using a combination of sliders for controlling the co-authorship network, similarity of current interests measured the fusion and a Venn diagram to visualize results. Ekstrand et al. [9] as intersection of their bookmarked talks, and the distance of their discussed a recommender-switching feature to let the users choose place of affiliation to the target user. Each of the relevance is defined recommender algorithms. Tsai and Brusilovsky [16] offered user below. controllable interfaces, a two-dimensional scatter-lot and multiple • Publication Similarity is determined by the degree of pub- relevance sliders, to a social recommender system for conference lication similarity between two attendees using cosine simi- attendees. Bailey et al. [3] further provide a visualization for data larity. The function is defined as: analytic task using the conference data. Sim Academic (x, y) = (t x · ty )/∥t x ∥∥ty ∥ (1) User controllability has also been recognized as a crucial compo- nent in supporting the exploratory search, i.e., allowing the users where t is word vectors for user x and y. We used TF*IDF to to narrow down the number of items and inspect the details during create the vector with a word frequency upper bound of 0.5 the information seeking process [6]. Ahn et al. [1] present a sum- and lower bound of 0.01 to eliminate both commonly and mary of search results in the form of entity clouds, which allows rarely used words. the users to explore the results in a controllable interface. Han et al. [10] offered users an option to re-sort people search results based • Topic Similarity is a metric that measures the distance on multiple user-related factors. di Sciascio et al. [7] proposed a between topic distributions [8]. This is another approach uRank interface for understanding, refining and reorganizing doc- to measure the similarity between the publications of two uments. di Sciascio et al. [6] integrated controllable social search researchers. The approach assumes that a mixture of topics functionality into an exploratory search system. An effective inter- is used to generate a string (document), where each topic active visualization representation can enable users to control the is a distribution of topical words. A recommender engine, process of recommendation [11]. based on the topic-based approach, can represent the schol- ars’ research interests through the learned topics. The topic Exploring User-Controlled Hybrid Recommendation in Conference Contexts IUI Workshops’19, March 20, 2019, Los Angeles, USA Figure 1: Relevance Tuner: the interface attached five controllable sliders to a people recommender interface, which allows the users to adjust the importance or preference of different relevance aspects. similarity could be computed as the pairwise similarity of • Section B shows the stackable relevance score bar of each the topic distributions [17]. recommended item in the ranked list. The color corresponds to the features in section A. It would adaptively adjust the bar • Co-Authorship Similarity approximates the social simi- score (length) from 0 to 20, based on the weighting percent- larity between the target and recommended users by combin- age of the sliders. A stackable color bar interface is known ing co-authorship network distance and common neighbor for its ability to enhance controllability and transparency similarity from publication data. We adopted the depth-first in a multi-aspect ranking [7]. In our system, the stackable search (DFS) method to calculate the shortest path p and color bars help the user to see how different relevant aspects common neighborhood (CN) for the number n of coauthors of a recommended item are coordinated while adding trans- overlapping in two degrees for users x and y. parency to the multi-aspect recommendation process. Sim Social (x, y) = p + n (2) • Section C shows the recommended scholar’s meta-data, in- • CN3 Interest Similarity is determined by the the number cluding name, social connection, affiliation, position, title, of co-bookmarked papers and co-connected authors within and country. The user can sort the ranked list by clicking the experimental social system. The function is defined as the head of each column or can inspect the explanation Sim I nt er est (x, y) = (bx ) ∩ (by ) + (c x ) ∩ (cy ) (3) tabs (same as Section C in Figure 1) by clicking the explana- tion icon [18], which is designed to enhance the algorithmic where bx , by represent the paper bookmarking of user x and transparency by offering several visualizations regarding the y; c x , cy represents the friend connection of user x and y. recommendation relevance. • Geographic Distance is a measure of geographic distance between attendees. We retrieve longitude and latitude data 3.2 Paper Recommendation: Paper Tuner based on attendees’ affiliation information. We used the 3.2.1 Relevance Sources. The current implementation of Paper Haversine formula to compute the geographic distance be- Tuner uses three personalized and two global social contexts to tween any pair of attendees. generate the hybrid recommendation. Each source uses a different Sim Dist ance (x, y) = Haversine(Geo x , Geoy ) (4) type of information to estimate the relevance of each recommended paper to the target user. where Geo are pairs of latitude and longitude coordinates for user x and y. • Publication Similarity estimates relevance by the degree of text similarity between the user’s past publications and 3.1.2 Visual Design. The design of the Relevance Tuner is shown the recommended item. We first create a bag of words by in Figure 1 and firstly introduced by [16]. The design can be sum- concatenating Title, Abstract, and Keywords of each publi- marized in three sections. cation and then use TF*IDF to create the word frequency • Section A contains five controllable sliders with the dif- vector. This vector is compared with a similar vector created ferent colors representing the features of the Personalized from user publications using traditional cosine similarity. Relevance Model. The scale of the slider ranges from 0 to 10. The user can change the weighting on the fly to re-rank the • Bookmark Similarity is determined by the degree of text ranked recommendation list. It provides controllability for similarity between “bookmarked” presentations (i.e., presen- the user to adjust the ranking to different recommendation tation that the user added to her personalized schedule in needs and preferences. CN3) and the recommended item. Similarly to Publication Similarity, we create a weighted vector of keywords for all IUI Workshops’19, March 20, 2019, Los Angeles, USA Chun-Hua Tsai, Behnam Rahdari, and Peter Brusilovsky Figure 2: The interface of Paper Tuner with sliders, which allow to adjust the importance of different relevance aspects. papers in the user’s scheduled papers list and compare it to changes the sliders, i.e., the length of the “green” section will the vector of the recommended item. increase when the green slider is moved right. • Followee Similarity is based on the ability to follow an- • The ranked list of results also provides details for each rec- other user provided by the conference support system as well ommended item (Figure 2: Section C). Users can click on as by many modern social networks. We create a weighed the link on Paper Title and Author(s) columns to get more keyword vector from the entire collection of papers pub- information such as the abstract of the publication, people lished by the user’s followees and estimate relevance as the planning to attending the presentation, etc. cosine similarity between this vector and the vector of the recommended item. 4 FIELD STUDY • Publication Popularity: Unlike three previous relevance 4.1 Context and Data Collection sources, Publication Popularity offers not personalized, but To explore the value of the two interface designs, Relevance Tuner social relevance ranking. The Publication popularity is de- and Paper Tuner, we organized a field study in the EC-TEL 2018 termined by the total number of bookmarks received by an conference held in Leeds (UK) from September 3 to 6, 2018. The item in CN3. We normalized this numerical value and use it two interfaces were released to all conference users as a part of to rank items by popularity. their host system Conference Navigator 3 (CN3). To mitigate the cold start problem that occurs when users have no publications or • Author Popularity: Similar to Publication popularity this co-authorship information related to the event for which the recom- is a social relevance source based on the popularity for each mendations are produced, the system integrates the AMiner dataset. author in the system. The popularity of each author is calcu- The live system is available at http://halley.exp.sis.pitt.edu/cn3/. lated by the average number of bookmarks received by the We sent out an invitation email seven days before the conference author’s publications in the system. Once we had this value date to introduce the recommendation feature available in CN3 to for each author, we can define the Author popularity of each all the 158 attendees of the conference. The user IDs were created for recommended item as the average popularity of its authors. each conference attendee based on their registration data. The EC- TEL 2018 conference had accepted a total of 142 papers. We deployed 3.2.2 Visual Design. The Paper Tuner is an interface for user con- and collected system log data from August 27 to September 14, 2018, trollable recommendation of research papers. It consist of three which is one week before and after the official conference date. main parts (Figure 2). The conference attendees also received several reminder emails during the event date, and the CN3 system link was attached to • Section A contains five sliders to control the importance of the homepage of the conference website. The conference website recommendation sources used to generate the ranked list of is located at http://www.ec-tel.eu/index.php?id=805. the results. Users can adjust the weight of each source from 0 to 10 by sliding to the right (increase) or left (decrease). Setting the value of each criterion to 0 will disable the con- 4.2 Log Analysis for People Recommendation tribution of that source to the final results. Table 1 presents the system usage for Relevance Tuner. A total of 44 users accessed pages with recommended authors or attendees. • Section B located on the right side of the interface and dis- Around 30% of these users (14 users) interacted with the tuner func- plays a stacked relevance bar next to each result. The full tion. The users in Tuner Group (those who click on the sliders at length of the bar displays the combined relevance of a rec- least once) tuned the recommendations 14.28 times on average. The ommended item to the target user. Each colored segment slider Publication Similarity and related to it slider Topic Similar- displays how much a specific source contributed to the to- ity received the highest user attention followed by Co-Authorship tal relevance given the current position of the source slider. Similarity, however the slider of CN3 Interest Similarity was used The segments of the stacked bars update each time the user less frequently. The data indicated that the conference attendees Exploring User-Controlled Hybrid Recommendation in Conference Contexts IUI Workshops’19, March 20, 2019, Los Angeles, USA Table 1: User Interaction Log: Relevance Tuner All Users Tuner Group Non-Tuner Group User User User Action M (SE) M (SE) M (SE) Count Count Count View Author/Attendee Page 3.61 (3.37) 44/44 3.92 (2.46) 14/14 3.46 (3.74) 30/30 Name Link Clicks 0.95 (1.66) 19/44 1.21 (1.88) 8/14 0.83 (1.57) 11/30 Paper Link Clicks 0.50 (2.00) 8/44 1.00 (3.46) 2/14 0.26 (0.63) 6/30 Total Tuner Clicks 4.54 (11.16) 14/44 14.28 (16.20) 14/14 0 (0) 0/30 Tuner: Publication 1.61 (4.13) 8/44 5.07 (6.14) 8/14 0 (0) 0/30 Tuner: Topic 1.11 (3.03) 7/44 3.50 (4.63) 7/14 0 (0) 0/30 Tuner: Co-Authorship 1.02 (3.92) 5/44 3.21 (6.57) 5/14 0 (0) 0/30 Tuner: CN3 Interest 0.18 (0.62) 4/44 0.57 (1.01) 4/14 0 (0) 0/30 Tuner: Geographic 0.63 (2.70) 4/44 2.00 (4.60) 4/14 0 (0) 0/30 Total Explanation Clicks 1.54 (4.11) 8/44 1.00 (2.54) 3/14 1.80 (4.69) 5/30 Exp: Publication 0.34 (0.80) 8/44 0.28 (0.72) 2/14 0.36 (0.85) 6/30 Exp: Topic 0.43 (1.12) 7/44 0.42 (1.15) 2/14 0.43 (1.13) 5/30 Exp: Co-Authorship 0.50 (1.64) 7/44 0.21 (0.57) 2/14 0.63 (1.95) 5/30 Exp: CN3 Interest 0.11 (0.38) 4/44 0.07 (0.26) 1/14 0.13 (043) 3/30 Exp: Geographic 0.15 (0.56) 4/44 0 (0) 0/14 0.23 (0.67) 4/30 Table 2: User Interaction Log: Paper Tuner Action Total Adjustments Median Standard Deviation User Count Total Tuner Adjustment 112 9.33 5.35 12/12 Tuner : Publication Similarity 32 2.67 1.44 11/12 Tuner : Bookmark Similarity 22 1.83 1.85 8/12 Tuner : Followee Similarity 21 1.75 1.60 8/12 Tuner : Publication Popularity 19 1.58 1.78 7/12 Tuner : Author Popularity 18 1.50 1.09 10/12 Reverse Functionality 15 1.25 1.14 8/12 Paper Link Clicks 28 2.33 0.98 12/12 Author Link Click 19 1.58 1.16 9/12 emphasize the publication text in exploring the conference authors 5 SUMMARY AND CONCLUSION and attendees. Note, however, that for those who are new to the sys- In our past work, we explored sliders as a tool for the user-controlled tem, the CN3 Interest may be less useful due to lack of bookmarking hybrid recommendation of conference attendees in a controlled data. It might explain the lower use of the slider. user study [16]. In this paper, we attempted to implement the idea of user-controlled hybrid recommendation in a broader set of con- ference context and explore it in a field study. The result of log analysis indicated that the users to a considerable extent adopted 4.3 Log Analysis for Paper Recommendation the user-controlled interface in exploring the hybrid people and Table 2 provides information about authors and attendees inter- paper recommendations of a conference. This result provides some actions with the paper tuner during the EC-TEL 2018 conference. early evidence about the effectiveness of the proposed user inter- The analysis revealed that all system users who explored the Paper face in a real-world, outside of controlled student context setting. Tuner component used the sliders to control the weights of rele- Based on the preliminary finding, we see a great potential to deploy vance sources. As in the case of Relevance Tuner, the publication the controllable interface to other relevance exploration tasks or similarity was the most popular slider. For the case of the papers, recommendation contexts. however, the bookmark similarity was the second most popular We also are aware of some limitations in this experiment. First, one. Social relevance sources were adjusted less frequently than the multiple relevances were combined with linear fashion. Second, personalized sources. Altogether, it looks like the Paper Tuner pro- we reported the observational findings due to the nature of the field. vided valuable information to the users as they frequently requested The experimental robustness requires further investigating. Third, additional details about recommended papers including 28 clicks in each interface, we had engaged only a small sample of users. All to receive paper details and 19 clicks to receive author details. these limitations would be addressed in our future works. IUI Workshops’19, March 20, 2019, Los Angeles, USA Chun-Hua Tsai, Behnam Rahdari, and Peter Brusilovsky REFERENCES [11] Chen He, Denis Parra, and Katrien Verbert. 2016. Interactive recommender [1] Jae-wook Ahn, Peter Brusilovsky, Jonathan Grady, Daqing He, and Radu Florian. systems: A survey of the state of the art and future research challenges and 2010. Semantic annotation based exploratory search for information analysts. opportunities. Expert Systems with Applications 56 (2016), 9–27. Information processing & management 46, 4 (2010), 383–402. [12] Dietmar Jannach, Sidra Naveed, and Michael Jugovac. 2016. User control in [2] Jae-wook Ahn, Peter Brusilovsky, Daqing He, Jonathan Grady, and Qi Li. 2008. recommender systems: Overview and interaction challenges. In International Personalized Web Exploration with Task Models. In the 17th international confer- Conference on Electronic Commerce and Web Technologies. Springer, 21–33. ence on World Wide Web, WWW ’08. ACM, 1–10. [13] John O’Donovan, Barry Smyth, Brynjar Gretarsson, Svetlin Bostandjiev, and [3] Samuel M Bailey, Justin A Wei, Chaoli Wang, Denis Parra, and Peter Brusilovsky. Tobias Höllerer. 2008. PeerChooser: Visual Interactive Recommendation. In 2018. CNVis: A Web-Based Visual Analytics Tool for Exploring Conference Proceedings of the SIGCHI Conference on Human Factors in Computing Systems Navigator Data. Electronic Imaging 2018, 1 (2018), 1–11. (CHI ’08). ACM, New York, NY, USA, 1085–1088. https://doi.org/10.1145/1357054. [4] Svetlin Bostandjiev, John O’Donovan, and Tobias Höllerer. 2012. TasteWeights: a 1357222 visual interactive hybrid recommender system. In Proceedings of the sixth ACM [14] Denis Parra and Peter Brusilovsky. 2015. User-controllable personalization: A conference on Recommender systems. ACM, 35–42. case study with SetFusion. International Journal of Human-Computer Studies 78 [5] Robin Burke. 2007. Hybrid Web recommender systems. In The Adaptive (2015), 43–67. Web: Methods and Strategies of Web Personalization, Peter Brusilovsky, Alfred [15] J. Ben Schafer, Joseph A. Konstan, and John Riedl. 2002. Meta-recommendation Kobsa, and Wolfgang Neidl (Eds.). Lecture Notes in Computer Science, Vol. 4321. Systems: User-controlled Integration of Diverse Recommendations. In Proceedings Springer-Verlag, Berlin Heidelberg New York, 377–408. of the Eleventh International Conference on Information and Knowledge Manage- [6] Cecilia di Sciascio, Peter Brusilovsky, and Eduardo Veas. 2018. A study on ment (CIKM ’02). ACM, New York, NY, USA, 43–51. https://doi.org/10.1145/ user-controllable social exploratory search. In 23rd International Conference on 584792.584803 Intelligent User Interfaces. ACM, 353–364. [16] Chun-Hua Tsai and Peter Brusilovsky. 2018. Beyond the Ranked List: User-Driven [7] Cecilia di Sciascio, Vedran Sabol, and Eduardo E. Veas. 2016. Rank As You Go: Exploration and Diversification of Social Recommendation. In 23rd International User-Driven Exploration of Search Results. In the 21st International Conference Conference on Intelligent User Interfaces (IUI ’18). ACM, New York, NY, USA, on Intelligent User Interfaces (IUI’16). ACM, 118–129. https://doi.org/10.1145/ 239–250. https://doi.org/10.1145/3172944.3172959 2856767.2856797 [17] Chun-Hua Tsai and Peter Brusilovsky. 2019. Explaining Recommendations in an [8] Jianguang Du, Jing Jiang, Dandan Song, and Lejian Liao. 2015. Topic modeling Interactive Hybrid Social Recommender. In Proceedings of the 2019 Conference on with document relative similarities. IJCAI. Intelligent User Interface. ACM, 1–12. [9] Michael D Ekstrand, Daniel Kluver, F Maxwell Harper, and Joseph A Konstan. [18] Chun-Hua Tsai and Peter Brusilovsky. 2019. Exploring Social Recommendations 2015. Letting users choose recommender algorithms: An experimental study. In with Visual Diversity-Promoting Interfaces. ACM Transactions on Interactive Proceedings of the 9th ACM Conference on Recommender Systems. ACM, 11–18. Intelligent Systems (TiiS) (2019). [10] Shuguang Han, Daqing He, Jiepu Jiang, and Zhen Yue. 2013. Supporting ex- [19] Katrien Verbert, Denis Parra, Peter Brusilovsky, and Erik Duval. 2013. Visualizing ploratory people search: a study of factor transparency and user control. In recommendations to support exploration, transparency and controllability. In Proceedings of the 22nd ACM international conference on Information & Knowledge Proceedings of the 2013 international conference on Intelligent user interfaces. ACM, Management. ACM, 449–458. 351–362.