=Paper=
{{Paper
|id=Vol-2327/ESIDA8
|storemode=property
|title=Exploring User-Controlled Hybrid Recommendation in a Conference Context
|pdfUrl=https://ceur-ws.org/Vol-2327/IUI19WS-ESIDA-8.pdf
|volume=Vol-2327
|authors=Chun-Hua Tsai,Peter Brusilovsky,Behnam Rahdari
|dblpUrl=https://dblp.org/rec/conf/iui/TsaiBR19
}}
==Exploring User-Controlled Hybrid Recommendation in a Conference Context==
https://ceur-ws.org/Vol-2327/IUI19WS-ESIDA-8.pdf
Exploring User-Controlled Hybrid Recommendation
in Conference Contexts
Chun-Hua Tsai Behnam Rahdari Peter Brusilovsky
University of Pittsburgh University of Pittsburgh University of Pittsburgh
Pittsburgh, USA Pittsburgh, USA Pittsburgh, USA
cht77@pitt.edu ber58@pitt.edu peterb@pitt.edu
ABSTRACT Typically, paralleled hybrid recommender fuses multiple rele-
A hybrid recommender system fuses multiple data sources to deliver vance sources by assigning static weights to different sources. The
recommendations. One challenge of this approach is to match the optimal weights are trained or learned using ground truth data
changing user preferences with a list of static recommendations. In (i.e., known ratings). The problem with this approach is that users
this paper, we present two user-controllable hybrid recommender might seek recommendations for different reasons and in different
interfaces, Relevance Tuner (for people recommendation) and Paper contexts. The individual sources in a hybrid recommender might
Tuner (for paper recommendation), which offer a set of sliders to become more or less valuable depending on each case. As a result,
tune the multiple relevance sources on the final recommendation while the “optimal” static fusion could provide the best ranking
ranking on-the-fly. We deployed the user interfaces to a real-world with high algorithm accuracy, it might be sub-optimal for the users
international academic conference with a field study. The result in some specific cases.
of the log analysis showed the conference attendees did adopt the The problem of optimal source fusion has been originally ex-
interface in exploring the hybrid recommendations. The finding plored in the domain of information retrieval where it was demon-
provided evidence in supporting the proposed controllable interface strated that the user might be in a better position to decide which
can be deployed to a broader set of conference context. weight should be assigned to each relevance source in each case
[2]. The idea of user-controlled personalization has been further
CCS CONCEPTS explored in recommender systems domain by O’Donovan et al.,
Schafer et al. [13, 15]. More recently, Bostandjiev et al. [4] intro-
• Human-centered computing → Web-based interaction; User
duced sliders as an approach to engage the user into tuning various
interface design; Empirical studies in interaction design.
parameters of a recommendation approach. Following that, the use
of sliders as a way to support user-controlled fusion has been ex-
KEYWORDS plored in the domain of recommender systems [14] and information
Hybrid Recommendation; User-Driven Fusion; User Interface De- retrieval [7] brings additional evidence in favor of using sliders for
sign; Conference Contexts user-controlled personalization.
ACM Reference Format: In our past work, we explored sliders as a tool for the user-
Chun-Hua Tsai, Behnam Rahdari, and Peter Brusilovsky. 2019. Exploring controlled hybrid recommendation in a research conference context
User-Controlled Hybrid Recommendation in Conference Contexts. In Joint where it was applied to suggest meeting with the most relevant
Proceedings of the ACM IUI 2019 Workshops, Los Angeles, USA, March 20, attendees [16]. A controlled user study demonstrated the benefits
2019, 6 pages. of this approach. However, it remains unclear whether conference
attendees would adopt this approach outside of a controlled study
1 INTRODUCTION context where the use of sliders was strongly recommended. This
Hybrid recommender systems [5] have been gradually becoming paper expands our study of user-controlled hybrid recommenda-
more and more popular due to their ability to combine strong tion in the same conference context by adding two new aspects.
features of different recommender approaches. One promising hy- First, we attempted to extend this approach by applying it to the
bridization design is the paralleled hybrid recommender [5], which context of attendee, author, and paper recommendation. Second,
fuse recommendation results produced by diverse types of existing instead of performing another controlled study, we assessed the
recommender algorithms as well as multiple kinds of traces left by new implementation in an uncontrolled field study by releasing the
modern internet users, i.e., browsing trails, bookmarks, ratings, cre- updated system to attendees of the EC-TEL 2018 conference.
ated social links, etc. In this paper, we will refer to each contributing The results of our log analysis showed the conference atten-
data source or approach that can generate a list of recommendation dees did adopt the proposed controllable interface in browsing the
ranked by relevance to the target user as a relevance source. Each recommendations. The finding supported the effectiveness of the
of these sources could be used to build a profile of user interests proposed user interface and its applicability in a broader set of
and deliver valuable recommendations. conference context. In the following sections, we review a few like-
minded research projects, explain the design of the user-controllable
recommender interface and how it can be applied for recommend-
IUI Workshops’19, March 20, 2019, Los Angeles, USA ing both papers and people, and review research evidence obtained
Copyright © 2019 for the individual papers by the papers’ authors. Copying permitted
for private and academic purposes. This volume is published and copyrighted by its
from this field study.
editors.
IUI Workshops’19, March 20, 2019, Los Angeles, USA Chun-Hua Tsai, Behnam Rahdari, and Peter Brusilovsky
2 RELATED WORK 3 USER CONTROLLED HYBRID
Nowadays, it became easier to leverage large amounts of user data RECOMMENDATIONS FOR ACADEMIC
to enhance personalization in online applications. A recommender CONFERENCES
system can create user models that utilize users’ web browsing In this paper, we discuss a visual interface design with user-driven
trails, item ratings, demographic information or connected social control function and meaningful visual encoding. The design aims
networks for providing personalized recommendations in differ- to help the users to inspect or tune the ranked recommendations
ent contexts. An effective user model can predict the relevance in a hybrid recommender system with multiple relevance sources.
of each recommendable item for the user [12]. In search of bet- Our proposed interface combines several features that have been
ter performance (i.e., algorithm accuracy), multiple data sources found beneficial by the past work including slider control of source
or recommendation techniques could be fused using different hy- importance [14, 16] and stackable bars for visualizing combined
bridization strategies, e.g., paralleled hybrid recommender fuses relevance [7, 16]. The design can be applied to general relevance
multiple relevance sources by training a classifier for determining exploration tasks or recommendation contexts. However, in this pa-
the relevance sources’ weighting [5]. The approaches have been per, we particularly focus on two conferences-focused information
widely adopted in many real-world online applications. However, needs, i.e., people and paper recommendations.
user interests and information needs might not be constant, which We implemented the design as two recommender user interfaces:
makes it is hard to predict user preferences in every situation. That Relevance Tuner (for people recommendation) and Paper Tuner
is, it is difficult to find a “one-fit-for-all” weights for a paralleled (for paper recommendation). The two interfaces were served as
hybrid recommender in all cases. To overcome this limitation, one components of conference support system Conference Navigator
promising solution is to offer some form of user control so the users 3 (CN3), which equipped with the recommendation functions to
can interact with the system based on their current situation. the conference attendees. CN3 has been used to support more
Bringing user control to a hybrid recommender system allows than 45 conferences at the time of writing this paper and has data
the users to have an immediate effect on the recommendations [12], on approximately 7,045 articles presented at these conferences;
i.e., the users can further filter or re-sort the recommendation based 13,055 authors; 7,407 attendees; 32,461 bookmarks; and 1,565 social
on their preference or information need. It usually requires an in- connections. The earlier version of the Relevance Tuner has been
teractive visualization framework that combines recommendation explored in a controlled user study [16], however, in the past we
with visualization techniques to support user interaction or inter- have not explored this approach in different contexts and have not
vention into the recommendation process [11]. The idea of the user- assessed it in a field study.
controllable interface of different recommendation approaches was
originally presented in [15]. Bostandjiev et al. [4] suggested a slider- 3.1 People Recommendation: Relevance Tuner
based interface that the user can adjust the weights of the items
and the social connections. Verbert et al. [19] encouraged users to 3.1.1 Relevance Sources. To rank the recommended attendees by
choose the most appropriate sources of relevance for each case and their relevance to the target user, the system uses five separate rec-
provided a cluster-map interface to support user-driven exploration ommender engines that rank other attendees along five dimensions.
and control of tags, agents and users. Parra and Brusilovsky [14] at- Text similarity of their academic publications, topic similarity of
tempted to increase both controllability and transparency of hybrid research interests using topic modeling, social similarity through
recommendation by using a combination of sliders for controlling the co-authorship network, similarity of current interests measured
the fusion and a Venn diagram to visualize results. Ekstrand et al. [9] as intersection of their bookmarked talks, and the distance of their
discussed a recommender-switching feature to let the users choose place of affiliation to the target user. Each of the relevance is defined
recommender algorithms. Tsai and Brusilovsky [16] offered user below.
controllable interfaces, a two-dimensional scatter-lot and multiple • Publication Similarity is determined by the degree of pub-
relevance sliders, to a social recommender system for conference lication similarity between two attendees using cosine simi-
attendees. Bailey et al. [3] further provide a visualization for data larity. The function is defined as:
analytic task using the conference data. Sim Academic (x, y) = (t x · ty )/∥t x ∥∥ty ∥ (1)
User controllability has also been recognized as a crucial compo-
nent in supporting the exploratory search, i.e., allowing the users where t is word vectors for user x and y. We used TF*IDF to
to narrow down the number of items and inspect the details during create the vector with a word frequency upper bound of 0.5
the information seeking process [6]. Ahn et al. [1] present a sum- and lower bound of 0.01 to eliminate both commonly and
mary of search results in the form of entity clouds, which allows rarely used words.
the users to explore the results in a controllable interface. Han et al.
[10] offered users an option to re-sort people search results based • Topic Similarity is a metric that measures the distance
on multiple user-related factors. di Sciascio et al. [7] proposed a between topic distributions [8]. This is another approach
uRank interface for understanding, refining and reorganizing doc- to measure the similarity between the publications of two
uments. di Sciascio et al. [6] integrated controllable social search researchers. The approach assumes that a mixture of topics
functionality into an exploratory search system. An effective inter- is used to generate a string (document), where each topic
active visualization representation can enable users to control the is a distribution of topical words. A recommender engine,
process of recommendation [11]. based on the topic-based approach, can represent the schol-
ars’ research interests through the learned topics. The topic
Exploring User-Controlled Hybrid Recommendation in Conference Contexts IUI Workshops’19, March 20, 2019, Los Angeles, USA
Figure 1: Relevance Tuner: the interface attached five controllable sliders to a people recommender interface, which allows
the users to adjust the importance or preference of different relevance aspects.
similarity could be computed as the pairwise similarity of • Section B shows the stackable relevance score bar of each
the topic distributions [17]. recommended item in the ranked list. The color corresponds
to the features in section A. It would adaptively adjust the bar
• Co-Authorship Similarity approximates the social simi- score (length) from 0 to 20, based on the weighting percent-
larity between the target and recommended users by combin- age of the sliders. A stackable color bar interface is known
ing co-authorship network distance and common neighbor for its ability to enhance controllability and transparency
similarity from publication data. We adopted the depth-first in a multi-aspect ranking [7]. In our system, the stackable
search (DFS) method to calculate the shortest path p and color bars help the user to see how different relevant aspects
common neighborhood (CN) for the number n of coauthors of a recommended item are coordinated while adding trans-
overlapping in two degrees for users x and y. parency to the multi-aspect recommendation process.
Sim Social (x, y) = p + n (2)
• Section C shows the recommended scholar’s meta-data, in-
• CN3 Interest Similarity is determined by the the number cluding name, social connection, affiliation, position, title,
of co-bookmarked papers and co-connected authors within and country. The user can sort the ranked list by clicking
the experimental social system. The function is defined as the head of each column or can inspect the explanation
Sim I nt er est (x, y) = (bx ) ∩ (by ) + (c x ) ∩ (cy ) (3) tabs (same as Section C in Figure 1) by clicking the explana-
tion icon [18], which is designed to enhance the algorithmic
where bx , by represent the paper bookmarking of user x and
transparency by offering several visualizations regarding the
y; c x , cy represents the friend connection of user x and y.
recommendation relevance.
• Geographic Distance is a measure of geographic distance
between attendees. We retrieve longitude and latitude data 3.2 Paper Recommendation: Paper Tuner
based on attendees’ affiliation information. We used the 3.2.1 Relevance Sources. The current implementation of Paper
Haversine formula to compute the geographic distance be- Tuner uses three personalized and two global social contexts to
tween any pair of attendees. generate the hybrid recommendation. Each source uses a different
Sim Dist ance (x, y) = Haversine(Geo x , Geoy ) (4) type of information to estimate the relevance of each recommended
paper to the target user.
where Geo are pairs of latitude and longitude coordinates
for user x and y. • Publication Similarity estimates relevance by the degree
of text similarity between the user’s past publications and
3.1.2 Visual Design. The design of the Relevance Tuner is shown the recommended item. We first create a bag of words by
in Figure 1 and firstly introduced by [16]. The design can be sum- concatenating Title, Abstract, and Keywords of each publi-
marized in three sections. cation and then use TF*IDF to create the word frequency
• Section A contains five controllable sliders with the dif- vector. This vector is compared with a similar vector created
ferent colors representing the features of the Personalized from user publications using traditional cosine similarity.
Relevance Model. The scale of the slider ranges from 0 to 10.
The user can change the weighting on the fly to re-rank the • Bookmark Similarity is determined by the degree of text
ranked recommendation list. It provides controllability for similarity between “bookmarked” presentations (i.e., presen-
the user to adjust the ranking to different recommendation tation that the user added to her personalized schedule in
needs and preferences. CN3) and the recommended item. Similarly to Publication
Similarity, we create a weighted vector of keywords for all
IUI Workshops’19, March 20, 2019, Los Angeles, USA Chun-Hua Tsai, Behnam Rahdari, and Peter Brusilovsky
Figure 2: The interface of Paper Tuner with sliders, which allow to adjust the importance of different relevance aspects.
papers in the user’s scheduled papers list and compare it to changes the sliders, i.e., the length of the “green” section will
the vector of the recommended item. increase when the green slider is moved right.
• Followee Similarity is based on the ability to follow an- • The ranked list of results also provides details for each rec-
other user provided by the conference support system as well ommended item (Figure 2: Section C). Users can click on
as by many modern social networks. We create a weighed the link on Paper Title and Author(s) columns to get more
keyword vector from the entire collection of papers pub- information such as the abstract of the publication, people
lished by the user’s followees and estimate relevance as the planning to attending the presentation, etc.
cosine similarity between this vector and the vector of the
recommended item. 4 FIELD STUDY
• Publication Popularity: Unlike three previous relevance 4.1 Context and Data Collection
sources, Publication Popularity offers not personalized, but To explore the value of the two interface designs, Relevance Tuner
social relevance ranking. The Publication popularity is de- and Paper Tuner, we organized a field study in the EC-TEL 2018
termined by the total number of bookmarks received by an conference held in Leeds (UK) from September 3 to 6, 2018. The
item in CN3. We normalized this numerical value and use it two interfaces were released to all conference users as a part of
to rank items by popularity. their host system Conference Navigator 3 (CN3). To mitigate the
cold start problem that occurs when users have no publications or
• Author Popularity: Similar to Publication popularity this co-authorship information related to the event for which the recom-
is a social relevance source based on the popularity for each mendations are produced, the system integrates the AMiner dataset.
author in the system. The popularity of each author is calcu- The live system is available at http://halley.exp.sis.pitt.edu/cn3/.
lated by the average number of bookmarks received by the We sent out an invitation email seven days before the conference
author’s publications in the system. Once we had this value date to introduce the recommendation feature available in CN3 to
for each author, we can define the Author popularity of each all the 158 attendees of the conference. The user IDs were created for
recommended item as the average popularity of its authors. each conference attendee based on their registration data. The EC-
TEL 2018 conference had accepted a total of 142 papers. We deployed
3.2.2 Visual Design. The Paper Tuner is an interface for user con- and collected system log data from August 27 to September 14, 2018,
trollable recommendation of research papers. It consist of three which is one week before and after the official conference date.
main parts (Figure 2). The conference attendees also received several reminder emails
during the event date, and the CN3 system link was attached to
• Section A contains five sliders to control the importance of the homepage of the conference website. The conference website
recommendation sources used to generate the ranked list of is located at http://www.ec-tel.eu/index.php?id=805.
the results. Users can adjust the weight of each source from
0 to 10 by sliding to the right (increase) or left (decrease).
Setting the value of each criterion to 0 will disable the con- 4.2 Log Analysis for People Recommendation
tribution of that source to the final results. Table 1 presents the system usage for Relevance Tuner. A total of
44 users accessed pages with recommended authors or attendees.
• Section B located on the right side of the interface and dis- Around 30% of these users (14 users) interacted with the tuner func-
plays a stacked relevance bar next to each result. The full tion. The users in Tuner Group (those who click on the sliders at
length of the bar displays the combined relevance of a rec- least once) tuned the recommendations 14.28 times on average. The
ommended item to the target user. Each colored segment slider Publication Similarity and related to it slider Topic Similar-
displays how much a specific source contributed to the to- ity received the highest user attention followed by Co-Authorship
tal relevance given the current position of the source slider. Similarity, however the slider of CN3 Interest Similarity was used
The segments of the stacked bars update each time the user less frequently. The data indicated that the conference attendees
Exploring User-Controlled Hybrid Recommendation in Conference Contexts IUI Workshops’19, March 20, 2019, Los Angeles, USA
Table 1: User Interaction Log: Relevance Tuner
All Users Tuner Group Non-Tuner Group
User User User
Action M (SE) M (SE) M (SE)
Count Count Count
View Author/Attendee Page 3.61 (3.37) 44/44 3.92 (2.46) 14/14 3.46 (3.74) 30/30
Name Link Clicks 0.95 (1.66) 19/44 1.21 (1.88) 8/14 0.83 (1.57) 11/30
Paper Link Clicks 0.50 (2.00) 8/44 1.00 (3.46) 2/14 0.26 (0.63) 6/30
Total Tuner Clicks 4.54 (11.16) 14/44 14.28 (16.20) 14/14 0 (0) 0/30
Tuner: Publication 1.61 (4.13) 8/44 5.07 (6.14) 8/14 0 (0) 0/30
Tuner: Topic 1.11 (3.03) 7/44 3.50 (4.63) 7/14 0 (0) 0/30
Tuner: Co-Authorship 1.02 (3.92) 5/44 3.21 (6.57) 5/14 0 (0) 0/30
Tuner: CN3 Interest 0.18 (0.62) 4/44 0.57 (1.01) 4/14 0 (0) 0/30
Tuner: Geographic 0.63 (2.70) 4/44 2.00 (4.60) 4/14 0 (0) 0/30
Total Explanation Clicks 1.54 (4.11) 8/44 1.00 (2.54) 3/14 1.80 (4.69) 5/30
Exp: Publication 0.34 (0.80) 8/44 0.28 (0.72) 2/14 0.36 (0.85) 6/30
Exp: Topic 0.43 (1.12) 7/44 0.42 (1.15) 2/14 0.43 (1.13) 5/30
Exp: Co-Authorship 0.50 (1.64) 7/44 0.21 (0.57) 2/14 0.63 (1.95) 5/30
Exp: CN3 Interest 0.11 (0.38) 4/44 0.07 (0.26) 1/14 0.13 (043) 3/30
Exp: Geographic 0.15 (0.56) 4/44 0 (0) 0/14 0.23 (0.67) 4/30
Table 2: User Interaction Log: Paper Tuner
Action Total Adjustments Median Standard Deviation User Count
Total Tuner Adjustment 112 9.33 5.35 12/12
Tuner : Publication Similarity 32 2.67 1.44 11/12
Tuner : Bookmark Similarity 22 1.83 1.85 8/12
Tuner : Followee Similarity 21 1.75 1.60 8/12
Tuner : Publication Popularity 19 1.58 1.78 7/12
Tuner : Author Popularity 18 1.50 1.09 10/12
Reverse Functionality 15 1.25 1.14 8/12
Paper Link Clicks 28 2.33 0.98 12/12
Author Link Click 19 1.58 1.16 9/12
emphasize the publication text in exploring the conference authors 5 SUMMARY AND CONCLUSION
and attendees. Note, however, that for those who are new to the sys- In our past work, we explored sliders as a tool for the user-controlled
tem, the CN3 Interest may be less useful due to lack of bookmarking hybrid recommendation of conference attendees in a controlled
data. It might explain the lower use of the slider. user study [16]. In this paper, we attempted to implement the idea
of user-controlled hybrid recommendation in a broader set of con-
ference context and explore it in a field study. The result of log
analysis indicated that the users to a considerable extent adopted
4.3 Log Analysis for Paper Recommendation the user-controlled interface in exploring the hybrid people and
Table 2 provides information about authors and attendees inter- paper recommendations of a conference. This result provides some
actions with the paper tuner during the EC-TEL 2018 conference. early evidence about the effectiveness of the proposed user inter-
The analysis revealed that all system users who explored the Paper face in a real-world, outside of controlled student context setting.
Tuner component used the sliders to control the weights of rele- Based on the preliminary finding, we see a great potential to deploy
vance sources. As in the case of Relevance Tuner, the publication the controllable interface to other relevance exploration tasks or
similarity was the most popular slider. For the case of the papers, recommendation contexts.
however, the bookmark similarity was the second most popular We also are aware of some limitations in this experiment. First,
one. Social relevance sources were adjusted less frequently than the multiple relevances were combined with linear fashion. Second,
personalized sources. Altogether, it looks like the Paper Tuner pro- we reported the observational findings due to the nature of the field.
vided valuable information to the users as they frequently requested The experimental robustness requires further investigating. Third,
additional details about recommended papers including 28 clicks in each interface, we had engaged only a small sample of users. All
to receive paper details and 19 clicks to receive author details. these limitations would be addressed in our future works.
IUI Workshops’19, March 20, 2019, Los Angeles, USA Chun-Hua Tsai, Behnam Rahdari, and Peter Brusilovsky
REFERENCES [11] Chen He, Denis Parra, and Katrien Verbert. 2016. Interactive recommender
[1] Jae-wook Ahn, Peter Brusilovsky, Jonathan Grady, Daqing He, and Radu Florian. systems: A survey of the state of the art and future research challenges and
2010. Semantic annotation based exploratory search for information analysts. opportunities. Expert Systems with Applications 56 (2016), 9–27.
Information processing & management 46, 4 (2010), 383–402. [12] Dietmar Jannach, Sidra Naveed, and Michael Jugovac. 2016. User control in
[2] Jae-wook Ahn, Peter Brusilovsky, Daqing He, Jonathan Grady, and Qi Li. 2008. recommender systems: Overview and interaction challenges. In International
Personalized Web Exploration with Task Models. In the 17th international confer- Conference on Electronic Commerce and Web Technologies. Springer, 21–33.
ence on World Wide Web, WWW ’08. ACM, 1–10. [13] John O’Donovan, Barry Smyth, Brynjar Gretarsson, Svetlin Bostandjiev, and
[3] Samuel M Bailey, Justin A Wei, Chaoli Wang, Denis Parra, and Peter Brusilovsky. Tobias Höllerer. 2008. PeerChooser: Visual Interactive Recommendation. In
2018. CNVis: A Web-Based Visual Analytics Tool for Exploring Conference Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Navigator Data. Electronic Imaging 2018, 1 (2018), 1–11. (CHI ’08). ACM, New York, NY, USA, 1085–1088. https://doi.org/10.1145/1357054.
[4] Svetlin Bostandjiev, John O’Donovan, and Tobias Höllerer. 2012. TasteWeights: a 1357222
visual interactive hybrid recommender system. In Proceedings of the sixth ACM [14] Denis Parra and Peter Brusilovsky. 2015. User-controllable personalization: A
conference on Recommender systems. ACM, 35–42. case study with SetFusion. International Journal of Human-Computer Studies 78
[5] Robin Burke. 2007. Hybrid Web recommender systems. In The Adaptive (2015), 43–67.
Web: Methods and Strategies of Web Personalization, Peter Brusilovsky, Alfred [15] J. Ben Schafer, Joseph A. Konstan, and John Riedl. 2002. Meta-recommendation
Kobsa, and Wolfgang Neidl (Eds.). Lecture Notes in Computer Science, Vol. 4321. Systems: User-controlled Integration of Diverse Recommendations. In Proceedings
Springer-Verlag, Berlin Heidelberg New York, 377–408. of the Eleventh International Conference on Information and Knowledge Manage-
[6] Cecilia di Sciascio, Peter Brusilovsky, and Eduardo Veas. 2018. A study on ment (CIKM ’02). ACM, New York, NY, USA, 43–51. https://doi.org/10.1145/
user-controllable social exploratory search. In 23rd International Conference on 584792.584803
Intelligent User Interfaces. ACM, 353–364. [16] Chun-Hua Tsai and Peter Brusilovsky. 2018. Beyond the Ranked List: User-Driven
[7] Cecilia di Sciascio, Vedran Sabol, and Eduardo E. Veas. 2016. Rank As You Go: Exploration and Diversification of Social Recommendation. In 23rd International
User-Driven Exploration of Search Results. In the 21st International Conference Conference on Intelligent User Interfaces (IUI ’18). ACM, New York, NY, USA,
on Intelligent User Interfaces (IUI’16). ACM, 118–129. https://doi.org/10.1145/ 239–250. https://doi.org/10.1145/3172944.3172959
2856767.2856797 [17] Chun-Hua Tsai and Peter Brusilovsky. 2019. Explaining Recommendations in an
[8] Jianguang Du, Jing Jiang, Dandan Song, and Lejian Liao. 2015. Topic modeling Interactive Hybrid Social Recommender. In Proceedings of the 2019 Conference on
with document relative similarities. IJCAI. Intelligent User Interface. ACM, 1–12.
[9] Michael D Ekstrand, Daniel Kluver, F Maxwell Harper, and Joseph A Konstan. [18] Chun-Hua Tsai and Peter Brusilovsky. 2019. Exploring Social Recommendations
2015. Letting users choose recommender algorithms: An experimental study. In with Visual Diversity-Promoting Interfaces. ACM Transactions on Interactive
Proceedings of the 9th ACM Conference on Recommender Systems. ACM, 11–18. Intelligent Systems (TiiS) (2019).
[10] Shuguang Han, Daqing He, Jiepu Jiang, and Zhen Yue. 2013. Supporting ex- [19] Katrien Verbert, Denis Parra, Peter Brusilovsky, and Erik Duval. 2013. Visualizing
ploratory people search: a study of factor transparency and user control. In recommendations to support exploration, transparency and controllability. In
Proceedings of the 22nd ACM international conference on Information & Knowledge Proceedings of the 2013 international conference on Intelligent user interfaces. ACM,
Management. ACM, 449–458. 351–362.