Recent e orts in recommender systems research focus increasingly on human factors that a ect acceptance of recommendations, such as user satisfaction, trust, transparency, and user control. In this paper, we present a scalable visualisation to interleave the output of several recommender engines with human-generated data, such as user bookmarks and tags. Such a visualisation enables users to explore which recommendations have been bookmarked by like-minded members of the community or marked with a speci c relevant tag. Results of a preliminary user study (N =20) indicate that e ectiveness and probability of item selection increase when users can explore relations between multiple recommendations and human feedback. In addition, perceived effectiveness and actual e ectiveness of the recommendations as well as user trust into the recommendations are higher than a traditional list representation of recommendations.
Human-centered computing ! Information visualisation; Empirical studies in visualisation; User interface design; Interactive visualisation; recommender systems; set visualisation; scalability; user study
When recommendations fail, a user's trust in a recommender system often decreases, particularly when the sys
Copyright remains with the authors and/or original copyright holders, 2016.
tem acts as a \black box" [
Our work has been motivated by the presence of
multiple relevance prospects in modern social tagging systems.
Items bookmarked by a speci c user o er a social relevance
prospect : if this user is known or appears to be like-minded,
a collection of her bookmarks is perceived as an interesting
set that is worth to explore. Similarly, items marked by a
speci c tag o er a content relevance prospect. In a social
tagging system extended with a personalised recommender
engine [
Existing personalised social systems do not allow their
users to explore and combine these di erent relevance prospects.
Only one prospect can be explored at any given time: a list
of items suggested by a recommender engine, a list of items
bookmarked by a user, or a list of items marked with a
speci c tag. In our work, we focus on the use of
visualisation techniques to support exploration of multiple
relevance prospects, such as relationships between di erent
recommendation methods, socially connected users, and tags,
as a basis to increase acceptance of recommendations. In
earlier work, we investigated how users explore these
recommendations using a cluster map visualisation [
In this paper, we present the use of a scalable visualisation
that combines personalised recommendations with two
additional prospects: (1) bookmarks of other users (a social
relevance prospect), and (2) tags (content relevance prospect).
Personalised recommendations are generated with four
different recommendation techniques and embodied as agents
to put them on the same ground as users (i.e.,
recommendations made by agents are treated in the same way as
bookmarks left by users). We use the UpSet visualisation [
RQ1 Under which condition may a scalable visualisation increase user acceptance of recommended items? RQ2 Does a scalable set visualisation increase perceived e ectiveness of recommendations? RQ3 Does a scalable set visualisation increase user trust in recommendations? RQ4 Does a scalable set visualisation improve user satisfaction with a recommender system? The contribution of this research is threefold: 1. First, we present a novel interface that integrates a simpli ed version of the UpSet visualisation, allowing the user to exibly combine multiple prospects to explore recommended items. 2. Second, we present a preliminary user study that assesses the e ect of combining multiple relevance prospects on the decision-making process. We nd that users explore combinations of recommendations with users and tags more frequently than recommendations only based on agents. Further, this combination is found to provide more relevant results, leading to an increase in user acceptance. 3. Third, we nd indications of an increase in user trust, user satisfaction, and both perceived and actual e ectiveness of recommendations compared to a baseline system. This shows the positive e ects of combining multiple prospects on user experience.
This paper is organized as follows: rst, we present related work in the area of interactive recommender systems. We then introduce the design of IntersectionExplorer, an interactive visualisation that allows users to explore recommendations by combining multiple relevance prospects in a scalable way. We assess its impact on the decision-making process and nish with a discussion of the results.
In a recent study, we analyzed 24 interactive recommender
systems that use a visualisation technique to support user
interaction [
In addition, TasteWeights [
Relevance-based or set-based visualisation attempts to
spatially organize recommendation results. This type of
visualisation has its roots in the eld of information retrieval and
was used for the display of search results. For example: for a
query that uses three terms, this type of visualisation would
create seven set areas. Three sets will show the results
separately for each term. Another set of three will show results
for any combination of two of these terms. Finally, one set
will show results that are relevant to all three terms together.
The classic example of such a set-based relevance
visualisation is InfoCrystal [
In the context of similar work, the novelty of the approach
suggested in this paper is twofold: rst, we use a set-based
relevance approach that is not limited to keywords or tags,
but which combines these with other relevance-bearing
entities (users and recommendation agents). The major di
erence and innovation of our work is that we allow end-users
to combine multiple relevance prospects to increase richness
and relevance of recommendations. Second, we present and
evaluate the use of a novel scalable visualisation technique
(UpSet [
IntersectionExplorer (IE) is an interactive visualisation tool that enables users to combine suggestions of recommender agents with user bookmarks and tags in order to nd relevant items. We describe the visualisation and interaction design of the system, followed by its implementation. 3.1
We have adapted the UpSet [
One of the biggest advantages of a visual matrix is scalability. Whereas a Venn diagram can only display the intersections of a limited number of sets, the UpSet technique can present many sets in parallel, as only a single column has to be added to add another set to the visualisation. This greatly reduces space requirements while increasing the information density. The visual encoding of IE is identical for any number and constellation of sets. In practice, users may wish to rst familiarise themselves with the display of a small number of sets, but due to the consistent and spacee cient design, they can seamlessly increase the set numbers without altering the view. 3.2
An overview of the full IE interface is shown in Figure 2. The interface is separated into three connected parts. In the left part, the user can select di erent entities: agents, users and tags. If an agent is selected, the set of items suggested by this agent is added to the matrix visualisation in the canvas area. If a user is added, the set of bookmarks of this user is added. Similarly, if a tag is added, the set of papers marked with this tag is added to the view.
The canvas area represents user-selected sets as columns in a matrix view, allowing the user to explore overlaps between these sets. Each row represents relations between the di erent columns as explained in the previous section.
The user can explore the details of data items related to a certain row by clicking on the row. For example, after clicking the rst row in Figure 2, the right part shows the title and authors of two papers that are bookmarked by \P Brusilovsky" and also suggested by three di erent agents.
The user can explore the items related to a speci c set by clicking on the column header: all containing items of this set are then presented in the panel on the right. Meanwhile, the rows related to this set are also gathered at the top to facilitate exploration of relations with other sets.
At the top of the set view, the user can also sort the rows (set intersections) by number of items or number of related sets in ascending or descending order. The example of Figure 2 sorts the rows by the number of related sets in descending order. The rst row represents items in the intersection of four sets. The second row represents items in the intersection of three sets and the next ve rows represent items in the intersection of two sets. The other rows represent items related to a single set only. 3.3
We have implemented IE on top of data from Conference
Navigator 3 (CN3). CN3 is a social personalised system that
supports attendees at academic conferences [
We use the list of conference talks as data items in IE.
CN3 o ers four di erent recommendation services that rely
on di erent recommendation engines. The tag-based
recommender engine matches user tags (tags provided by the
user) with item tags (tags assigned to di erent talks by the
community of users) using the Okapi BM25 algorithm [
The suggestions of these four recommender engines are represented as separate agents in IE. Users can explore which items are suggested by a single agent, for instance the tagbased recommender, but they can also explore which items are recommended by multiple agents to lter out the potentially more relevant recommendations. In addition, users can explore relations between agent suggestions and bookmarks of real users. As shown in Figure 2, the third row represents items suggested by the tag-based agent that have also been bookmarked by \P Brusilovsky", but that are not suggested by the two other agents and that have not been bookmarked by the active user (\K Verbert"). In this paper, we evaluate whether enabling users to explore relations between recommendations of di erent techniques, real users, and tags increases the acceptance of recommendations.
The set visualisation shows the relations of the selected sets as described in section 3.1. The column of the current user is displayed in blue while the other columns are represented in grey. As presented in Figure 2, the bar chart below the column headers of users overlays a blue bar that encodes the number of common bookmarks with the current user. The similarity between users is also represented next to the user name in the panel on the left: \P Brusilovsky (9/31)" indicates that the user \P Brusilovsky" has 31 bookmarks in total. Nine out of these 31 talks are also bookmarked by the active user (\K Verbert").
For the user study presented in this paper, we used the data from the EC-TEL conferences of 2014 and 2015. ECTEL is a large conference on technology enhanced learning. We retrieved user bookmarks and tags of these conferences, and had access to the di erent recommender services for both the 2014 and 2015 edition of the conference. Attendees of the EC-TEL conference participated in the user study that is presented in the next section.
To investigate to what extent the set visualisation may support users in nding relevant items, we conducted a withinsubjects study with 20 users (mean age: 32.9 years; SD: 6.32; female: 3) in two conditions, both of which had to be completed by all participants.
In the rst condition (baseline), users were tasked to explore recommendations presented to them using the CN3 \my recommendations" page with four ranked lists. In the second condition, users explored recommendations using IntersectionExplorer (IE). To avoid a learning e ect, each condition used a separate data set from which to generate recommendations. The baseline condition (CN3) used the ECTEL 2014 proceedings (172 items), the IE condition used the EC-TEL 2015 proceedings (112 items).
To prepare for the study, users bookmarked and tagged ve items in each of the proceedings. In addition, users' publication history and academic social bookmark systems (CiteULike and Bibsonomy) were read. From the combined data, recommendations were generated in both conditions using the four di erent techniques described in Section 3. These were then presented as four individual agents: a tagbased agent, a bookmark-based agent, an external bookmark agent and a biography agent.
To explore the impact of the IE visualisation on the users'
acceptance of items, users were tasked to explore the
recommendations of the four agents freely and to bookmark
ve items. During this period we recorded the time and
amount of steps taken to create a bookmark. In
particular, we recorded the following actions: selection/deselection
of agents, users and tags, sorting, hovering over a result
row (if mouse position was held for more than two seconds),
clicking onto a paper's title, and clicking the bookmark
button. Further, we collected data using a think-aloud
protocol, synchronizing screen recording and microphone input.
Finally, users completed a questionnaire using a ve-point
Likert scale. The questions were based on ResQue [
Before exploring the recommendations using IE, users were shown a three-minute video to explain the system's operation. In the IE view, users saw the intersections with the agents' recommendations. In the CN3 (baseline) view, users saw the full results of the bookmark agent and could navigate to the recommendations generated by the three other agents, as presented in Figure 3. The study was counterbalanced by mode of exploration (CN3/IntersectionExplorer). Five users completed the study with a researcher present in the same room, whereas 15 users completed the study via an on-line video call. To establish users' backgroundknowledge, we asked each participant a set of questions using a ve-point Likert scale after the study. Mean results were as follows:
Users were familiar with technology-enhanced learning (mean: 4; SD: 1.1).
Users were familiar with recommender systems (mean: 4; SD: 0.95).
Users were familiar with visualisation techniques (mean: 4.05; SD: 0.86).
Users occasionally followed the advice of recommender systems (mean: 4.25; SD: 0.77).
Eight participants had never heard of CN3 before. Twelve had heard of it, but had no particular familiarity with the system (mean: 3.25; SD: 1.13).
One user had no publications, four had two to four publications, fteen had ve publications or more. Within the last group, 93.3% had published on an EC-TEL conference in the past.
4.1.1
The main focus of this study was to investigate under which condition the visualisation may increase user acceptance of recommended items. To answer our question, we need to analyse the in-depth behaviour of users exploring the recommendations using various combinations of recommender agents and the bookmarks and tags of other users.
In order to be able to determine the impact of visualising relations between agents, users, and tags, we de ned two measures: e ectiveness and yield.
E ectiveness measured how frequently the exploration of a speci c set providing a number of intersections (henceforth called `size') eventually led to the user bookmarking another paper (from the recommended set of papers). By the exploration of a set we mean clicking on a row of intersections in the visualisation (Figure 1, Figure 2) to show the items belonging to the intersection of the selected sets.
E ectiveness was calculated as the number of cases where the exploration of an intersection of a speci c type and size resulted in a new user bookmark, divided by the number of times this intersection type and size was explored. Intersection types could be a single agent, a combination of agents, or a combination of agents with another entity (user or tag). The size represented the number of sets in the intersection. For instance, users explored suggestions of a single agent 26 times in total (one agent, Figure 4, rst row). Exploration of these sets resulted in the creation of ve bookmarks. Thus, the single agent's e ectiveness is 5/26 = 19%.
Yield measured the fraction of items of an explored set that were actually bookmarked by all users in total. For instance, if the results of the intersection with one agent listed a total of 93 items for all users combined, but only ve bookmarks were created from this type and size of intersection across the whole study, its overall yield was 5/93=0.05 (Figure 5, rst row).
Figure 4 and Figure 5 reveal an interesting e ect: sets which included the recommendations of agents and other entities, such as other users' bookmarks and tags, appeared to have a higher yield and e ectiveness than sets based on agent recommendation alone, even if the number of intersections were the same. To further explore this aspect, we divided the results for e ectiveness and yield into two groups: those obtained for interaction with one to four agents, and those obtained from interaction with the recommendations of di erent numbers of agents and another entity (user or tag). A Friedman test indicated a signi cant e ect of recommendation source on e ectiveness, c2(1) = 4, p = :046 revealing that users who explored the recommendations of agents combined with another entity in the recommendation matrix of IE (median: .43), tended to nd more than twice as many relevant items as when only using the agents for the recommendation (median: 0.21) (Figure 4). These results correspond to our ndings that the richer the set (the more \perspectives" contribute the recommendation), the higher the yield (Figure 6). In general, Figure 7 shows that the larger the amount of intersections with a speci c type, the higher the yield. Pearson's correlation showed a positive correlation between the number of intersections and yield (r = .839, n = 6, p = .037).
Overall, these results suggests that enriching automated recommendations based on tags, previous bookmarks, publication history and academic social bookmarks with socially collected relevance evidence, such as the bookmarks made by other users of the same conference or a tag, greatly increases the relevance of recommendations, resulting in a higher acceptance rate.
Regarding the overall operability of IE, an ANOVA of task completion time showed an e ect of task number F (4, 44) = 20.5, p < .001 on interaction time. However, a post-hoc a Bonferroni-Holm correction, di erences were not statistically signi cant. This suggests that while IE may have a higher learning curve than CN3, no statistically signi cant di erences exist in terms of e ciency of operation after acquaintance with the system (Figure 8). Bonferroni-Holm-corrected Wilcoxon signed-rank test indicated that di erences were not statistically signi cant.
A Greenhouse-Geisser corrected ANOVA of the amount of steps needed to complete the bookmarking tasks showed an e ect of condition, F (1, 11) = 7.86, p = .017, and an e ect of task order, F (2.09, 23) = 168.82, p = .002. A Wilcoxon signed-rank test showed a trend for task one taking more steps when using IE (median: 11) than when using CN3 (median: 4), Z = 2.5, p = .012, but after applying
In addition, a trend was observed that users experienced IE to be more fun than CN3 (Z = 2.28, p = .023) and to provide a higher choice satisfaction (Z = 2.1, p = .039). However, after applying a Bonferroni-Holm correction, differences were not statistically signi cant.
Similarly, the results for the novelty of items (median: 4), e ort to use the systems (median: 2), usefulness (median: 4), and ease of use (median: 4) were the same for both systems. Users tended to perceive the creation of bookmarks as more di cult in IE (median: 3) than in CN3 (median: 2), but tended to read the bookmarked papers afterwards more frequently when using IE (median: 4) that when using CN3 (median: 3).
As for the IE-speci c aspects shown in Figure 11, users
perceived the visualisation to be adequate (median: 4) and
the amount of information provided by the system to be
su cient to make a bookmark decision (median: 4). Users
tended to be undecided regarding the interaction adequacy
of IE (median: 3.5, see [
RQ1 Under which condition may a scalable visualisation increase user acceptance of recommended items?
Our research showed that user acceptance of recommended items increased with the amount of sources used. However, the most important nding is that the addition of humangenerated data { such as bookmarks of other users or tags { to the agent-generated recommendations resulted in a significant increase of e ectiveness and yield. Our data suggests that providing users with insight into relations of recommendations with bookmarks and tags of community members increases user acceptance. We thus recommend to combine automated sources and personal sources whenever possible.
RQ2 Does a scalable set visualisation increase perceived e ectiveness of recommendations?
Perceived e ectiveness (expressed in the questionnaire) and actual e ectiveness (how frequently users bookmarked a recommended paper) were increased by this type of visualisation.
RQ3 Does a scalable set visualisation increase user trust in recommendations?
The evaluation of the subjective data showed that user trust into the recommended items was increased with setbased visualisation of recommendation sources.
RQ4 Does a scalable set visualisation improve user satisfaction with a recommender system?
Overall, user satisfaction was higher when using the visualisation, suggesting this to be a key feature of the approach. 4.2
4.2.1
The analysis of task completion time and amount of steps needed to complete the bookmarking tasks has shown that users require more time and interactions to set their rst bookmark in IE, but that after this `training phase', the operational e ciency between IE and CN3 does not di er. This corresponds to the observations made during the analysis of the think-aloud study, where it was found that some users initially struggled to understand the meaning of the di erent circle types or what a `set' was.
However, the analysis of the subjective data has shown that users perceived IE to be more e ective and its recommendations more trustworthy than those given by CN3. Especially the last point may be the result of removing the frequently lamented \black box" problem of recommenders by simply visualising how and why certain items are selected. In addition, users perceived items resulting from their use of IE to be of higher quality and found the overall experience more satisfying. This positive user experience may compensate for the initial conceptual problems encountered in the rst exploration of the application and suggests that IE may be a helpful addition to the conference explorer service. 4.2.2
In our previous work we presented the idea of
combining recommendations embodied as agents with bookmarks
of users and tags as a basis to increase e ectiveness of
recommendations [
IE employs the novel UpSet visualisation technique that was presented at IEEE VIS in 2014. We simpli ed the interface and deployed it on top of data collected by Conference Navigator. The approach addresses the previous limitations regarding ease of use and scalability: in this study users did explore many intersections, enabling us to investigate the e ect of the approach on acceptance of recommendations.
One limitation is the low number of participants. Further, the study was conducted with researchers from the eld of technology enhanced learning with a high degree of visualisation expertise (mean: 4.05, SD: 0.86). Such users may be biased due to their graph literacy. In addition, our data was limited to that provided by the EC-TEL conferences.
We presented a study that used the UpSet visualisation technique to combine agent-based recommendations with human-generated recommendations in the form of bookmarks and tags. Despite the initial learning curve (when compared to the baseline system CN3), we found that this combination resulted in a higher degree of item exploration and acceptance of recommendations, than when using agent-only results. This way, user trust, usefulness, quality, and e ectiveness were increased. We could thereby demonstrate the positive e ects of the combination of multiple prospects on user experience and relevance of recommendations.
Future work will explore the applicability of our ndings to a more diverse dataset and audience, as well as di erent types of visualisations. We have currently deployed IntersectionExplorer for attendees of the ACM IUI 2016 conference and will evaluate whether the visualisation can be used in an open setting, without the presence of a researcher. In addition, we plan to deploy the visualisation on top of data from large conferences, including the Digital Humanities conference series. Follow-up studies will assess the added value of our visualisation on top of larger data collections and with a less technical audience. With these studies, we intent to reach a wider range of users to further evaluate the e ect of the approach on the e ectiveness of recommendations.
We thank all participants for their participation and useful comments. Part of this work has been supported by the Research Foundation Flanders (FWO), grant agreement no. G0C9515N, and the KU Leuven Research Council, grant agreement no. STG/14/019. The author Denis Parra was supported by CONICYT, project FONDECYT 11150783.