Recommender systems are currently an ubiquitous presence on the web, helping us nd relevant items in the ever-growing plethora of information available. However, there is not a one-size ts-all for recommender systems, and exibility and control are crucial for enabling the possibility of adapting the recommender system to dierent user preferences. In this paper, we present the results of a study designed to assess user interaction with IntersectionExplorer (IEx), a multi-perspective tool for exploring conference paper recommendations. e study was conducted at the Digital Humanities 2016 Conference, an event with a rather large, heterogeneous, and not technology-oriented audience. e results obtained indicate that the IEx multi-perspective approach lends enough exibility to accommodate dierent user preferences. When contrasting these results with a previous study conducted at a conference with a highly technological audience, it becomes apparent that the exibility of IEx is key to empower users with dierent proles to customize their approach to nding relevant recommendations.
•Information systems →Information systems applications;
Recommender systems are nowadays a common xture in many
environments like the web, where they play a pivotal role in helping
us nd our way through the ever more dense information jungle
[
In addition to the “black box” problem, other factors have an
impact in how recommender systems perform with users (e.g., the
“cold start” issue), and research indicates that the nature of the
system itself and that of its users may also condition
recommendation acceptance. Indeed, as Guy el al. [
IEx is a tool for exploring conference papers that proposes a dierent way of interacting with recommendations - through the exploration of multiple, intertwining perspectives of relevance. In this work we dene “perspective of relevance” as an umbrella term encompassing the source and nature of recommendations. We identify three types of perspective, each one occupying its own place in IEx’s user interface (UI): (1) the perspective of personalized relevance; (2) the perspective of social relevance and (3) the perspective of content relevance. e rst of these perspectives is composed by sets of papers that have been suggested by dierent recommendation engines: since recommender systems leverage previous knowledge about the user to provide suggestions that would likely t his/her interests and goals, their suggestions are relevant mainly because they are personalized. e perspective of social relevance is composed by sets of papers that have been marked as relevant by other users of the system: if another user is perceived as like-minded, a collection of his/her items of interest may likely be considered as a set worth exploring. Finally, the perspective of content relevance is composed of sets of papers tagged by the community with the same keywords applied by the user. Since these keywords are usually drawn or derived from the contents or the experience of people with an item, they provide insightful glances about the contents of the tagged items. A key feature of IEx is the seamless way it allows users to combine sets from these three perspectives, making no distinction between them in terms of interaction or UI representation.
is approach lends IEx enough exibility to allow its users to
explore and combine recommendations based on human-generated
data and produced by automatic agents in a seamless manner, all
carrying the same potential weight and relevance. In order to
understand if users do indeed leverage IEx’s adaptability potential,
we conducted a user study at the 2016 edition of the Digital
Humanities (DH2016), a conference with a heterogeneous and not
technology-oriented audience. We discuss the results of this study
in this work and contrast our ndings with those of a previous
study [
Social recommendation based on people and tags has been
researched extensively (e.g., [
We can also nd research focused on hybrid recommenders, i.e.,
systems involving dierent recommendation techniques in synergy.
An interesting reection on this approach was made by Guy et
al. [
Regarding visualization-based approaches, TasteWeights is a
system designed to allow its users to control the inuence of friends’
and peers’ proles and behaviors on the recommendation processes
and, like IEx, it features a UI for presenting and interacting with
recommendations. e recommendation process is adapted at
runtime by user-entered preference and relevance feedback. is idea
can be traced back to the work of Schafer et al. [
As previously stated, IEx is a platform that allows for multi-perspective
exploration of recommendations. An overview of its user interface
is shown in Figure 1. IEx uses a simplied version of UpSet [
e Set Selection View allows the user to select sets of recommendations from three dierent perspectives: the Perspective of Personalized Relevance, the Perspective of Social Relevance and the Perspective of Content Relevance (Figure 1, labels a, b and c, respectively). e Perspective of Personalized Relevance lists the papers suggested by dierent recommendation engines, the Perspective of Social Relevance is composed of papers that have been bookmarked by other users of the system and, nally, the Perspective of Content Relevance shows sets of papers labelled by the community with a specic tag. While the rst perspective is clearly associated to automatic processes, the last two are based on human-generated data meaning that, in a sense, IEx’s users play the role of ”human recommenders”.
In the Set Exploration View the user can explore all possible combinations between the sets selected in the Set Selection View. Sets of papers are represented as columns (the current user is highlighted in blue) and set combinations are depicted as rows (e.g., Figure 1, label d), where intersecting sets are represented as lled circles. e horizontal bar next to circle rows represents the relative (the row itself) and the absolute (the number by the row) amount of papers in the selected intersection. For example, the row selected in Figure 1 (the fourth row) indicates that there are 5 papers in common between the suggestions of the bookmark-based agent and papers bookmarked by the user named “User 1”.
e Intersection Exploration View allows the user to explore the details and bookmark the papers contained in the selected intersection (Figure 1, label e). In the example of Figure 1, the user is exploring the 5 papers contained in the intersection represented by the fourth row of the Set Exploration View. 4 4.1
To provide IEx with data, we have deployed it on top of Conference
Navigator 3 (CN3) [
e CN3 also supplies IEx with data regarding other user’s bookmarks and community-tagged papers, which respectively compose the laer’s perspectives of social- and content-relevance. To address the well-known cold start problem, we requested participants to bookmark and tag a minimum of ve papers from the conference proceedings its CN3 proceedings page.
In order to understand how exible IEx’s multi-perspective approach is, we conducted a user study at the DH2016 Conference, an event with a rather large, heterogeneous, and not technologyoriented audience, mainly composed of researchers from the areas of social sciences and humanities. We recruited 37 participants through direct invitation out of the DH2016 aendees, 11 female and averaging 38 years (SD: 10). For background, our previous EC-TEL2015 study had 20 participants, 3 female, averaging 32.9 years old (SD: 6.32).
Before starting the tests, all participants received the same presentation that introduced IEx, explained its functionality and covered its essential concepts. All participants were asked to perform the same task: to freely explore the DH2016’s papers through IEx, and bookmark ve relevant papers.
We collected data about participants’ actions, like paper bookmarking actions and visualizations. To provide some denitions, we consider that a set of papers is “explored” when the user clicks on its respective row (Figure 1, d); that papers are “visualized” when they are listed in the Intersection Exploration View (Figure 1, e); and that a paper is “bookmarked” when the user clicks on the “Bookmark this paper” link that is adjacent to each visualized paper. In order to simplify our analysis, we dene the metric precision as the fraction of papers that were visualized and bookmarked, across all users (e.g., if the user was to bookmark one paper out of ve he/she visualizes, that would yield a precision of 1/5, or 0.2). 4.2
In Table 1, we can see the results of participant interactions with agents, namely single agents (exploring the suggestions of a single agent), multiple agents (exploring the overlapping suggestions of more than one agent) and augmented agents (exploring the overlaps between the suggestion of agents and sets of papers from other perspectives). It is noticeable that in our DH2016 study single and augmented agents were explored the most, with comparable precision scores, while participants of our rst study mainly explored the suggestions of multiple agents. 57 (59) 96 (25) 66 (45) 87 (39) 48 (25) 105 (59) 5 DISCUSSION e results of our studies allow us to conclude positively about the exibility of IEx’s approach to accommodate dierent user preferences. Indeed, aer the denition of “precision” that we make in this work (see section 4.1), our results indicate that the perspective of content-relevance (composed by sets of tagged papers) is the one accounting for the higher precision (see Table 2). is may be explained in light of the nature of this perspective, since well-applied tags provide accurate insights into the contents of the labeled items, and conference papers are interesting to readers mainly because of their content. Also, we found that there is a tendentially higher precision when sets of tagged papers are involved in explorations (see Table 3). Since this involvement implies that all explored papers are also community-tagged papers, this nding provides support to our previous observation.
Another interesting result reports to participant interaction with automatic recommendation agents (see Table 1). It is noticeable that while participants of our rst study were mainly interested in the suggestions of multiple agents, those of our second study were not (respectively 40 vs. 4 explorations). In turn, while the precision was higher in our rst study for augmented agents, the precision was the highest in our DH2016 study for single and augmented agent explorations. ese ndings suggest that IEx use data reects the nature of its users, i.e., technology-oriented users prefer to explore the overlaps of automatic processes while less technology-oriented people were more interested in complementing the recommendations of automatic agents with sets of suggestions based on humangenerated data - or, in other words, in having a human perspective over machine-produced recommendations.
ese results can be extrapolated to conclude about the control that IEx lends to its users. Indeed, our platform seems to be exible enough to allow them to select and explore the perspectives they judge the most productive and, what is perhaps more interesting, Bruno Cardoso, Peter Brusilovsky, and Katrien Verbert to mix them freely and discover new and customized approaches that t best with their personal objectives.
Our results indicate that IEx’s multi-perspective approach is a promising way of presenting recommendations to its users, exible enough to adapt and allow them to follow their own path to trustworthy recommendations. For the future, it would be interesting to further challenge IEx in domains of application other than the recommendation of conference papers, and also with dierent audiences. While the UpSet is an eective way of presenting intersections between sets, its focus on information entails domain agnosticism. erefore, dierent, multi-perspective visualizations may also be considered to bring IEx closer to its users.
e research has been partially nanced by the KU Leuven Research Council (grant agreement no. C24/16/017).