Whenever we gather or organize knowledge, the task of searching inevitably takes precedence. As exploration unfolds, it becomes cumbersome to reorganize resources along new interests, as any new search brings new results. Despite huge advances in retrieval and recommender systems from the algorithmic point of view, many real-world interfaces have remained largely unchanged: results appear in an infinite list ordered by relevance with respect to the current query. We introduce uRank, a user-driven visual tool for exploration and discovery of textual document recommendations. It includes a view summarizing the content of the recommendation set, combined with interactive methods for understanding, refining and reorganizing documents on-the-fly as information needs evolve. We provide a formal experiment showing that uRank users can browse the document collection and efficiently gather items relevant to particular topics of interest with significantly lower cognitive load compared to traditional list-based representations.
Theory recommending interface, exploratory search, visual analytics, sensemaking
With the advent of electronic archival, seeking for information occupies a large portion of our daily productive time. Thus, the skill to find and organize the right information has become paramount. Exploratory search is part of a discovery process in which the user often becomes familiar with new terminology in order to filter out irrelevant content and spot potentially interesting items. For example, after inspecting a few documents related to robots, sub-topics like human-robot interaction or virtual environments could attract the user’s attention. Exploration requires careful inspection of at least a few titles and abstracts, when not full documents, before Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee.
IntRS ’15 Copyright 20XX ACM X-XXXXX-XX-X/XX/XX ...$15.00. becoming familiar with the underlying topic. Advanced search engines and recommender systems (RS) have grown as the preferred solution for contextualized search by narrowing down the number of entries that need to be explored at a time.
Traditional information retrieval (IR) systems strongly depend
on precise user-generated queries that should be iteratively
reformulated in order to express evolving information needs. However,
formulating queries has proven to be more complicated for humans
than plainly recognizing information visually [
Sometimes RS can be more limited than IR systems if they do
not tackle trust factors that may hinder user engagement in
exploration. As Swearingen et al. [
In this work we focus mainly on transparency and controllability aspects and, to some extent, on predictability as well. uRank 1 is and interactive user-driven tool that supports exploration of textual document recommendations through:
i) an automatically generated overview of the document collection depicted as augmented keyword tags,
ii) a drag-and-drop-based mechanism for refining search interests, and
iii) a transparent stacked-bar representation to convey document ranking and scores, plus query term contribution. A user study revealed that uRank incurs in lower workload compared to a traditional list representation. 1http://eexcessvideos.know-center.tugraz.at/ urank-demo.mp4 2.1
Modern search interfaces assist user exploration in a variety of
ways. For example, query expansion techniques like Insyder’s
Visual Query [
Faceted search interfaces allow for organizing or filtering items
throughout orthogonal categories. Despite being particularly useful
for inspecting enriched multimedia catalogs [
Rankings conveying document relevance have been discouraged
as opaque an under-informative [
Although ranked list are not a novelty, our approach attempts
to leverage the advantages provided by lists; i.e. user familiarity,
and augment them with stacked-bar charts to convey document
relevance and query term contribution in a transparent manner.
Insyder’s bar graph [
LineUp [
In recent years, considerable efforts have been invested into
leveraging the power of social RS through visual interfaces [
Federated RS Directory Listing Knowledge Management System
User Collection Interactive process
Automatic process
among recommendations, users and tags in a transparent manner,
while SetFusion emphasizes controllability over a hybrid RS.
Rankings are not transparent though, as there is no explanation as to how
they were obtained. Kangasraasio et al. [
With uRank we intend to enhance predictability through document hint previews (section 3.1.1), allow the user to control the ranking by choosing keywords as parameters, and support understanding by means of a transparent graphic representation for scores (section 3.2). 3.
uRank is a visual analytics approach that combines lightweight text analytics and an augmented ranked list to assist in exploratory search of textual recommendations. The Web-based implementation is fed with textual document surrogates by a federated RS (FRS) connected to several sources. A keyword extraction module analyzes all titles and abstracts and outputs a set of representative terms for the whole collection and for each document. The UI allows users to explore the collection content and refine information needs in terms of topic keywords. As the user selects terms of interest, the ranking is updated, bringing related documents closer to the top and pushing down the less relevant ones. Figure 1 outlines the workflow between automatic and interactive components.
uRank’s layout is arranged in a multiview fashion that displays
different perspectives of the document recommendations.
Following Baldonados’s guidelines [
The GUI presents the data in juxtaposed views that add to a
semantic Overview+Detail scheme [
These views can be modified through interaction with the Ranking Controls (Figure 2.F) and the Query Box (Figure 2.B). The former provides controls to reset the ranking or switch ranking modes between overall and maximum score. The latter is the container where the user drops keywords tags to trigger changes in the ranking visualization. 3.1
uRank automatically extracts keywords from the recommended documents with a twofold purpose: i) give an overview of the collection, and ii) provide manipulable elements that serve as input for an on-the-fly ranking mechanism (see section 3.2).
Summarizing the collection in a few representative terms allows the user to scan the recommendations and grasp the general topic at a glance, before even reading any of them. This is particularly important in the context of collections brought by RS, where the user is normally not directly generating the queries that feed the search engine. 3.1.1
The Tag Box provides a summary of the recommended texts as
a whole by presenting extracted keywords as tags. Keywords tags
are arranged in a bag-of-words fashion, encoding relative
frequencies through position and intensity (Figure 2.A). The descending
ordering conveys document frequency (DF) while five levels of
blue shading help the user identify groups of keywords in the same
frequency range. Redundant coding is intentional and aims at
maximizing distinctiveness among items in the keyword set [
At first glance, the Tag Box gives an outline of the covered topic in terms of keywords and their relative frequencies. Nevertheless, a bag-of-words representation per se does not supply further details about how a keyword relates to other keywords or documents. We bridge this gap by augmenting tags with two compact visual hints – visible on mouse over – that reveal additional information: i) cooccurence respect to other keywords, and i) a preview of the effect of selecting the keyword.
The document hint (Figure 3) consists in a pie chart that conveys the proportion of documents in which the keyword appears. A tooltip indicates the exact quantity and percentage. Upon clicking on the document hint, unrelated documents are dimmed so that documents containing the keyword remain in focus Even unranked documents become discretely visible at the bottom of the Document List. This hint provides certain predictability regarding the effect of selecting a keyword, in terms of which ranked items will change their scores and which documents will be added to the ranking.
The co-occurrence hint (Figure 2.A) shows the number of frequently co-occurring keywords in a red circle. Moving the mouse pointer over it brings co-occurring terms to focus by dimming the others in the background. Clicking on the visual hint locks the view so that the user can hover over co-occurring keywords, which shows a tooltip stating the amount of co-occurrences between the hovered and the selected keyword. This hint supports the user in finding possible key phrases and sub-topics within the collection. 3.1.2
The aforementioned interactive features are supported by a combination of well-known text-mining techniques that extend the recommended documents with document vectors and provide meaningful terms to populate the Tag Box.
Document vectors ideally include only content-bearing terms like
nouns and frequent adjectives – appearing in at least 50% of the
collection –, hence it is not enough to just rely on a list of stop words
to remove meaningless terms. Firstly, we perform a part-of-speech
tagging (POS tagging) [
Next, we generate a weighing scheme by computing TF-IDF (term frequency inverse document frequency) for each term in a document vector. The score is a statistical measure of how important the term is to a document in a collection. Therefore, the more frequent a term is in a document and the fewer times it appears in the corpora, the higher its score will be. Documents’ metadata are extended with these weighted document vectors.
To fill the Tag Box with representative keywords for the collection set, all document keywords are collected in a global keyword set. Global keywords are sorted by document frequency (DF), i.e. the number of documents in which they appear, regardless of the frequency within documents. To avoid overpopulating the Tag Box, only terms with DF above certain threshold (by default 5) are taken into account. Note that terms used to label keyword tags are actual words and not plain stems. Scanning a summary of stemmed words would turn unintuitive for users. Thus, we keep a record of all term variations matching each stem, in order to allow for reverse stemming and pick one representative word as follows: 1. if there is only one term for a stem, use it to label the tag, 2. if a stem has two variants, one in lower case and the other in upper case or capitalized, use it in lower case, 3. use a term that ends in ’ion’, ’ment’, ’ism’ or ’ty’, 4. use a term matching the stem, 5. use the shortest term.
To feed the document hint (Figure 3), uRank attaches a list of bearing documents to each global keyword. For the case of cooccurrence hints (Figure 2.A), keyword co-occurrences with a maximum word distance of 5 and a minimum of 4 repetitions are recorded. 3.2
In theory, recommendations returned by a RS are already ranked by relevance. However, in practice the lack of control thereof could hinder user engagement if the GUI does not provide enough ratiob nale for the recommendations and features for shaping the recommendation criteria. Hence, one of uRank’s major features is the user-driven mechanism for re-organizing documents as information needs evolve, along with its visually transparent logic. 3.2.1
The ranking-based visualization consists of a list of document titles (Figure 2.C) and stacked bar charts (Figure 2.D) depicting rank and relevance scores for documents and keywords within them. Document titles are initially listed following the order in which they were supplied by the F-RS.
Interactions with the view are the means for users to directly
or indirectly manipulate the data [
Addition: keyword tags in the Tag Box can be manually unpinned (Figure 4a), dragged with the mouse pointer and dropped into the Query Box (Figure 4b).
Weight change: tags in the Query Box contain weight sliders that can be tuned to assign a keyword a higher or lower priority in the ranking.
Deletion: tags can be removed from the Query Box and returned to their initial position in the Tag Box by clicking on the delete button.
As the document ranking is generated, the Document List is resorted in descending order by overall score and stacked bars appear in the Ranking View, horizontally aligned to each list item. Items with null score are hidden, shrinking the list size to fit only ranked items. The total width of stacked bars indicates the overall score of a document and bar fragments represent the individual contribution of keywords to the overall score. Bar colors match the color encoding for selected keywords in the Query Box, enabling the user to make an immediate association between keyword tags and bars. Missing colored bars in a stack denote the absence of certain words in the document surrogate. Additionally, each item in the Document List contains two types of numeric indicators: the first one - in a dark circle - shows the position of a document in the ranking while the adjacent colored number reveals how many positions the document has shifted, encoding upward and downward shifts in green and red, respectively. This graphic representation attempts to help the user concentrate only on useful items and ignore the rest by bringing likely relevant items to the top, pushing less relevant ones to the bottom and hiding those that seem completely irrelevant.
uRank allows for choosing between two ranking modes: overall score (selected by default) and maximum score (Figure 5). In maximum score mode, the Ranking View renders a single color-coded bar per document in order to emphasize its most influential keyword. Finally, resetting the visualization clears the Query Box and the Ranking View, relocating all selected keywords in the Tag Box and restoring the Document List to its initial state. 3.2.2
Quick content exploration in uRank depends on its ability to readily re-sort documents according to changing information needs. As the user manipulates keyword tags and builds queries from a subset of the global keyword collection, uRank computes documents scores to arrange them accordingly in a document ranking. We assume that some keywords are more important to the topic model than others and allow the user to assign weights to them.
Document scores are relevance measures for documents respect to a query. As titles and snippets are the only content available for retrieved document surrogates, these scores are computed with a term-frequency scheme. Term distribution schemes are rather adequate for long or full texts and are hence out of our scope. Boolean models have the disadvantages that they not only consider every term equally important but also produce absolute values that preclude document ranking.
The Ranking Model implements a vector space model to
compute document-query similarity using the document vectors
previously generated during keyword extraction (section 3.1.2).
Nonetheless, a single relevance measure like cosine similarity alone is not
enough to convey query-term contribution, given that the best
overall matches are not necessarily the ones in which most query terms
are found [
Given a document collection D and a set of weighted query terms T , such that 8t 2 T : 0 wt 1; the relevance score for term t in document vector d 2 D respect to query terms T is calculated as follows: s(td ) = t f id f (td ) jdj √jT j wt ; where t f id f (td ) is the tf-idf score for term t in document d and jdj is the norm for vector d.
The overall score of a document S(d) is then computed as the sum of each individual term score s(td ). The collection D is next sorted in descending order by overall score with the quicksort algorithm and ranking positions are assigned in such way that documents with equivalent overall score share the same place.
Alternatively, users can rank documents by maximum score, in which case S(d) = max(s(td )). 3.3
Once the user identifies documents that seem worth further inspecting, the next logical step is to drill down one by one to determine whether the initial assumption holds. The Document Viewer (Figure 2.D) gives access to textual content - title and snippet and available metadata for a particular document. Query terms are highlighted in the text following the same color coding for tags in the Query Box and stacked bars in the Ranking View. These simple visual cues pop out from their surroundings, enabling the user to preattentively recognize keywords in the text and perceive their general context prior to conscious reading. 3.4
We favor the use of animation to convey ranking-state transitions
rather than abrupt static changes. Animated transitions are
inherently intuitive and engaging, giving a perception of causality and
intentionality [
Depending on the type of ranking transition, the Document List shows a specific effect: – If the ranking is generated for the first time, an accordionlike upward animation shows that its nature has changed from a plain list to a ranked one. – If the ranking is updated, list items shift to their new positions at a perceptible pace. – If ranking positions remain unchanged, the list stays static as a soft top-down shadow crosses it.
Green or red shading effects are applied on the left side of list items moving up or down, respectively, disappearing after a few seconds.
Stacked bars grow from left to right revealing new overall and keyword scores.
The user can closely follow how particular documents shift positions by clicking on the watch - eye-shaped - icon. The item is brought to focus as it is surrounded with a slightly darker shadow and the title is underlined. Also, watched documents remain on top of the z-index during list animations, avoiding being overlaid by other list items.
The same principle of softening changes is applied to re-direct
user attention when a document is selected in the Ranking View.
The selected row is highlighted and the snippet appears in the
Document Viewer in a fade-in fashion. Animated transitions for
rankingstate changes and document selection help the user intuitively switch
contexts, either from the Tag Box to the Document List and
Ranking View, or from the latter to the Document Viewer. As Baldonado
[
The goal of this study was to find out how people responded when working with our tool. In the current scenario, recommendations were delivered in a sorted list with no relevance information. Since we aim at supporting exploratory search, we hypothesized that participants using uRank would be able to gather items faster and with less difficulty, compared to a typical list-based UI.
We were also interested in observing the effect of exposing users to different sizes of recommendation lists. We expected that without this relevance information, a slight growth in the number of displayed items would frustrate the user at the moment of deciding which items should be inspected in detail in the first place. For example, finding the 5 most relevant items in a list of ten appears as an easy task, whereas accomplishing the same task but searching a list of forty or sixty items would be more time consuming and entail a heavier cognitive load. 4.1
We conducted an offline evaluation where participants performed four iterations of the same task with either uRank (U) or a baseline list-based UI (L) with usual Web browser tools, e.g. Control+F keyword search. Furthermore, we introduced two variations in the number of items to which participants were exposed, namely 30 or 60 items. Therefore, the study was structured in a 2 x 2 repeated measures design with tool and #items as independent variables, each with 2 levels (tool = U/L, #items = 30/60).
The general task goal was to "find 5 relevant items" for the given topic and all participants had to perform one task for each combination of the independent variables, i.e. U-30, U-60, L-30 and L-60.
To counterbalance learning effects, we chose four different topics covering a spectrum of cultural, technical and scientific content: Women in workforce (WW), Robots (Ro), Augmented Reality (AR) and Circular economy (CE). Thus, topic was treated as a random variable within constraints. We corroborated that participants were not knowledgeable in any of the topics. All variable combinations were randomized and assigned with balanced Latin Square.
Wikipedia provides a well-defined article for each topic mentioned above. We considered them as fictional initial exploration scenarios but participants were not exposed to them. Instead, we simulated a situation in which the user has already received a list of recommendations while exploring certain Wikipedia page. Therefore, we prepared static recommendation lists of 60 and 30 items for each topic and used them as inputs for uRank throughout the different participants and tasks. To create each list, portions of texts from the original Wikipedia articles were fed to the F-RS, which preprocessed the text and created queries that were forwarded to a number of content providers. The result was a sorted merged list of items from each provider with no scoring information.
Each task comprised three sub-tasks (Q1, Q2 and Q3) that consisted in finding the 5 most relevant items for a given piece of text. In Q1 and Q2 we targeted a specific search and the supplied text was limited to two or three words. Q3 was designed as a broadsearch sub-task where we provided an entire paragraph extracted from the Wikipedia page and the users had to decide themselves which keywords described the topic better. The motivation to ask for the "most relevant" documents was to avoid careless selection.
We recorded completion time for every individual sub-task and for the overall task. To measure workload, we leveraged a 7-likert scale NASA TLX questionnaire covering six workload dimensions. 4.1.1
Twenty four (24) participants took part in the study (11 female, 13 male, between 22 and 37 years old). We recruited mainly graduate and post-graduate students from the medical and computer science domains. None of them is majoring in the topic areas selected for the study. 4.1.2
A session started with an introductory video explaining the functionality of uRank. Each participant got exactly the same instructions. Then came a short training session with a different topic (Renaissance) to let participants familiarize with uRank and the baseline the tool. At the beginning of the first task, the system showed a short text describing the topic and the task to be fulfilled. After reading the text, the participant pressed "Start" to redirect the browser to the corresponding UI. At this point, the first sub-task began and the internal timer initiated the count, without disturbing the user. The goal of the task and the reference text were shown in the upper part of the UI. Participants were able to select items by clicking on the star-shaped icon and inspect them later on a drop-down list. In a pilot study, we realized that asking for the "most" relevant items made the experiment overly long, as participants tried to carefully inspect their selections (particularly in the L condition). Then we decided to limit the duration of the three tasks to 3m, 3m and 6m respectively. The time constraint was not a hard deadline. During the study the experimenter reminded the participants when the allotted time was almost over, but did not force them to abandon. The sub-task concluded when the participant clicked on the "Finished" button. The UI alerted participants when attempting to finish without collecting 5 items, but allowed them to continue if desired. The second sub-task started immediately afterward and once the whole task was completed they had to fill the NASA TLX questionnaire. The procedure for the remaining tasks was repeated following the same steps. Finally, participants were asked about comments and preferences. 4.2
Workload: A two-way repeated measures ANOVA with tool and #items as independent variables revealed a significant effect of tool on perceived workload F(1,23)=35.254, p < :01;e = :18. Bonferroni post-hoc tests showed significantly lower workload when using uRank (p < :001). We also assessed the effect for each workload dimension. Again, ANOVA showed a significant effect of tool in all of them, as shown in Table 1. (#items) did not have a major effect in any case.
Completion Time: We analyzed the task overall completion time, as well as completion times for each sub-task. A two-way repeated measures ANOVA revealed a significant effect of tool on overall completion time F(1,23)=4.94, p < :05;e = :02. This effect disappeared in a Bonferroni post-hoc comparison. For Q1 and Q2 ANOVA reported no significant effect, but it showed a significant effect of tool on completion time for Q3, F(1,23)=6.2,
Performance: Relevance is a rather subjective measure. Hence, instead of contrasting item selections to some ground truth, we analyzed “consensus” in item selection.
We aggregated the collections gathered under the manipulated conditions and computed cosine similarity across UI (tool), data set size (#items), topic (WW, Ro, AR, and CE) and sub-task (Q1, Q2 and Q3).
Overall, there was a high similarity between collections produced with uRank and those obtained with the list-based UI across all sub-tasks. Choices regarding relevant documents matched three out of four times (M = :73, SD = :1).
Table 2 shows that collections produced with our tool (U) for the two variations of #items (U-30 vs U-60) turned highly similar regardless of topic and sub-task (M = :8, SD = :12, with a minimum of :62). Comparisons for a typical list-based UI (L) displaying 30 and 60 items (L-30 vs L-60) denote greater diversity (M = :67, SD = :16, with a minimum of :33) in item selection.
Interestingly, similarity values tend to decrease for broad search task (Q3) (M = :66, SD = :13) respect to targeted search (Q1 and Q2) (M = :77, SD = :13). 4.3
The study results shed a light on how people interact with a tool like uRank. For each hypotheses we contrasted the results with the subjective feedback acquired after evaluation.
Workload: The results support our hypothesis that uRank incurs in lower workload during exploratory search, both in specific and broad search tasks. Participants commented feeling alleviated when they could browse the ranking and instantly discard document that did not contain any word of interest. As a remark, the majority claimed that a few tasks were too hard to solve, especially without the uRank, because sometimes the terms of interest barely appeared in the titles or were perceived as too ambiguous, e.g. "participation of women in the workforce". Also dealing with technical texts about unfamiliar topics was posed some strain. For example, two participants had to momentarily interrupt exploration to look up a word they did not understand. In spite of that, workload was significantly lower with uRank across all dimensions.
Completion Time: We expected people would be faster performing with uRank than using a browser-based keyword filter, but completion times were not significantly different. The closing interview revealed that participants who had collected five items before the due time exploited the remainder to refine their selections. In general, participants understood that they were not expected to perform perfectly but to do their best in the given time. However, we noticed that a small group that behaved in the opposite way reported feeling more pressed by time and not satisfied with their performance. The general tendency is reflected in the significant result on temporal demand: participants felt significantly less pressed to finish while performing with uRank. The lower subjective time pressure suggests that participants indeed had more time to analyze their choices with uRank.
Performance: The results suggest that our tool produces more uniform results as the number of items to which users are exposed grows. Nevertheless, the proportion of matching documents in listgenerated collections – two out of three – still conveys a moderate consensus.
The decrease in consensus for broad search task respect to targeted search could be explained by the inherent variability across participants at the moment of chosing the terms of interest for a given text larger than a couple of words. 5.
We introduced a visual tool for exploration, discovery and analysis of recommendations of textual documents. uRank aims to help the user: i) quickly overview the most important topics in a collection of documents, ii) interact with content to describe a topic in terms of keywords, and iii) on-the-fly reorganize the documents along keywords describing a topic.
This paper presented the reasoning line for the visual and interactive design and a comparative user study where we evaluated the experience of collecting relevant items to topics of interest. Participants found it significantly more relaxing to work with uRank, and most of them wanted to start actively using it in their scientific endeavors (e.g., report or paper writing). Yet, selecting the right keywords to describe a topic is not a trivial task, as it showed on the performance results of the evaluation. We will continue to explore different techniques, e.g. topic modeling, in the near future. As for the GUI, we will work further on solving scaling problems, for example when the amount of tags in the Tag Box or the length of the result list becomes unmanageable. Moreover, we will leverage the document selections collected during the evaluation as feedback to improve recommendations, closing the interactive loop with the RS as depicted in Figure 1.