EuroHCIR)2013)was)organised)with)the)specific)goal)of)better)engaging)the)IR) community,)who)have)been)underrepresented)at)previous)EuroHCIR) conferences.)Thus)we)proposed)to)have)the)workshop)at)the)ACM)SIGIR) conference)in)Dublin.)Research,)Industry,)and)Position)papers)were)invited,)and) although)very)few)industry)submissions)were)received,)we)received)a)number)of) research)and)position)papers)focusing)on)the)intersection)of)IR)and)HCI) evaluations,)several)focusing)on)adapting)the)TREC)paradigm.)Many)interesting) system)and)demonstrator)papers)were)also)accepted.)
Proceedings+of+the+! rd
Page"3"F"" Fading"Away:"Dilution"and"User"Behaviour"(Orally)Presented)"
Paul%Thomas,%Falk%Scholer,%Alistair%Moffat% Page"7"F"" Exploratory"Search"Missions"for"TREC"Topics"(Orally)Presented)"
Martin%Potthast,%Matthias%Hagen,%Michael%Völske,%Benno%Stein% Page"11"F"" Interactive"Exploration"of"Geographic"Regions"with"WebFbased"Keyword" Distributions"
Chandan%Kumar,%Dirk%Ahlers,%Wilko%Heuten,%Susanne%Boll% Page"15"F"" Inferring"Music"Selections"for"Casual"Music"Interaction"(Orally)Presented)"
Daniel%Boland,%Ross%McLachlan,%Roderick%MurrayESmith" Page"19"F"" Search"or"browse?"Casual"information"access"to"a"cultural"heritage"collection"
Robert%Villa,%Paul%Clough,%Mark%Hall,%Sophie%Rutter% Page"23"F"" Studying"Extended"Session"Histories"
Chaoyu%Ye,%Martin%Porcheron,%Max%L.%Wilson% Page"27"F" Comparative"Study"of"Search"Engine"Result"Visualisation:"Ranked"Lists"Versus" Graphs"
Casper%Petersen,%Christina%Lioma,%Jakob%Grue%Simonsen"
Page"31"F"" Evolving"Search"User"Interfaces"(Orally)Presented)"
Tatiana%Gossen,%Marcus%Nitsche,%Andreas%Nürnberger" Page"35"F"" A"Pluggable"WorkFbench"for"Creating"Interactive"IR"Interfaces"(Orally)Presented)"
Mark%M.%Hall,%Spyros%Katsaris,%Elaine%Toms" Page"39"F"" A"Proposal"for"UserFFocused"Evaluation"and"Prediction"of"Information"Seeking" Process"(Orally)Presented)"
Chirag%Shah% Page"43"F"" Directly"Evaluating"the"Cognitive"Impact"of"Search"User"Interfaces:"a"TwoF Pronged"Approach"with"fNIRS"
Horia%A.%Maior,%Matthew%Pike,%Max%L.%Wilson,%Sarah%Sharples% Page"47"F"" Dynamics"in"Search"User"Interfaces"
Marcus%Nitsche,%Florian%Uhde,%Stefan%Haun%and%Andreas%Nürnberger%
Page"51"F"" SearchPanel:"A"browser"extension"for"managing"search"activity"
Simon%Tretter,%Gene%Golovchinsky,%Pernilla%Qvarfordt"" Page"55"F"" A"System"for"PerspectiveFAware"Search"
M.%Atif%Qureshi,%Arjumand%Younus,%Colm%O’Riordan,%Gabriella%Pasi,%Nasir%Touheed" Fading Away: Dilution and User Behaviour
Paul Thomas
CSIRO ICT Centre paul.thomas@csiro.au
Falk Scholer School of Computer Science and Information Technology
RMIT University
Alistair Moffat Department of Computing and
Information Systems
The University of Melbourne falk.scholer@rmit.edu.au ammoffat@unimelb.edu.au ABSTRACT When faced with a poor set of document summaries on the first page of returned search results, a user may respond in various ways: by proceeding on to the next page of results; by entering another query; by switching to another service; or by abandoning their search. We analyse this aspect of searcher behaviour using a commercial search system, comparing a deliberately degraded system to the original one. Our results demonstrate that searchers naturally avoid selecting poor results as answers given the degraded system; however, the depth of the ranking that they view, their query reformulation rate, and the amount of time required to complete search tasks, are all remarkably unchanged.
Categories and Subject Descriptors H.3.4 [Information Storage and Retrieval]: Systems and software—performance evaluation.
General Terms Experimentation, measurement. Retrieval experiment, evaluation, system measurement.
INTRODUCTION
While carrying out a search, users have a number of tactics available to them. Intuitively, it seems likely that these tactics or behaviours will vary based on the quality of the results that are returned by the retrieval system. For example, other things being equal, a user who cannot find any relevant items on the first page of search results might be more inclined to reformulate their query (by entering another query into the search interface) than a user who has found a large number of relevant items. Possible tactics when using an apparently ineffective system include: 1. Looking further in the results list, visiting pages beyond the
first, hoping that the results improve; Presented at EuroHCIR2013. Copyright © 2013 for the individual papers by the papers’ authors. Copying permitted only for private and academic purposes. This volume is published and copyrighted by its editors. 2. Submitting another query, hoping for better results; 3. Switching to a different search engine and entering the same
query, hoping that it provides better results; 4. Trying to find the information through other techniques, for
example by browsing.
We investigate the first two possibilities, reporting on differences in user behaviour when a standard retrieval system is compared to an adjusted system in which results are diluted by inserting non-relevant answers. Our results indicate that searchers remained attentive to the task in the degraded system, and adapted their behaviour to avoid clicking on non-relevant snippets. However, all other aspects of their behaviour were remarkably consistent, including the amount of time spent on tasks; the number of query reformulations undertaken; and their perceptions of search difficulty. 2.
METHODS
We designed a user experiment to explore ways in which
behaviour changes with retrieval quality. A total of n = 34 participants,
comprising staff and students from the Australian National
University, carried out six search tasks of differing complexity, covering
the remember, analyse and understand tasks of Wu et al. [
The search results displayed to participants were sourced from
the Yahoo! API, and presented in the usual way as an ordered list
consisting of query-biased summaries, with ten results per page. No
branding from the underlying search service was shown. Without
telling our participants, we simulated search systems of two
different effectiveness levels by showing results in one of two modes:
full, where the ranking obtained from the search service was
displayed in its original form; and diluted, where the original results
were interleaved with answers from a related but incorrect query [
Dilution was operationalised by leveraging the capacity-enhancing (and obfuscatory) power of “management-speak”: the original stakeholder information need was actioned going forward by enhancing it through the win-win inclusion of a jargon competency chosen randomly from a list of outside-the-box strategies, thereby disempowering the results paradigm. For example, if the task was to “find 0 3 . 0 5 2 . 0 0 2 n . itropo .150 roP .00 1 0 5 0 . 0 0 0 . 0 the Eurovision Song Contest home page”, a user’s initial full query might be “eurovision”; whereas in the diluted system half of the results displayed might instead be derived from the query “eurovision best practice”. There were a small number of queries issued for which it was not possible to generate five such results; these 22 out of 5930 page interactions are excluded from the analysis below.
Most interactions with the search system were logged while participants carried out the six search tasks, including: submitted search queries; clicks on snippets in order to open documents for viewing; assessments of document usefulness; and the point of gaze on the screen, captured using an eye tracker. Task order was balanced across the participants and topics so as to minimise the risk of bias; similarly, whether the full or diluted approach got applied for each participant-task combination was pre-determined as part of the experimental design.
RESULTS
User behaviour, and the differences caused by the full and diluted query treatments, can be measured in a range of ways.
User click behaviour: The normalised click frequency at each rank
position in the answer pages is shown in Figure 1. In the diluted
retrieval system the “incorrect but plausible” documents were
inserted in positions 1, 3, 5, 7 and 9. The pattern of click behaviour
demonstrates that our experimental manipulation was successful:
for the full search results, the click distribution follows the expected
pattern of users clicking more frequently on items that are higher
in the ranked list [
A second check to confirm that our system dilution had an impact on search effectiveness is to consider the rates at which users saved documents that they viewed (that is, the likelihood that a document was found to be relevant after it was clicked). The mean rate is 0.733 for the full system, compared to 0.597 for the diluted system, a statistically significant difference (t-test, p < 0.05).
While Figure 1 establishes that our user study participants responded differently in terms of rank-specific click behaviour, the high-level aggregated click behaviour across all participants and search tasks was not distinctive: in total (all tasks, and all users) there were 323 clicks for the full system, and 322 for the diluted system. Unsurprisingly this difference is not statistically significant 1st results page
2nd results page full diluted (c 2 test, p = 0.97). The number of items that were determined as being useful was also similar in the two conditions: 201 for full, and 214 for diluted (c 2 test, p = 0.52). Our participants needed to read a remarkably similar number of documents, and a remarkably similar number of useful documents, to satisfy the (assigned) needs regardless of the search system.
Given this difference in click rates, it is reasonable to expect other changes in behaviour and we consider this below.
Depth of result page viewing: When presented with a search results page, the user chooses which snippets require further evaluation.
In line with commercial search engines, our experimental participants were presented with ten answers per page, with the option of accessing subsequent results pages.
Faced with a relatively poor quality results list, a plausible strategy for a user who is looking for an answer document is to look further down the results page. Table 1 shows the frequency with which results pages were viewed (that is, the user visited a results page and looked at one or more items on the screen as recorded using eye-tracking), summed across users and queries. When using the full system, participants moved on to the second page of results for 15 out of 207 issued queries (with a corresponding mean page depth of 1.07), while in the diluted system the second results page was visited for 22 out of the total of 212 queries that were issued (a mean page depth of 1.10). The difference in depth was not significant (c 2 test, p = 0.34). No participants viewed results beyond the the second page with either system.
Figures 2 and 3 provide a more detailed view of gaze behaviour, showing the deepest rank position that searchers examined while carrying out a query, and the last rank position that was viewed before finishing the query. The distributions of the lowest rank positions viewed are similar between the full and diluted systems: both show peaks at rank positions 7 (the last item above the fold) and 10 (the last item in each page of search results). The distribution of the last position viewed before finishing a query (which arises when either enough relevant items have been found, or the user types a fresh query) are also broadly similar. However, for the diluted system, rank position 1 has a larger proportion of the probability mass. A possible reason is that searchers mentally compare answers as they view items in the results list, and most users scan at least the top few items. The diluted system is likely to have a non-relevant document in position one, and so reviewing that snippet may serve as a final confirmation, before the user commits to a click on a deeper-ranked snippet from the underlying full results.
Query reformulation: A second way in which a user might respond to search systems of differing quality is to change the rate at which they stop looking through the current set of search results, and instead enter a new query.
The number of queries used by participants when carrying out their search tasks is shown in Figure 4. Overall the number was low for both systems, with a median of 1 and 2 queries (0 and 1 reformulations) for the full and diluted results, respectively. This difference was not statistically significant (Wilcoxon signed-rank test, p = 0.46).
Ability to identify relevant answers: When a retrieval system serves unhelpful answers, it might be that the ability of the searcher to identify useful answers is similarly affected. However, based on our experiments, the mean rate at which clicked items were saved as being relevant was 0.787 for the full system and 0.747 for the diluted system, showing no significant difference (t-test, p = 0.25). Thus the ability of users to identify relevant answers, once documents have been selected for viewing via their snippets, did not differ between the experimental treatments. 4 1 2 1 irse 10 e u q fro 8 e b m uN 6 4 2 ● ● ● ● ● ● ● Ful
Diluted ● ● ● ●
Time spent on tasks: While depth of viewing and query re-formulation do not show significant differences in searcher behaviour, it could still be the case that using an inferior system makes querying slower. Differences in system quality might alter the time spent by users when viewing and processing result pages. However, the average gaze duration when viewing snippets, measured as the sum of fixation durations that occurred in the screen area defined by each search result summary, was 0.586 second for full queries and 0.589 seconds for diluted queries. This difference was not statistically significant (t-test, p = 0.89).
Differences could also occur at a higher level of system interaction. The mean time that participants spent working on each search task, including viewing search result pages, viewing selected documents, and making relevance decisions, was 2.70 minutes for the full treatment, and 2.54 minutes for the diluted one. This difference was not statistically significant (t-test, p = 0.62).
Finally, we consider the interaction between time and query reformulations. When using the full system, participants entered an average of 1.50 queries per minute while completing each task.
For the diluted system, the rate was 1.52 queries per minute. The difference was not significant (t-test, p = 0.95).
Overall, these results indicate that the quality of the search system did not affect the rate at which participants were able to process information on search results pages, or how much time they spent working on tasks before feeling that they had achieved their goals.
The only significant difference between the two treatments was the click distribution, and the rate at which clicked documents were judged to be useful.
Searcher assessment of task difficulty: After carrying out each search task, experimental participants were asked to answer two questions: “How difficult was it to find useful information on this topic?”, and “How satisfied were you with the overall quality of your search experience?”. The 5-point response scale for these questions was anchored with the labels “Not at all” (assigned a value of 1) and “Extremely” (assigned a value of 5).
Searchers found the tasks relatively easy to complete: the median response rate for the search difficulty question was 2 for both the diluted and full systems; this difference was not significant (Wilcoxon test, p = 0.73). Satisfaction levels were also highly consistent between the two systems, with a median response level of 4 for both systems (Wilcoxon test, p = 0.91). Overall, there were no systematic differences in participants’ perceptions of search difficulty or the overall experience resulting from the two different treatments.
DISCUSSION AND CONCLUSIONS
It seems “obvious” that user behaviour will be influenced by the quality of results that returned by a search service. Seeing many poor results near the start of an answer list may influence the user’s decision about whether to continue viewing subsequent answer pages, to enter a new query, or to abandon the search altogether.
Previous work has supported this view. For example, in a study of
36 users completing 12 search tasks with different search systems,
Smith and Kantor [
However our study, in which 34 subjects carried out search tasks using an evenly balanced combination of full and diluted search systems, contrasts strongly with that intuition and previous findings.
Overall, searchers took around the same amount of time to complete their tasks in both experimental treatments; were able to save a similar number of documents as being relevant; exhibited consistent viewing behaviour when looking at the search results lists returned by the treatments; and did not perceive significant differences in the difficulty of carrying out tasks with both systems. The key difference in participant behaviour was their click rate at particular ranks: in essence, they successfully avoided poor answers, as demonstrated by the shift in the click probability mass, shown in Figure 1.
A possible explanation for the divergence in observed user
behaviour between the two studies may be the context in which the
searches were carried out. Participants in the Smith and Kantor
study were instructed to “find good information sources” for an
unspecified “boss”, with an incentive to find the most good and
fewest bad sources possible [
Alternatively, it may be that our diluted system, while certainly poorer in overall quality (in the sense that non-relevant answers were introduced into the ranking), was not poor enough to induce different behaviour. Smith and Kantor used results typically from the 300th position in Google’s results: even today, these are unreliable for the simplest of our topics, and in 2008 will almost certainly have produced a poor result set. Importantly, our diluted system always included a few high-ranked results.
Either way, our results raise an important question about how the effectiveness of search systems should be analysed. While some fine-grained aspects of user clicking behaviour differed between the full and diluted treatments, the majority of behaviours did not.
This outcome is in line with previous results that found little
relationship between user behaviour and system quality as measured
by common IR evaluation metrics such as MAP [
Exploratory Search Missions for TREC Topics Martin Potthast
Matthias Hagen
Michael Völske
Benno Stein Bauhaus-Universität Weimar
99421 Weimar, Germany <first name>.<last name>@uni-weimar.de ABSTRACT We report on the construction of a new query log corpus that consists of 150 exploratory search missions, each of which corresponds to one of the topics used at the TREC Web Tracks 2009–2011. Involved in the construction was a group of 12 professional writers, hired at the crowdsourcing platform oDesk, who were given the task to write essays of 5000 words length about these topics, thereby inducing genuine information needs. The writers used a ClueWeb09 search engine for their research to ensure reproducibility. Thousands of queries, clicks, and relevance judgments were recorded. This paper overviews the research that preceded our endeavors, details the corpus construction, gives quantitative and qualitative analyses of the data obtained, and provides original insights into the querying behavior of writers. With our work we contribute a missing building block in a relevant evaluation setting in order to allow for better answers to questions such as: “What is the performance of today’s search engines on exploratory search?” and “How can it be improved?” The corpus will be made publicly available.
Categories and Subject Descriptors: H.3.3 [Information Search and Retrieval]: Query formulation
INTRODUCTION
Humans frequently conduct task-based information search, i.e., they interact with search appliances in order to conduct the research deemed necessary to solve knowledge-intensive tasks. Examples include long-lasting interactions which may involve many search sessions spread out across several days. Modern web search engines, however, are optimized for the diametrically opposed task, namely to answer short-term, atomic information needs. Nevertheless, research has picked up this challenge: in recent years, a number of new solutions for exploratory search have been proposed and evaluated. However, most of them involve an overhauling of the entire search experience. We argue that exploratory search tasks are already being tackled, after all, and that this fact has not been sufficiently investigated. Reasons for this shortcoming can be found in the lack of publicly available data to be studied. Ideally, for any given task that fits the aforementioned description, one would have a large set of search interaction logs from a diversity of humans solving it. Obtaining such data, even for a single task, has not been done at scale until now. Even search companies, which have access to substantial amounts of raw query log data, face difficulties in discerning individual exploratory tasks from their logs.
In this paper, we contribute by introducing the first large corpus of long, exploratory search missions. The corpus was constructed via Presented at EuroHCIR2013. Copyright c 2013 for the individual papers by the papers’ authors. Copying permitted only for private and academic purposes. This volume is published and copyrighted by its editors.. crowdsourcing by employing writers whose task was to write long essays on given TREC topics, using a ClueWeb09 search engine for research. Hence, our corpus forms a strong connection to existing evaluation resources that are used frequently in information retrieval.
Further, it captures the way how average users perform exploratory search today, using state-of-the-art search interfaces. The new corpus is intended to serve as a point of reference for modeling users and tasks as well as for comparison with new retrieval models and interfaces. Key figures of the corpus are shown in Table 2.
After a brief review of related work, Section 2 details the corpus construction and Section 3 gives first quantitative and qualitative analyses, concluding with insights into writers’ search behavior. 1.1
Related Work
To date, the most comprehensive overview of research on
exploratory search systems is that of White and Roth [19]. More
recent contributions not covered in this body of work include the
approaches proposed by Morris et al. [
Regarding the evaluation of exploratory search systems, White and Roth [19] conclude that “traditional measures of IR performance based on retrieval accuracy may be inappropriate for the evaluation of these systems” and that “exploratory search evaluation [...] must include a mixture of naturalistic longitudinal studies” while “[...] simulations developed based on interaction logs may serve as a compromise between existing IR evaluation paradigms and [...] exploratory search evaluation.” The necessity of user studies makes evaluations cumbersome and, above all, expensive. By providing part of the solution (a decent corpus) for free, we want to overcome the outlined difficulties. Our corpus compiles a solid database of exploratory search behavior, which researchers may use for comparison purposes as well as for bootstrapping simulations.
Regarding standardized resources to evaluate exploratory search,
hardly any have been published up to now. White et al. [
A potential source of data for the purpose of assessing current
exploratory search behavior is to detect exploratory search tasks
within raw search engine logs, such as the 2006 AOL query log [
However, most session detection algorithms deal with short term
tasks only and the few algorithms that aim to detect longer search
missions still have problems when detecting interesting semantic
connections of intertwined search tasks [
To justify our choice of an exploratory task, namely that of writing
an essay about a given TREC topic, we refer to Kules and Capra [
CORPUS CONSTRUCTION
As discussed in the related work, essay writing is considered a valid approach to study exploratory search. Two data sets form the basis for constructing a respective corpus, namely (1) a set of topics to write about and (2) a set of web pages to research about a given topic. With regard to the former, we resort to topics used at TREC, specifically to those from the Web Tracks 2009–2011. With regard to the latter, we employ the ClueWeb09 (and not the “real web in the wild”). The ClueWeb09 consists of more than one billion documents from ten languages; it comprises a representative cross-section of the real web, is a widely accepted resource among researchers, and it is used to evaluate the retrieval performance of search engines within several TREC tracks. The connection to TREC will strengthen the compatibility with existing evaluation methodology and allow for unforeseen synergies. Based on the above decisions, our corpus construction steps can be summarized as follows: 1. Rephrasing of the 150 topics used at the TREC Web Tracks
2009–2011 so that they invite people to write an essay. 2. Indexing of the English portion of the ClueWeb09 (about 0.5 billion documents) using the BM25F retrieval model plus additional features. 3. Development of a search interface that allows for answering queries within milliseconds and that is designed along the lines of commercial search interfaces. 4. Development of a browsing interface for the ClueWeb09, which serves ClueWeb09 pages on demand and which rewrites links on delivered pages so that they point to their corresponding ClueWeb09 pages on our servers. 5. Recruiting 12 professional writers at the crowdsourcing plat
form oDesk from a wide range of hourly rates for diversity. 6. Instructing the writers to write essays of at least 5000 words length (corresponds to an average student’s homework assignment) about an open topic among the initial 150, using our search engine and browsing only ClueWeb09 pages. 7. Logging all writers’ interactions with the search engine and
the ClueWeb09 on a per-topic basis at our site. 8. Double-checking all of the 150 essays for quality.
After the deployment of the search engine and successfully completed usability tests (see Steps 2-4 and 7 above), the actual corpus construction took nine months, from April 2012 through December 2012. The post-processing of the data took another four months, so that this corpus is among the first, late-breaking results from our efforts. However, the outlined experimental setup can obviously serve different lines of research. The remainder of the section presents elements of our setup in greater detail.
Used TREC Topics.
Since the topics from the TREC Web Tracks 2009–2011 were not amenable for our purpose as is, we rephrased them so that they ask for writing an essay instead of searching for facts. Consider for example topic 001 from the TREC Web Track 2009:
Query. obama family tree Description. Find information on President Barack Obama’s family history, including genealogy, national origins, places and dates of birth, etc.
Sub-topic 1. Find the TIME magazine photo essay “Barack Obama’s Family Tree.” Sub-topic 2. Where did Barack Obama’s parents and grandparents come from? Sub-topic 3. Find biographical information on Barack
Obama’s mother.
This topic is rephrased as follows:
Obama’s family. Write about President Barack Obama’s family history, including genealogy, national origins, places and dates of birth, etc. Where did Barack Obama’s parents and grandparents come from? Also include a brief biography of Obama’s mother.
In the example, Sub-topic 1 is considered too specific for our purposes while the other sub-topics are retained. TREC Web track topics divide into faceted and ambiguous topics. While topics of the first kind can be directly rephrased into essay topics, from topics of the second kind one of the available interpretations is chosen.
A Search Engine for Controlled Experiments.
To give the oDesk writers a familiar search experience while
maintaining reproducibility at the same time, we developed a tailored
search engine called ChatNoir [
ChatNoir is based on the BM25F retrieval model [
Our choice of retrieval model and ranking features is intended to provide a reasonable baseline performance. However, it is neither near as mature as those of commercial search engines nor does it compete with the best-performing models proposed at TREC. Yet, it is among the most widely accepted models in the information retrieval community, which underlines our goal of reproducibility.
In addition to its retrieval model, ChatNoir implements two search facets: text readability scoring and long text search. The former facet, similar to that provided by Google, scores the readability of a text found on a web page via the well-known Flesh-Kincaid grade level formula: it estimates the number of years of education required in order to understand a given text. This number is mapped onto the three categories “simple”, “intermediate”, and “expert.” The long text search facet omits search results which do not contain at least one continuous paragraph of text that exceeds 300 words. The two facets can be combined with each other. They are meant to support writers that want to reuse text from retrieved search results.
Especially interesting for this type of writers are result documents containing longer text passages and documents of a specific reading 1http://boston.lti.cs.cmu.edu/clueWeb09/wiki/tiki-index.php?page=PageRank level such that reusing text from the results still yields an essay with homogeneous readability.
When clicking on a search result, ChatNoir does not link into the real web but redirects into the ClueWeb09. Though ClueWeb09 provides the original URLs from which the web pages have been obtained, many of these page may have gone or been updated since. We hence set up an interface that serves web pages from the ClueWeb09 on demand: when accessing a web page, it is pre-processed before being shipped, removing all kinds of automatic referrers and replacing all links to the real web with links to their counterpart inside ClueWeb09. This way, the ClueWeb09 can be browsed as if surfing the real web and it becomes possible to track a user’s movements.
The ClueWeb09 is stored in the HDFS of our 40 node Hadoop cluster, and web pages are fetched with latencies of about 200ms.
ChatNoir’s inverted index has been optimized to guarantee fast response times, and it is deployed on the same cluster.
Hired Writers.
Our ideal writer has experience in writing, is capable of writing
about a diversity of topics, can complete a text in a timely
manner, possesses decent English writing skills, and is well-versed in
using the aforementioned technologies. This wish list lead us to
favor (semi-)professional writers over, for instance, volunteer
students recruited at our university. To hire writers, we made use of
the crowdsourcing platform oDesk.2 Crowdsourcing has quickly
become one of the cornerstones for constructing evaluation
corpora, which is especially true for paid crowdsourcing. Compared
to Amazon’s Mechanical Turk [
Table 1 gives an overview of the demographics of the writers we hired, based on a questionnaire and their resumes at oDesk. Most of them come from an English-speaking country, and almost all of them speak more than one language, which suggests a reasonably good education. Two thirds of the writers are female, and all of them have years of writing experience. Hourly wages were negotiated individually and range from 3 to 34 US-dollars (dependent on skill and country of residence), with an average of about 12 US-dollars.
In total, we spent 20 468 US-dollars to pay the writers.
CORPUS ANALYSIS
This section presents the results of a preliminary corpus analysis that gives an overview of the data and sheds some light onto the search behavior of writers doing research. 2http://www.odesk.com Corpus Characteristic Writers Topics Topics / Writer Queries Queries / Topic Clicks Clicks / Topic Clicks / Query Sessions Sessions / Topic Days Days / Topic Hours Hours / Writer Hours / Topic Irrelevant Irrelevant / Topic Irrelevant / Query Relevant Relevant / Topic Relevant / Query Key Key / Topic Key / Query min Corpus Statistics.
Table 2 shows key figures of the query logs collected, including the absolute numbers of queries, relevance judgments, working days, and working hours, as well as relations among them. On average, each writer wrote 12.5 essays, while two wrote only one, and one very prolific writer managed more than 30 essays.
From the 13 651 submitted queries, each topic got an average of 91. Note that queries often were submitted twice requesting more than ten results or using different facets. Typically, about 1.7 results are clicked for consecutive instances of the same query.
For comparison, the average number of clicks per query in the aforementioned AOL query log is 2.0. In this regard, the behavior of our writers on individual queries does not seem to differ much from that of the average AOL user in 2006. Most of the clicks we recorded are search result clicks, whereas 2457 of them are browsing clicks on web page links. Among the browsing clicks, 11.3% are clicks on links that point to the same web page (i.e., anchor links using a URL’s hash part). The longest click trail observed lasted 51 unique web pages but most click trails are very short. This is surprising, since we expected a larger proportion of browsing clicks, but it also shows our writers relied heavily on the search engine. If this behavior generalizes, the need for a more advanced support of exploratory search tasks from search engines becomes obvious.
The queries of each writer can be divided into a total of 931
sessions with an average 12.3 sessions per topic. Here, a session is
defined as a sequence of queries recorded on a given topic which
is not divided by a break longer than 30 minutes. Despite other
claims in the literature (e.g., in [
In their essays, writers referred to web pages they found during their search, citing specific passages and topic-related information used in their texts. This forms an interesting relevance signal which allows us to separate irrelevant from relevant web pages. Slightly different to the terminology of TREC, we consider web pages referred to in an essay as key documents for its respective topic, whereas web pages that are on a click trail leading to a key document are relevant. The fact, that there are only few click trails of this kind explains the unusually high number of key documents compared to that of relevant ones. The remainder of web pages which were accessed but discarded by our writers may be considered irrelevant.
The writer’s search interactions are made freely available as the
Webis-Query-Log-12.3 Note that the writing interactions are the
focus of our accompanying ACL paper [
Exploring Exploratory Search Missions.
To get an inkling of the wealth of data in our corpus, and how it may influence the design of exploratory search systems, we analyze the writers’ search behavior during essay writing. Figure 1 shows for each of the 150 topics a curve of the percentage of queries at any given time between a writer’s first query and an essay’s completion.
We have normalized the time axis and excluded working breaks of more than five minutes. The curves are organized so as to highlight the spectrum of different search behaviors we have observed: in row A, 70–90% of the queries are submitted toward the end of the writing task, whereas in row F almost all queries are submitted at the beginning. In between, however, sets of queries are often submitted in short “bursts,” followed by extended periods of writing, which can be inferred from the plateaus in the curves (e.g., cell C12). Only in some cases (e.g., cell C10) a linear increase of queries over time can be observed for a non-trivial amount of queries, which indicates continuous switching between searching and writing.
From these observations, it can be inferred that query frequency alone is not a good indicator of task completion or the current stage of a task, but different algorithms are required for different mission types. Moreover, exploratory search systems have to deal with a broad subset of the spectrum and be able to make the most of few queries, or be prepared that writers interact only a few times with them. Our ongoing research on this aspect focuses on predicting the type of search mission, since we found it does not simply depend on the writer or a topic’s difficulty as perceived by the writer. 4. SUMMARY
We introduce the first corpus of search missions for the exploratory task of writing. The corpus is of representative scale, comprising 150 different writing tasks and thousands of queries, clicks, and relevance judgments. A preliminary corpus analysis shows the wide variety of different search behavior to expect from a writer conducting research online. We expect further insights from a forthcoming in-depth analysis, whereas the results mentioned demonstrate the utility of our publicly available corpus. 3http://www.webis.de/research/corpora 5.
Interactive Exploration of Geographic Regions with Web-based Keyword Distributions
Chandan Kumar University of Oldenburg,
Oldenburg, Germany chandan.kumar@unioldenburg.de
Wilko Heuten
OFFIS – Institute for Information Technology,
Oldenburg, Germany wilko.heuten@offis.de
Dirk Ahlers NTNU – Norwegian University of Science and Technology,
Trondheim, Norway dirk.ahlers@idi.ntnu.no
Susanne Boll University of Oldenburg,
Oldenburg, Germany susanne.boll@unioldenburg.de ABSTRACT The most common and visible use of geographic information retrieval (GIR) today is the search for specific points of interest that serve an information need for places to visit. However, in some planning and decision making processes, the interest lies not in specific places, but rather in the makeup of a certain region. This may be for tourist purposes, to find a new place to live during relocation planning, or to learn more about a city in general. Geospatial Web pages contain rich spatial information content about the geo-located facilities that could characterize the atmosphere, composition, and spatial distribution of geographic regions. But the current means of Web-based GIR interfaces only support the sequential search of geo-located facilities and services individually, and limit the end users on abstracted view, analysis and comparison of urban areas. In this work we propose a system that abstracts from the places and instead generates the makeup of a region based on extracted keywords we find on the Web pages of the region. We can then use this textual fingerprint to identify and compare other suitable regions which exhibit a similar fingerprint. The developed interface allows the user to get a grid overview, but also to drill in and compare selected regions as well as adapt the list of ranked keywords.
Categories and Subject Descriptors
H.3.3 [Information Storage and Retrieval]: Information
Search and Retrieval; H.5.2 [Information Interfaces and
Presentation]: User Interfaces
1. INTRODUCTION
Geospatial search has become a widely accepted search mode
o↵ ered by many commercial search engines. Their
interfaces can easily be used to answer relatively simple requests
such as “restaurant in Berlin” on a point-based map
interface, which additionally gives extended information about
entities [
To support more complex tasks we propose a new retrieval method based on entities. For example, sometimes the distribution of results on a map can already inform certain views about areas, e.g., a search for “bar” may show a clustering of results that can be used for “eyeballing” a region of nightlife even without sophisticated geospatial analysis.
However, as users become more used to local search, more
complex search types and supporting analysis are desired
that enable a combined view onto the underlying data [
Exploration of geographic regions and their characterization
was found as one of the key desire of local search users in
our requirement study [
We also aim to provide new visual tools for the exploration of geographic regions. While the necessary multi-dimensional geospatial data is already available, there is no suitable interface to query them, let alone to deal with the multi-criteria complexity. In this paper we describe a visual-interactive GIR system to support the retrieval of relevant geospatial regions and enable users to explore and interact with geospatial data. We propose a new query-by-spatial-example interaction method in which a user-selected region’s characteristic is fingerprinted to present similar regions. Users can interactively refine their query to use those characteristics of a region that are most important to them. For a more detailed overview, we use the full text of georeferenced Web pages for queries and analysis. This work goes beyond conventional GIR interfaces as it allows users to interact with aggregated spatial information via spatial queries instead of only textual querying, which is especially important to define regions of interest. We discuss the necessary input, visualization, comparison, refinement, and ranking methods in the remainder of this paper. 2. USING THE GEOSPATIAL WEB TO CHAR
ACTERIZE GEOGRAPHIC REGIONS
The distribution of geo-entities is used to illustrate the
characteristics and dynamics of a geographic region. A
geoentity is a real life entity at a physical location, e.g., a
restaurant, theatre, pub, museum, business, school, etc. To
open these entities up for aggregate and multi-criteria region
characterization, they need a certain depth of information
associated with them. It is obvious that only position
information or the name of a place is insu cient, so categorial
or textual description is needed. For initial studies [
To actually make the connection from a location to Web
pages, we assume that the presence of location references
on a page is a strong indication that the page is associated
with the entity at that location. We use our geoparser to
extract location references and thereby assess the geographical
scopes of a page. The geoparser is trained to the presence of
location references in the form of addresses within the page
content. This is a suitable approach for the urban areas
we are addressing in this work, because we need a
geospatial granularity at the sub-neighborhood level.
Knowledgebased identification and verification of the addresses is done
against a gazetteer extended with street names, which we
1http://www.openstreetmap.org/
fetched from OSM for the major cities of Germany. To
retrieve actual pages, we crawled the Web with a geospatially
focused crawler [
GEOGRAPHIC REGIONS OF INTEREST We have implemented two main interaction modes in the Web interface as shown in Figure 1. A user intends to compare multiple geographic regions of Frankfurt (target region, right in the dual-map view) with respect to a certain relevant region in Berlin (query region, left). The current reference region of interest is specified via a visual query. The user can then either select regions by placing markers onto the map, or alternatively use a grid overview (right side of Figure 1). In both cases, the system computes the relevance of the target regions with respect to the characteristics of the query region. 3.1 Query-by-spatial-example Most GIR interfaces use a conventional textual query as input method to describe user’s information need or use the currently selected map viewport. We wanted to give users the ability to arbitrarily define their own spatial region of interest. The free definition of the query region is important, as users may not always want a neighborhood that is easily describable by a textual query. We therefore enabled to query by spatial example, where users can define the query region by drawing on map. Figure 1 shows an example of a user selected region of interest via a polygon query (by mouse clicks and drag) in the city of Berlin. 3.2 Visualization of suitable geographic regions Users can select several location preferences in their target region that they would like to explore by positioning markers on the map interface. The system defines the targets with a circle around the user-selected locations with the same diameter as the reference region polygon. The target regions obtain the ranking with respect to their similarity with the reference region. Their relevance is shown by the percentage similarity and the heatmap based relevance visualization. We used a color scheme of di↵ erent green tones which di↵ ered in their transparency. Light colors represented low relevance, dark colors were used to indicate high relevance.
The color scheme selection was aided by ColorBrewer 2.
As an example, Figure 1 shows 4 user-selected locations on the city map of Frankfurt, the circle regions around these 4 markers have the same diameter as the query region in Berlin. The target region in the centre of the city is most relevant with the similarity of 88%, and consequently has the darkest green tone. If a user has not yet formed any preference, we o↵ er an aggregate overview of geo-entities.
We partition the map area using a grid raster [
regions via keyword distributions Interaction models should provide end users the opportunity to explore the characteristics of selected regions, and adapt it further to their requirements. We initially show the most relevant keywords of the respective region using a word cloud. The word cloud provides more detailed information on keyword distribution when the mouse hovers over it. The font size and order of the keywords signify their relevance. Figure 2 shows the comparison of the query region with the most relevant target region via both their keyword distributions. In this case, the distributions of both regions are very similar, leading to the high relevance score for the target region.
Since the keyword characteristics of a query region is derived from the georeferenced Web pages, there are situations where a user might not be satisfied with the spatial descrip2http://colorbrewer2.org tion and wants to influence the keywords. In the example of Figure 3, a user decides that pubs are more important than restaurant, fast food is not an aspect of his lifestyle and should be replaced by education facilities near his new home. In such scenarios users need to interact and adapt the generated keyword distributions of query regions. We make the word cloud interactive and editable. Users can drag keywords to alter their position and thus their significance. They can also edit, delete or replace keywords in the word cloud to change the criteria. After modifying the keyword distribution, users can revisualize the target regions to update their ranking. Figure 3 shows this user interaction with the word cloud, including the revisualization of the updated ranking of target regions, which are visibly di↵ erent from the previous ranking of Figure 2. 4. TEXT-BASED CHARACTERIZATION AND
RANKING OF GEOGRAPHIC REGIONS
We adapt common IR methods for ranking and similarity
measures. In relevance-based language models, the
similarity of a document to a query is the probability that a given
document would generate the query [
Regions are considered as compound documents built from
the Web pages of the entities inside them. We can then
define the similarity of document clusters of regions based
on the probability that the target region can generate the
query region. The Kullback-Leibler divergence is used for
comparison [
For a geospatial document d, we estimate P (w|d) , which is a unigram language model , with the maximum likelihood estimator, simply given by relative counts: P (w|d) = tf(w,d) ,
|d| here tf (w, d) is the frequency of word w in the document d and |d| is the length of the document d. A geographic region contains several geospatial documents insides its footprint area. We define a geographic region based on a document cluster D which contains document {d1, d2....dk}, and the distribution of a particular word w in the geographic region would be estimated with its combine probability in the collection P (w|D) = k1 Pk
i=1 P (w|di). The word cloud represents the most prominent keywords of the region with respect to their ranked probability distribution P (w|D). The comparison of regions is done with respect to their probability distribution using KL-divergence. A target region x will be compared to the query region as following
Relevance(Regionx) =
X P (w|Dq)log w
P (w|Dq)
P (w|Dx)
The computation of this formula involves a sum over all
the words that have a non-zero probability according to
P (w|Dq). Each region Regionx gets a relevance score
according to its distribution comparison to the query region
Regionq. All target regions (user selected regions or grid
based divisions) are ranked with respect to their relevance
score for visualization.
5. RELATED WORK
The field of geographic information retrieval examines
documents’ geospatial features at a regional scale and also at
smaller granularities and usually supports keyword@location
queries [
However, search for multiple categories or other complex
tasks is usually not supported. Some non-conventional
spatial querying methods have been proposed, e.g.,
query-bysketch on a map [
None of these approaches make a larger word cloud available,
but only the main terms. Other geovisualization approaches
[
Our system interface di↵ ers in the granularity of information need and representation, i.e., we focus on the ranking of regions, but base it on high-granularity geo-entities that have a very exact location, which ensures that the spatial query does not produce overlap to neighboring regions and makes the multi-criteria analysis more exact to be executed at arbitrary region sizes. 6. CONCLUSIONS AND FUTURE WORK Most current local search interfaces do not o↵ er adequate support for the exploration and comparison of geographic areas and regions. End users need visual and interactive assistance from GIR systems for an abstracted overview and analysis of geospatial data. We proposed interactive interfaces for the characterization and assessment of relevant geographic regions that enable end-users to query, analyze and interact with the rich geospatial data available on the Web in user-selected geographic regions. The relevance of regions is based on the similarity of keyword distributions.
The observation of results shows satisfactory performance by uncovering realistic and meaningful keywords defining the regions. We observed that the characterization and comparison of geographic regions show good results with respect to geo-located facilities and infrastructure of German cities, e.g., clearly distinct characteristics for university, industrial, or party districts. In the future we plan a more formal qualitative and quantitative evaluation of these interfaces, to examine the acceptance of these visualizations with regard to user-centered aspects such as exploration ability, information overload, and cognitive demand. We would also like to explore more advanced interaction methods to enhance the usability of the proposed visualizations.
Additionally, we envision more powerful region similarity measures such as landscape and topological similarity, similarity via social media, and an integration of additional data sources.
Acknowledgments The authors are grateful to the DFG SPP 1335 ‘Scalable Visual Analytics’ priority program which funds the project UrbanExplorer. The 2nd author acknowledges funding from the ERCIM “Alain Bensoussan” Fellowship Programme.
Inferring Music Selections for Casual Music Interaction
Daniel Boland University of Glasgow
United Kingdom daniel@dcs.gla.ac.uk
Ross McLachlan University of Glasgow
United Kingdom r.mclachlan.1@ research.gla.ac.uk
Roderick Murray-Smith
University of Glasgow
United Kingdom rod@dcs.gla.ac.uk ABSTRACT We present two novel music interaction systems developed for casual exploratory search. In casual search scenarios, users have an ill-defined information need and it is not clear how to determine relevance. We apply Bayesian inference using evidence of listening intent in these cases, allowing for a belief over a music collection to be inferred. The first system using this approach allows users to retrieve music by subjectively tapping a song’s rhythm. The second system enables users to browse their music collection using a radio-like interaction that spans from casual mood-setting through to explicit music selection. These systems embrace the uncertainty of the information need to infer the user’s intended music selection in casual music interactions.
Categories and Subject Descriptors
H.5.2 [Information interfaces]: User Interfaces
General Terms
Design, Human Factors, Theory
1. INTRODUCTION
When interacting with a music system, listeners are faced
with selecting songs from increasingly large music
collections. With services like Spotify, these libraries can include
many songs the user has never heard of. This retrieval is
often a hedonic activity and may not serve a particular
information need. Users do not always have a song in mind
and are often just interested in setting a mood or finding
something ‘good enough’ [
Presented at EuroHCIR2013. Copyright c 2013 for the individual papers by the papers authors. Copying permitted only for private and academic purposes. This volume is published and copyrighted by its editors.
Music listeners are not always fully engaged with the
selection of music - as evidenced by the success of the shu✏ e
playback feature. Large libraries of music such as Spotify are
available but users often just want background music, not a
specific song out of millions. In these casual search
scenarios, users often satisfice i.e. search for something which is
‘good enough’ [
UNCERTAIN MUSIC SELECTION By asking ‘What would this user do?’, we can develop a likelihood model of user input within an interaction. With Bayes theorem, this allows for an uncertain belief over a music space to be inferred. Users can provide evidence of their listening intent as part of a casual music interaction, not needing to be fully engaged in the music retrieval. This is an explicitly user-centered approach, focusing on how a user will interact with the system. Both the systems discussed here have been iteratively developed by comparing real user behaviour against that predicted by the user input models.
We present two novel music retrieval systems which explore
two challenges with this approach: i) how to correctly
interpret evidence which may be subjective and ii) how to allow
users to set their current level of engagement:
i) ‘Query by Tapping’ is a music retrieval technique where
users tap the rhythm of a song in order to retrieve it [
As part of a user-centred development process, we identified that rhythmic queries are often subjective and so developed a model of rhythmic input which captures some of this subjective behaviour. This allows for the system to be trained to the user’s tapping style, giving significant improvements over previous e↵ orts at rhythmic music retrieval. ii) FineTuner is a prototype of a radio-like music interface that enables users to retrieve music at a level of engagement suited to their current information need. Users navigate their music collection using a dial, with the system using prior knowledge of the user to inform the music selection. A pressure sensor enables users to assert varying levels of control over the system – with no pressure, users can casually tune in to sections of their music collection to hear recommended music with common characteristics. As pressure is applied, the user is able to make increasingly specific selections from the collection. The inferred music selection is conditioned upon the asserted control, allowing for the seamless transition from casual mood-setting to engaged music interaction. User 1
User 2
MODELLING SUBJECTIVITY In this section we describe our e↵ orts to model the subjectivity of rhythmic queries, yielding a query by tapping system for casual music retrieval which can be trained to users to account for their subjective querying style. After training the system, a user can tap a rhythm to re-order their music collection by rhythmic similarity to their query. The top 20 highly ranked results are listed on-screen as a music playlist, from which the user could also then select a specific song.
Query by tapping provides an example of a casual music
interaction which su↵ ers from subjective queries. In
mobile music-listening contexts, it can often be inconvenient
for users to remove their mobile device from their pocket
or bag and engage with it to select music. This tapping of
music as a querying technique for music is depicted in figure
2. Tapping a rhythm is already a common act and rhythm
is a universal aspect of music [
In order to assign a belief to the songs in the music
collection given a rhythmic query, we compare the query to
those predicted by the user input model. This comparison is
done using the edit distance from string comparison
methods, scaling the mismatch penalty to the time di↵ erences
between the rhythmic sequences [
Their work is limited however in considering only monophonic rhythms i.e. the rhythm from only one instrument, as opposed to being polyphonic and comprising of multiple instruments. Their music corpus consists of MIDI representations of tunes such as ”You are my sunshine” which is hardly analogous to real world retrieval of popular music.
Rhythmic interaction has been recognised in HCI [
In previous implementations of QBT, each IOI is defined
relative to the preceding one [
100 )75 % ( e t a r iitno50 n g o c re25 0 Dial position
Dial position
Dial position
(a)
(b)
(c)
The approach to rhythmic interaction in [
Evaluation
The most important metric for the system to be usable was
whether a rhythmic input produced an on-screen (top 20)
result. We asked eight participants to provide queries for
songs selected from a corpus of 300 songs which we had
complete note onset data for. Participants listened to the
songs first to ensure familiarity and were asked to provide
training queries for each song. These training queries were
used to train the generative model using leave-one-out
crossvalidation. We use a state-of-the-art onset detection
algorithm (based on measuring spectral flux [
MODELLING ENGAGEMENT
We consider casual search interactions as spanning a range
of levels of engagement. How much a user is willing to
engage with a system and provide evidence of their listening
intent will undoubtedly vary with listening context. An
interaction which is fixedly casual would be as problematic
as one which requires a user’s full attention, with users
unable to take control when they wish to. An example of this
would be old analogue radios – whilst they o↵ er a simple
music interaction, users have limited control over what they
hear. Previous work by Hopmann et al. sought to bring the
benefits of interaction with vintage analog radio to modern
digital music collections [
We explore how the inference of listening intent can be conditioned upon the user’s level of engagement, with the music interaction spanning from casual mood-setting through to specific song selection. While it would be desirable to bring the simplicity of radio-like interaction to modern music collections, mapping a modern music collection to a dial such as in figure 5 would require prolonged scrolling. An alternative would be to instead support scrolling through an overview of the music space however this removes granularity of control from the user, leaving them unable to select specific items.
We developed a radio-like system called FineTuner that allows users to navigate their music, which is arranged along a mood axis. Users can ‘tune in’ to a mood to hear recommended songs based on their listening history. FineTuner allows the user to assert control over the music recommendation by applying pressure to a sensor. This enables users to seamlessly transition from a casual style of interaction akin to a radio to controlling styles such as specifying a particular sub-area of interest in a music space, or even selecting individual songs. FineTuner provides a single interaction which supports casual search through to fully engaged retrieval. Our system enables both casual and engaged forms of interaction, giving users varying degrees of control over the selection of music. In casual interactions where users apply less pressure, the system can become more autonomous – making inferences from prior evidence about what the user intended.
This handover of control was termed the ‘H-metaphor’ by
Flemisch et al. where it was likened to riding a horse – as
the rider asserts less control the horse behaves more
autonomously [
As the user begins an interaction, they have not applied pressure and therefore are not asserting control over the system.
The inferred selection is thus broad, covering an entire region of their collection and is biased towards popular tracks (fig. 4a). The music in the inferred selection is visualised by randomly sampling tracks from it and drawing beams from the dial position to the album art. The user may press in the knob to accept the selection and the sampled track is played.
At low levels of assertion it is likely that most tracks played would be highly popular tracks. This behaviour is a design assumption, users may want the system to use other prior evidence. When the user applies pressure, the system interprets this as an assertion of control. The inferred selection is smaller and the spread of beams becomes narrower, the album art visualisation zooms in to show the smaller selection (fig. 4b). This selection is a combination of evidence from the dial position with prior evidence i.e. their last.fm music history. When users fully assert control (max. pressure), they navigate the collection album by album (fig. 4c) and can make exact selections. By varying the pressure, users seamlessly move through this continuous range of control.
The smooth change in engagement is achieved using a simple model of user input. We assume that in an engaged interaction, users will point precisely at the song of interest (as in fig. 4c). For more casual selection, we assume that users will point in the general area (mood) of the music they want, modelled using a normal distribution as in (fig. 4b). As less pressure is applied the distribution is widened, leading to less precise selection and a greater role for a prior belief over the music collection such as listening history. 5. SUMMARY The scenarios explored here involve casual music retrieval, where users have an ill-defined information need and browse for hedonic purposes or to satisfice a music selection. In these cases, considering what input a user would provide for target songs and inferring selections is an intuitive approach which avoids the issue of defining relevance. We show two music interactions which support the uncertain selection of music, inferred from casual user input such as tapping a rhythm or turning a radio dial.
We have shown that modelling user input for inferring music selection can address issues of subjectivity by taking a user-centered approach to model development. The model can be iterated by comparing its predictions against actual user behaviour. Accounting for this subjectivity can yield significant improvements in retrieval performance as well as creating a more personalised search experience. A key feature of the second system, FineTuner, is its ability to span seamlessly from casual search scenarios, such as satisficing, through to more explicit selections of music. By conditioning the inference upon the user’s level of engagement, we are able to interpret the same input space (in this case the dial) according to the current context.
Our approach to casual music interaction empowers the user to enjoy their music while expending as much or as little e↵ ort in the retrieval as they wish, providing queries in their own subjective style. Instead of focusing solely on optimising the retrieval process, we consider it equally important to design retrieval systems which suit how the user currently wants to interact. By considering how users might provide casual evidence for their listening intent, we achieve music interactions as simple as tapping a beat or tuning a radio.
ACKNOWLEDGMENTS We are grateful for support from Bang & Olufsen and the Danish Council for Strategic Research.
Search or browse? Casual information access to a cultural heritage collection Robert Villa, Paul Clough, Mark Hall, Sophie Rutter
Information School University of Sheffield
Sheffield, UK
S1 4DP {r.villa, p.d.clough, m.mhall, sarutter1} @sheffield.ac.uk ABSTRACT Public access to cultural heritage collections is a challenging and ongoing research issue, not least due to the range of different reasons a user may want to access materials. For example, for a virtual museum website users may vary from professionals or experts, to interested members of the public visiting on a whim. In this paper, we are interested in the latter user: a user who visits a cultural heritage website without a clear goal or information need in mind. In the user study reported here, carried out within the context of the interactive task at CLEF (interactive CHiC), 20 participants explored a subset of Europeana with no explicit task provided using a custom-built interface that offered both search and browse functionalities. Results suggest that browsing is used considerably more by the majority of users when compared to text search (all participants used the category browser before carrying out a text search). This highlights the need for cultural heritage search interfaces to provide browsing functionality in addition to conventional text search if they wish to support casual search tasks.
General Terms Design, Experimentation, Human Factors.
Keywords
Cultural heritage, virtual museums, information access.
1. INTRODUCTION
Providing public access to cultural heritage is an ongoing and
challenging area of research. Previous work suggests that visitors
to online cultural heritage collections (e.g. virtual museum
visitors) are not necessarily motivated by an explicit task, and that
interacting with cultural heritage collections is exploratory in
nature [
The focus for this paper is how individuals explore a cultural heritage collection when given no task. The results may be used both to contrast with studies which have used explicit tasks, and to motivate changes to cultural heritage systems to better support a diverse range of user tasks.
Presented at EuroHCIR2013. Copyright © 2013 for the individual papers by the papers’ authors. Copying permitted only for private and academic purposes. This volume is published and copyrighted by its editors.
The work reported here is based on initial results from the
Interactive CHiC (Cultural Heritage in CLEF) track of CLEF1 as
run at Sheffield University. The interactive CHiC track is based
on the CHiC Europeana data set as used in 2011 and 2012 [
We consider three research questions:
RQ1. How do participants initiate their exploration? RQ2. Do participants use browse or search in their exploration
of the collection? RQ3. How do participants decide to search or browse, when
given no explicit task?
With RQ1 we are particularly interested whether users start their
exploration by browsing categories, or by search. RQ2 then
considers how users access the collection over their whole
session. For RQ3 we will present some initial qualitative data
from our lab-based interactive study, where the aim is to identify
reasons for the use of either the search or browse functions.
2. PREVIOUS WORK
A general review of museum informatics is provided in [
Museum visitors can, in some respects, be considered as examples
of “casual leisure” searchers, as outlined in [
Darby and Clough [
However, unlike in previous work, such as [
The interface is split into five main areas, clockwise from left to right: a category browser, search box, item display, bookbag, and search results. The search box operates in the conventional manner, allowing free text queries with search results being displayed as a grid below. When a result is clicked, it is displayed in the “item display” on the right. This information will typically include a small thumbnail, textual description, and the item’s associated metadata. Metadata is clickable, e.g. if an item is listed as being owned by the British Library, clicking on the field will search for British Library objects. At the bottom of the item display is a “more like this”, which displays the images of up to eight similar objects, which can be viewed three at a time.
On the left of the interface is the “category browser”, which
allows the user to browse the Europeana collection through a
hierarchy of categories. This hierarchy is automatically generated,
and is based on the work of [
In total 20 participants were recruited for the study, 11 male and 9 female. Eight participants were in the 18-25 year age band, nine in the 26-35 band; the other 3 between 36-45. The majority were students (13), with 5 employed, one unemployed, and one “other”. 13 had completed a higher education degree, while six were currently studying an undergraduate degree. 5. RESULTS 5.1 Initiation of exploration RQ1 asks how users initiate their exploration of the collection. To investigate this, we first looked at how users started their session, and in particular, their searching. For example, did they select a category or enter a query? Over the whole data set four different actions were used by participants to initiate their session (Table 1, column 2). For the majority of users, the first action was to select one of the categories (15 out of the 20 users). It should be noted that the interface, on startup, showed a set of default results to all users.
For three users, the first action was to display one of these default results, another user clicked the “next page” to view the next page of default results, while the final user’s first action was to bookmark one of the default result items.
We also investigated the logs to find out each user’s first search or browse action, which could be one of category select, text query, or metadata/more like this select. As shown in Table 1 (column 3), for all users this was a category select. In addition to counting the first actions, we also investigated how long each user spent before either clicking the interface, or starting a new 2 http://lucene.apache.org/solr/ 3 http://www.europeana.eu/ it generates a query message, which is usually connected to a Standard Results List.
Standard Results List
The Standard Results List component ([
Pagination
The Pagination component ([
Category Browsing
The Category Browsing component ([
Saved Documents
The Saved Documents component provides an area where the participant can save things that they have found interesting, to support them in their current task. Documents are added through a save_document message. The Saved Documents component supports an optional tagging feature enabling the participant to tag the document with values specified by the researcher. This can be used to let the participant specify why they have chosen that document or how much it helps them in their current task. 3.6
Task
The Task component provides a static display of the task information to show to the user. Two versions of this component are provided, one that displays a static text set in the configuration, and one that can fetch a task description from the database, based on a parameter passed to it.
The evaluation work-bench has so far been used to build two IIR experiments, very di↵ erent in their nature, clearly demonstrating the work-bench’s flexibility.
The first experiment (fig. 4) re-uses the standard Task, Search Box, Pagination, and Saved Documents components, and extends the Standard Results List to work with the specific search backend. This set-up re-creates what is essentially a relatively standard search UI configuration, that is being used to investigate query session behaviour.
The second experiment (fig. 5) demonstrates a much richer interface, with more modifications to the components and an experiment-specific component. It re-uses the Task and Category Browsing components, extends the default Search Box, Pagination, Standard Results List, and Saved Documents components, and adds a new Item View component. The message-passing nature of the system made it possible to quickly integrate the new component, so that when the participant clicks on a meta-data facet in the Item View, a query message is sent to the Standard Results List to find items with the same bit of meta-data. The interface was used to investigate un-directed exploration behaviour in a large digital cultural heritage collection.
WHERE TO GO NEXT?
The stated aim of this paper was to present a novel, pluggable, extensible, and configurable IIR interface work-bench, that supports our wider aim of improving IIR experiment comparability. The work-bench is su ciently flexible to support the wide range of web-based IIR experiments that are undertaken, while being su ciently simple and light-weight to encourage wide-spread use of the workbench.
To enable this wide-spread use, the system has been released under an open-source license6. We are also moving to engage with the wider research community to determine to what degree the work-bench satisfies their needs for an evaluation system and what needs to be done to achieve the wide-spread use needed to improve IIR experiment comparability.
ACKNOWLEDGEMENTS
The research leading to these results was supported by the Network of Excellence co-funded by the 7th Framework Program of the European Commission, grant agreement no. 258191. 6https://bitbucket.org/mhall/pyire
A Proposal for User-Focused Evaluation and Prediction of Information Seeking Process
Chirag Shah School of Communication & Information (SC&I)
Rutgers University 4 Huntington St, New Brunswick, NJ 08901, USA
chirags@rutgers.edu ABSTRACT One of the ways IR systems help searchers is by predicting or assuming what could be useful for their information needs based on analyzing information objects (documents, queries) and finding other related objects that may be relevant. Such approaches often ignore the underlying search process of information seeking, thus forgoing opportunities for making process-based recommendations. To overcome this limitation, we are proposing a new approach that analyzes a searcher’s current processes to forecast his likelihood of achieving a certain level of success in the future. Specifically, we propose a machine-learning based method to dynamically evaluate and predict search performance several time-steps ahead at each given time point of the search process during an exploratory search task. Our prediction method uses a collection of features extracted solely from the search process such as dwell time, query entropy and relevance judgment in order to evaluate whether it will lead to low or high performance in the future.
Experiments that simulate the effects of switching search paths show a significant number of subpar search processes improving after the recommended switch. In effect, the work reported here provides a new framework for evaluating search processes and predicting search performance. Importantly, this approach is based on user processes, and independent of any IR system allowing for wider applicability that ranges from searching to recommendations.
Categories and Subject Descriptors H.3: INFORMATION STORAGE AND RETRIEVAL H.3.3: Information Search and Retrieval: Search process; H.3: INFORMATION STORAGE AND RETRIEVAL H.3.4: Systems and Software: Performance evaluation (efficiency and effectiveness) General Terms Measurement, Performance, Experimentation Keywords Exploratory search, Evaluation, Performance prediction Presented at EuroHCIR2013.
Copyright © 2013 for the individual papers by the papers’ authors.
Copying permitted only for private and academic purposes. This volume is published and copyrighted by its editors. 2 1
INTRODUCTION
IR evaluations are often concerned with explaining factors
relating to user or system performance after the search and
retrieval are conducted [20]. Most recommender systems,
however, operate with an objective to suggest objects that could
be useful to a user based on his/her or others’ past actions
[
The underlying rational processes involved in information
search are reflected in the actions users make while
searching. These actions include entering search queries,
skimming the results, as well as selecting and collecting
useful information [
A searcher’s performance is a function of these actions
performed during a search episode [
With these assumptions, we propose to quantify a search process using various user actions, and use it for user performance (henceforth, ‘search performance’ or ‘performance’) prediction as well as search process recommendations.
BACKGROUND Past research on predictive models that relates to the approach we describe in this paper can be grouped into two main categories: (1) behavioral studies and (2) IR approaches. In both cases; however, the focus has been on end products instead of in the process required to produce them.
As far as the behavioral studies go, research has been conducted
to explore users models that help anticipating specific aspects of
the search process. One goal in this context has been the
determination of whether a search process will be completed in a
single or multiple sessions. For example, Agichtein et al. [
There have been also attempts to understand past users' behaviors in order to predict future ones in similar conditions.
For example, Adar et al. [
Research like the ones described above often relies on historic
data from large populations and the use of trend and seasonal
components, which are used to model long-term direction and
periodicity patterns of time-series [
From an IR perspective, Radinski et al. [
In a similar track, several works have been conducted on query
performance prediction, focusing on developing techniques that
help IR system to anticipate whether a query will be effective or
not to provide results that satisfy users’ needs [
Results from this study have direct implications to individual users by aiding the auto evaluation process of IR systems.
In information search, users may be unaware of their individual performance when solving an information search task. For instance, Shah & Marchionini [23] showed how lack of awareness about different objects involved in searching (queries, visited pages, bookmarks) could result in mistaken perception about search performance during an exploratory search task.
Even if an IR system is highly effective, users may run into multiple query formulation and evaluation of several pages before finding what they need. This process, which can be related to search strategies, implies effort and time that is usually underestimated by the users themselves. In this sense, instead of predicting end products (i.e., overall performance), the approach we introduce in this paper is oriented toward predictions at different times in order to increase the level of awareness of users about their own search process. Similar to weather forecast, this information could help users to be aware of possible trends based on past and current behavior.
For a more recent discussion on IR evaluations and their
shortcomings, see [
Unlike previous works in IR, we are not proposing to use time series analyses or seasonal components of historic data. Instead, we investigate predictive models based on machine learning (ML) techniques; namely: SVM, logistic regression, and Naïve Bayes which are trained over a set of features such as time, number of queries, and page dwell time. In contrast to most IR evaluations, our method focuses on user-processes. Also, unlike most recommender systems, our approach could output alternative strategies instead of similar/relevant products to help the searcher. In essence, the work reported here takes several lessons from tradition IR evaluations, recommender system designs, and weather/stock forecasting to come up with a new approach for evaluating and predicting search performance.
In the next section we provide a detailed description of our method, feature selection, and the measures we used in order to create ML-based predictive models. 3 METHOD In order to analyze the search processes followed by different users/teams, we assume that the underlying dynamics of the search processes are expressed by a collection of activities that take place from the beginning to the end of the search processes.
The first part of our method is a feature extraction step in which we extract a wide array of features relating to webpages, queries and snippets saved from the search processes for each unit of time t. This step is performed in order to evaluate how well we could use those features to capture the underlying dynamics which would lead to recognizing whether a search process is going to lead to high or low performance in the future time steps at t+n (n=1,2,….,N), where N is the furthest time step.
The decision to include or exclude a feature was based on
literature (e.g., [
Total coverage (u,t): The total number of distinct Webpages visited by a user (u) up to time t. This feature captures the Webpage based activity performed by a user and provides a measure to see how much distinct information has been found by the user up to this time.
Useful coverage (u,t): The total number of distinct
webpages in which a user spent at least 30 seconds, up to
time t. This measure evaluates out of the total pages he/she
has visited how many of them were useful in finding
relevant information leading to satisfaction with their
context in completing the exploratory task [
Number of queries (u,t): Total number of unique queries executed by a user up to time t. This feature implicitly relates to how much effort and cognitive thinking a user has put in to this task.
Number of saved snippets (u,t): Total number of snippets saved by user u up to time t. This measures the amount of information that the user thought that might be relevant in the future to complete the task and needed to be remembered. In other words, this feature is an indication of explicit relevance judgments made by the user.
Length of Query (u,q,t): Length of each query(q) executed by a user u based on the character count of the query up to time t. This feature captures how the user imposed the • queries and how long they were at different times of the search process.
Number of tokens in each query (u,q,t): This is the count of tokens/words in each query(q) executed by user u up to time t. This query based measure takes into account how specific a user was in defining the query. By inspecting the datasets, we realized that queries with a less number of tokens tend to get general results. On the other hand, composed queries with multiple terms are related to more specific searchers. We also observed that typically the users started with general queries with few words at the beginning of the search process but then went into more detailed queries to find more specific information later. For all these reasons, we found it to be useful to capture the number of token used in a query.
Query entropy (u,q,t): This measures the information content in a given query (q), by finding the expected value of information contained in a query. We used the widely recognized notion of Shannon entropy [24] in Information Theory to calculate the information content of a query. We calculated the number of unique characters appearing in each of the queries, which represent the observed counts of the random variable. This was used as the input to Shannon entropy calculation and we used to the maximumlikelihood method to calculate the entropy. Query entropy feature has been used in the past to predict goodness of a query for making query expansion decision [21].
The method used to assess the search performance of a user is
described below. We define a measure called Efficiency (u,t), for
each user u up to time t in order to predict whether a given
search process is going to yield in high/low performance in the
future We first define Effectiveness of user u up to time t as the
ratio of useful coverage and total coverage (both defined
earlier). A similar measure was used in [
Effectiveness(u,t) =
Useful coverage(u,t) Total coverage(u,t) (1) (2) We then calculated Efficiency as defined in Equation 2.
Efficiency(u,t) =
Effectiveness(u, t)
NumberofQueries(u,t) In other words, Efficiency is defined as the Effectiveness obtained per query, or how effective a query is in terms of achieving a certain level of useful coverage.
The performance for each user u at each time t was classified in to the two classes; high performance and low performance based on the following criteria: Class = { high ;if low ;else
Efficiency(u, t) ≥ Efficiency(u, t)
(3) Using various user studies data available to us, we constructed feature matrices which consist of all aforementioned features for each minute of time t for all the users in each dataset, and converted in to a long vector of features which we fed as the input to the classification models used.1 The class labels were generated as high/low performance at minute t+n based on the 1 In the interest of space and scope of work here, details of these experiments have been omitted, but will be available for discussion at the workshop. above mentioned criteria and threshold and used as the output class labels to be used in the n-step ahead prediction model. If a class label at n-step ahead was correctly predicted based on the features extracted up to time t from the classification model it was considered as correctly classified and if not as misclassified. 4 EXPERIMENTS In order to evaluate whether users who are predicted to perform at low performance in the future based on the current search process, could benefit from this analysis to improve their search process, we conducted some simple simulation analysis.
We considered the individual user search processes as a collection of search paths, where each search path is defined as the search process from the time a user issued a query up to the time user issued another quite different query. This was found out using generalized Levenshtein (edit) distance, which is a commonly used distance metric for measuring the distance between two character sequences. If the Levenshtein (edit) distance between two subsequent queries were greater than 2 (assuming less than 2 was when there were changes in the queries due to simple spelling mistakes or refining of the query), we considered the search process from the former query to the next query as a single search path.
Following this method, we found the first search path of each user and based on the features extracted up to the end of the first search path, and based on the classification model learnt from that corresponding n-step ahead prediction we predicted whether the user is going to have low/high performance at the end of the session. If the user was going to have low performance, then out of the users who predicted to have high performance, we looked at which high performing user has the lowest Levenshtein (edit) distance between the queries issued by low performing user within the first search path and considered it as a pair of users, whom we are going to use in the simulation. Then, for each low performing user and high performing user that was matched, we switched the search process of low performing user at the end of the first search path with the high performing user’s search path up to t=T minutes, where T is the total number of minutes for a session. Then we evaluated by switching the search process early during the overall process whether it would benefit each low performing user to improve their performance. We found that we were able to move most of the underperforming search processes to higher performance by early detection and switching, while keeping the higher performing processes unharmed.
These simulations provide verification that by realizing early during the search process whether a user is going to perform well or not, one could recommend better search processes/strategies for that user which would lead to uplifting the search performance of a previously destined to low performing user. 5 CONCLUSION When it comes to prediction, information retrieval and filtering systems are primarily focused on objects while assessing what and if something could help the users. These approaches are often system-dependent even though the process of information seeking is usually user-specific. Personalization and recommendations are frequently exercised as methods to address user-specific IR and filtering, but still limited to comparing and recommending objects, not focusing on underlying IR processes that are carried out by the searchers. We presented a new approach to address these shortcomings. We began by asking whether we could model a user’s search process based on the actions he/she is performing during an exploratory search task and forecast how well that process will do in the future. This was based on a realization that an information seeker’s search goal/task can be mapped out as a series of actions, and that a sequence of actions or choices the searcher makes, and especially the search path he/she takes, affects how well he/she will do. Thus, in contrast to approaches that measure the goodness of search products (e.g., documents, queries) as a way to evaluate the overall search effectiveness, we measured the likelihood of an existing search process to produce good results.
Here we presented simulations to demonstrate what could happen if one can make process-based predictions, but one could develop an actual recommender system using the proposed method. Another potential application of such prediction-based method would be to use such approach in IR systems to provide the awareness to users how their future performance will be based on the current/past search process. The system could identify that a user will have low performance if, he continues this manner at an early stage of the process, and what could be done to provide suggestions to improve overall performance.
Given that the proposed technique is independent of any specific kind of system, and solely focused on user-based processes, it will presumably be easy to apply it to a variety of IR systems and situations irrespective of retrieval, ranking, or recommendation algorithms. Finally, while we have used datasets borrowed from previous user studies, one could easily apply the proposed method to Web logs, TREC data, and other forms of datasets with various user actions recorded over time. 6 ACKNOWLEDGEMENTS The work reported here is supported by The Institute of Museum and Library Services (IMLS) Cyber Synergy project as well as IMLS grant # RE-04-12-0105-12. The author is also grateful to his PhD students Chathra Hendahewa and Roberto GonzalezIbanez for their valuable contributions to this work. 7 in collaborative IR systems. In Proceedings of the 75th Annual Meeting of the Association for Information Science and
Technology (ASIS&T). Baltimore, MD, USA.
Directly Evaluating the Cognitive Impact of Search User Interfaces: a Two-Pronged Approach with fNIRS Horia A. Maior1,2, Matthew Pike1, Max L. Wilson1, Sarah Sharples3 1Mixed Reality Lab, 2Horizon DTC, 3Human Factors - School of Engineering
University of Nottingham, UK {psxhama,psxmp8,max.wilson,sarah.sharples}@nottingham.ac.uk ABSTRACT Recent research has pointed towards further understanding the cognitive processes involved in interactive information retrieval, with most papers using secondary measures of cognition to do so. Our own research is focused on using direct measures of cognitive workload, using brain sensing techniques with fNIRS. Amongst various brain sensing technologies, fNIRS is most conducive to ecologically valid user studies, as it is less a↵ ected by body movement and can be worn while using a computer at a desk. This paper describes our two pronged approach focusing on a) moving fNIRS research beyond simple psychological tests towards actual interactive IR tasks and b) evaluating real search user interfaces.
Categories and Subject Descriptors H5.2 [Information interfaces and presentation]: Evaluation/methodology, Theory and methods Keywords Functional near-infrared spectroscopy(fNIRS), Brain-computer interface(BCI), Human cognition, Information processing system, Multiple resource model, Limited resource model 1. INTRODUCTION
The cognitive aspects of Information Retrieval (IR) have
repeatedly received focus over time, from Ingwersen’s
Cognitive Model [
Although some prior work (e.g. [
Functional Near Infrared Spectroscopy (fNIRS) is an emerging neuroimaging technique that is non-invasive, portable, inexpensive and suitable for periods of extended monitoring. fNIRS measures the hemodynamic response - the delivery of blood to active neuronal tissues. fNIRS is designed to be placed directly upon a participants scalp, typically targeting the prefrontal cortex. This paper describes our two-pronged approach to using fNIRS to study the cognitive workload created by SUIs, focused on a) task analysis and b) SUI analysis.
RELATED WORK
Understanding the cognitive aspects of interactive
searching (as well as interaction in general) has been a long-standing
goal for researchers in the field of Interactive IR. In the 1970s
Bates suggested that searchers employ both search tactics
and idea tactics [
In the main cognitive evolution of information seeking
research, Ingwersen proposed a cognitive model of IR [
More recently, researchers have focused on objectively
measuring interactive IR phases, in line with Back et al’s work,
Moshfeghi et al measured workload during relevance
assessments by asking people to make judgements while lying in
an fMRI machine. As making relevance judgements can be
performed without directly interacting with a computer, this
made use of an fMRI machine more realistic. Using more
commercialised tools, Anderson [
Using fNIRS, as introduced above, Peck [
RESEARCH PATHS
Pike et al [
Such a technique would allow researchers to analyse data by stage, and find e↵ ective points of comparison during several minutes of continuous measurements. The second path is focused on identifying which aspects of working memory are a↵ ected by di↵ erent features of SUIs, such that researchers can objectively evaluate the e↵ ect of di↵ erent SUI design decisions. A combination of both paths works towards being able to proactively evaluate how SUIs support searchers.
PATH 1: WORKLOAD MODELS
To understand the cognitive aspects of IIR, it is essential to learn about user’s capabilities and limitations in terms of their cognition: how people perceive, think, remember, and process information. This path of research focuses on existing models from Cognitive Psychology and Human Factors, models that conceptualize and highlight aspects that typically describe or influence elements of human cognition.
One important part of cognition during interactive searching involves human memory systems. There are two different types of memory [21]: working memory (sometimes called short-term memory) and long-term memory. Wickens describes working memory as the temporary holding of information that is “active”, while long-term memory involving the unlimited, passive storage of information that is not currently in working memory.
Working memory. Working memory, proposed by
Baddeley and Hitch (1974) [
Information processing system. As humans, we are exposed to large amounts of information via our sensory systems. One of our strengths is in selecting information from our environment, perceiving it, processing it, and creating a response. Therefore we can use this understanding of brain activity to identify which elements of an interactive IR environment need to be considered when measuring brain activity, and how we can reduce rather than increase a user’s mental workload via interface and system design.
Wicken’s Information Processing Model [21] aims to illustrate how elements of the human information processing system such as attention, perception, memory, decision making and response selection interconnect. We are interested in observing how and when these elements interconnect during IR. He describes three di↵ erent ‘stages’ (see STAGES dimension in Figure 2) at which information is transformed: a perception stage, a processing or cognition stage, and a response stage, the first two being processes involved in cognition. The first stage involves perceiving information that is gathered by our senses and provide meaning and interpretation of what is being sensed. The second stage represents the step where we manipulate and “think about” the perceived information. This part of the information processing system takes place in working memory and consists of a wide variety of the mental activities. In relation to IR, it is interesting to observe how elements of cognition, such as rehearsal of information, planning the search strategy and deciding on the search keywords interconnect.
Multiple Resource Model. One model of mental workload that has been widely accepted in Human Factors is Wickens Multiple Resource Model [20] (Figure 2). The elements of this model overlap with the needs and considerations of evaluating complex tasks (such as IR). He describes the aspects of human cognition and the multiple resource theory in four dimensions: • Avoid unnecessary zeros in codes to be remembered; • Encourage regular use of information to increase fre
quency and redundancy; • Encourage verbalization or reproduction of informa
tion that needs to be reproduced in the future; • Carefully design information to be remembered; Resource vs Demands. One other model that is of interest is the limited resource model [22] describing the relationship between the demands of a task, the resources allocated to the task and the impact on performance. • The STAGES dimension refers to the three main stages
of information processing system (Wickens, 2004 [21]). • The MODALITIES dimension indicating that audi
tory and visual perception have di↵ erent sources. • The CODES dimension refers to the types of memory
encodings which can be spatial or verbal. • The VISUAL PROCESSING dimension refers to a nested dimension within visual resources distinguishing between focal vision (reading text) and ambient vision (orientation and movement).
Our aim is to understand how these elements link together and compose more complex components/tasks. Additionally we want to consider how complex tasks (such as a search task) can be divided into primary components according to the models described. This will help identify possible problems in SUI design as well as indicating a possible solution to the problem (suggested implications by Wickens [21]): • Minimize working memory load of the SUI system and
consider working memory limits in instructions; • Provide more visual echoes (cues) of di↵ erent types
during IR (verbal vs spatial);
• Exploit chunking (Miller, 1956 [
The graph from Figure 3 is used to represent the limited resource model. The X-axes represent the resources demanded by the primary task and as we move to the right of the axes, the resources demanded by the primary task increase. The axes on the left indicate the resources being used, but also the maximum available resources point (if we think of working memory that is limited in capacity). The right axes indicate the performance of the primary task (the dotted line on the graph). The key element of this model is the concept of a limited set of resources which, if exceeded, has a negative impact on performance. However, it does not distinguish between resource modality, therefore we propose to use both the limited and multiple resources models to inform our work.
PATH 2: SUI EVALUATION
Relating quantitative data from brain sensing devices into feedback about SUI designs is one of our ultimate goals in conducting this research. SUIs are inherently information rich and thus a↵ ect both visual (results page layout) and verbal (text based results) memory. Detecting a change in either verbal or spatial working memory would help determine if a workload di↵ erence was caused by SUI design (spatial) or the amount of information the design provides (verbal).
Our first in-progress study has stimulated each memory type
in di↵ erent tasks - Verbal memory was tested by performing
an n-back [
Other studies have also looked at each type of memory and
confirmed fNIRS ability to detect changes in heamodynamic
responses accordingly [
In addition to developing an understanding of the extent to which we can monitor di↵ erent memory, our initial study also sought to measure the e↵ ect of artefacts on the fNIRS data. Controlling the environment and human derived sources of noise is a potentially di cult factor to control without e↵ ecting the ecological validity of a study.
Solovey et al [19] showed that fNIRS is relatively resilient to
motion derived artefacts when compared to EEG [
Designing tasks for experiments that measure cognitive effect via a brain sensor require careful consideration in order to ensure that results can be attributed to a cause. Thankfully this problem space has been well explored in the field of Psychology and we are able to adapt the approaches described in the literature to suit our task type requirements.
A primary example of this adaptation is demonstrated by
Peck et al [
Additionally, we are interested in exploring standard search
studies (without following a psychological study layout) and
seeing whether interesting states can be detected. Solovey
et al [
Using a similar approach, we could evaluate a SUI to determine whether a particular change in layout has a positive or negative impact on visual memory. Alternatively, to test the relevance of a results page (which would be dependant on the textual results), we could analyse the e↵ ects on verbal memory between 2 varied results pages, we could then reflect these changes to the Wickens Multiple Resource Model [20]. We are also working towards enabling the interpretation of data within the context of complex multimodal tasks to further extending our knowledge of the processes involved during IR and how they interact and e↵ ect one another. 6. SUMMARY
This paper has aimed to summarise our two-pronged approach towards actually evaluating the design of search user interfaces, in realistic ecologically valid study conditions, using fNIRS technology. The approach first involves braking down interactive IR tasks into how they e↵ ect the di↵ erent elements of working memory, and second understanding how SUIs are processed by di↵ erent parts of working memory. Our two paths of research will build towards a stage where we can combine them and objectively evaluate cognitive workload involved in interactive IR. We believe that this research will provide a novel new direction that SUI’s and indeed HCI in a broader sense can benefit from. The association of physical recordings in ecological valid settings, to an existing theoretical model, provides a new measure from which future SUI development and evaluation could benefit.
Dynamics in Search User Interfaces Marcus Nitsche, Florian Uhde, Stefan Haun and Andreas Nürnberger
Otto von Guericke University, Magdeburg, Germany {marcus.nitsche, stefan.haun, andreas.nuernberger}@ovgu.de,
florian.uhde@st.ovgu.de ABSTRACT Searching the WWW has become an important task in today’s information society. Nevertheless, users will mostly find static search user interfaces (SUIs) with results being only calculated and shown after the user triggers a button. This procedure is against the idea of flow and dynamic development of a natural search process. The main difficulty of good SUI design is to solve the conflict between good usability and presentation of relevant information. Serving a UI for every task and every user group is especially hard because of varying requirements. Dynamic search user interface elements allow the user to manage desired information fluently. They offer the possibility to add individual meta information, like tags, to the search process and enrich it thereby.
Keywords Search User Interface, User Experience, Exploratory Search.
Categories and Subject Descriptors H.3.3 [Information Storage and Retrieval]: Information Search and Retrieval.; H.5.2 [Information Interfaces and Presentation]: User Interfaces.
General Terms
Design, Human Factors, Management.
1. MOTIVATION
Since the launch of the WWW, users accumulated a vast amount of
information. With broadband technologies becoming a part of
everyday life1 the WWW offers a great opportunity in terms of
learning and education. University courses, for instance, are available
online and nearly every topic is handled somewhere in the great
amount of blogs, Q&A pages, fora, web pages or databases. Yet
there is no map, no guide leading through this vast amount of
information. Users need to search for information, to locate the bits
fitting to their specific information need, indexing the amount of
1http://www.internetworldstats.com/images/
world2012pr.gif, 02.05.2013
knowledge available online. Therefore, a proficient tool to
analyse the structure of the web and to provide guidance to specific
sources of information is needed. This task is accomplished by
modern search engines like Google2, Bing3, Yahoo4 and other
local or topic centred search engines. By the increase of
computational power in smart phones and wider access to online resources
the demand for these search tools has risen and the quality of the
search terms has changed. Instead of single-query-searches, users
tend to request complex answers5, trying to learn about topics in
deep. While the need for information and the expectations of users
increased, matching the broader knowledge base contained in the
Internet in the last few years. About 300 Mio. websites were added
in 20116. Search engines mainly remain the same. This leads to the
fact that a “significant design challenge for web search engine
developers is to develop functionality that accommodates the wide
variety of skills and information needs of a diverse user population”
[
According to this concept the result is not a single object. It is a set of sources, representing the learning process for a specific user.
Looking at the current process of information gathering in the
Internet there are only two places. The Internet itself, containing the
pool of existing information, in an unstructured form and a mental
model about the information (space) that is constructed. This
system may work perfect when dealing with short, exact search queries
like postal code New York City, but when it comes to complex
information needs, where the user needs to access a lot of information
and generate more detailed search queries while looming through
pages this system reaches it boundaries. The user might retrieve
only partial facts. For example, if the user needs explanation of a
term used in its initial query. The user is now in need of another
place, where he can store information, reorder it and put it into the
context of other information pieces.
3. STATE OF THE ART
Looking at Google, the most used search engine today [
One way to integrate dynamic elements into existing SUI infrastructure is to build an overlay. Thereby, dynamic UI utilize existing, well known search engines and provide a benefit by enriching them.
This approach is shown in the Boolify8 search engine, which
provides a dynamic drag and drop interface on top of Google’s search
engine. This engine is relatively new and was build to promote the
understanding of boolean queries. Users build a query by
dragging jigsaw like parts onto a search surface. These parts contain
words (general or exact) and linkers like AND and OR. Additional
parts have been added to provide search on a specific page or for
7http://www.google.com/about/company/
history/, 02.05.2013
8http://www.boolify.org/, 02.05.2013
synonyms. By adding and linking those parts the user constructs
a boolean query which will be submitted to the Google search
engine. Boolify was built for children and elderly. Tests in a third
grade technology class showed that children without any
knowledge of boolean queries were able to construct complex queries
just by pulling them together piece by piece9. A similar approach
was implemented at SortFix10. This tool offers the user the
“ability to drag and drop search terms in between several buckets” [
Issues. Comparing the state of the art with the process of
information gathering some issues appear, which may be resolved or at
least damped by using of dynamic elements. While collecting
information pieces for solving complex questions the user discovers
new sources, containing more information. These sources may not
form a linear search process every time. Sometimes there will be a
split and the user needs to decide which trace to follow first. This
issue is also noted in [
Google for example handles every search term as a new
operation. Data is stored, but contains only general information about
the user, queries are not related to each other and therefore
miss9http://ed-tech-axis.blogspot.de/2009/03/
boolified.htm, 02.05.2013
10SortFix.com, offline since 11/2011, Firefox plugin:
https://addons.mozilla.org/en-us/firefox/
addon/sortfix-Extension, 02.05.2013
11http://searchcloud.net/, 02.05.2013
12http://www.searchme.com, offline since 2009
ing its broader context. But when learning about a complex topic
refining the search query is more important to the user. In the
iteration of search processes, to narrow down the mass of information
and to tap new sources, the searcher needs to rewrite and modify
the query, to link it to other related search tasks. Building a
connection between parts of information and evaluating it against each
other is a core principle of learning. This leaves the user targeting
a broader, intense search, in the need to build a custom solution to
extract knowledge and manage it. This is strictly against the
guideline for online interfaces which suggests to “[..] not require users
to remember information from place to place on a Web site” [
By adding dynamic elements internal states can be visualized for
the user to give a better overview about the current position in the
search process. Furthermore it will allow the serialization of search
processes and to step in at every point of the process later on. As
stated in Beyond Box Search “different interfaces (or at least
different forms of interaction) should be available to match different
search goals” and “[t]he interface should facilitate the selection of
appropriate context for the search” [
By giving immediate feedback to the user it is easier to classify the current results. The context of the whole search process will be persistent over multiple search queries and provide a method of accumulation parts of the search process into a single object.
Four features are proposed and explained in this paper, showing a use-case for dynamic search interfaces and giving a suggestion how this can be accomplished. Together these features build up a mid instance to accumulate into a bigger context for a search process.
This clipboard (Fig. 1) reshapes the search process and provide the place to store information between search queries. Instead of trying to accumulate knowledge and information directly the user is able to construct a solution of the search query in this buffer and save it as a complete collection of the information retrieval process.
Reordering. Giving users the opportunity to reorder and therefore to rate a search result is an important step towards dynamics in SUIs. Every result is handled as a single item and can be picked by the user and dropped in another place. The other items reorder fluently, giving user feedback while the user moves on. The SUI holds an array of parameters, which is used to evaluate every item.
Possible criteria are Accuracy, Clarity, Currency and Source
Novelty. These and more criteria are mentioned and explained in [
When a user reorders items to fit his preferences the search engine
may use the information provided by this ranking to weight the
existing parameters to yield better results in the future. The engine
will be able to present results ranked according to the user’s
preference. This can be done for all users and also search process wide, as
some search tasks require documents and papers while others may
focus on web pages or media. This addition to classical user
interfaces can make great use of the up-trend for touch based devices, in
2012 89% of mobile phones and smart-books support touch [
Designing the SUI responsive to touch and gesture is maybe one of the most natural solutions for human computer interaction and adds an amount of possible actions based on gestures.
Workbench. The workbench targets the issue of loosing
information while switching between different searches. It adds a third
place to the proposed search process, located outside of the search
scope but still related to it. The user may drop queries here to keep
them throughout the whole search process. When entering a query,
indicators show how relevant items on the bench are. This allows
the user to classify new results in terms of integrity towards already
selected snippets. The workbench acts as a buffer between search
queries, adding a broader context to every entry. Like a frame, it
contains information exclusively attached to the current search
process, leading to the possibility of customization and user centred
search environments. When the user switches between queries he
can immediately determine how well the new results fit into already
selected items. This allows identifying false positive as well as
exploratory search [
Tag Cloud. The tag cloud is another feature to guide the user in the
search process. As shown in [
When entering a new search query the tag cloud displays the relevant tags and reorders the cloud to revolve around the current tags.
By combining distance and size of the entered tag with their direct neighbours the user can directly spot how homogeneous its current query is in terms of the whole process. The tag cloud can also use the existing tags to show the user other closely related tags and suggest query refinement based on tag proximity. Colours can indicate the state a tag is currently in. A possible color scheme for western culture can be based on the three colors used in traffic lights. The concept of three-coloured traffic lights also work for color-blind people, since they do have a given position. Therefore, we also use second coding paradigm: form. A green triangle is proposed for tags resulting from the current query, which are contained in the overall tag cloud spanned by the workbench. An orange circle indicates a warning for tags, either in the current query result or the bench, which are not related to the rest of the cloud. A red square is avoided for the reason that uncontained tags may not be bad, they can lead to a new direction or add a reasonable value to the whole search process. The tags are scaled depending on their frequency.
When the user selects any item from the bench or the search result the corresponding tags are centred. The other tags are located based on their coherence with the selected tags; closer means the tag is in a direct relation to the selected item. A user can quickly check the integrity of his search process by looking at the tag cloud.
A slim, packed cloud means the results are all related to each other, an open, wide cloud indicates a broad result field, covering many aspects. False positives may be filtered out, when enough items exist, as they stick out the rest of the cloud.
Search Map Support. The search map (Fig. 2) acts as a
representation of the whole search process, by storing every query and
following up querying and visualize it in a chronological order. The user
may select single nodes in the map to get into the state of search
process at this moment and refine it. The map provides a kind of top
view to the path of the search and shows where the user branched
out into new queries. It allows the user to cut off nodes and whole
branches if they are not needed any more to fulfil the need for
information. As it contains every action and some data in the current
search process, the search map might be serialized and stored to
retrieve the search process later on. With this map at hand a user can
save whole search tasks just like he saves favourite web pages. He
can step back into the process at any time and reconstruct the whole
learning process or correct parts of the search which has proven to
be not correct. This kind of Story Telling helps to visualize the
given data, “[...] lead to findings, which prompt actions [...] [and]
can indicate the need to forage for new data.” [
While not abolishing this the search map provides a visual representation to simulate parallelism. The map also allows scoping of the analysis by creating a horizontal or vertical bound. Only tags and items inside this bound will be considered, the rest is greyed out. This allows the user to dig deep into a certain topic (small vertical bounds) or create a better understanding of a certain term and add more results to a certain query (horizontal boundary). This can help the user to concentrate on smaller pieces of a big search process and to narrow down problems one by one. 5. CONCLUSION This paper has shown certain design flaws of today’s search engines and some proposed dynamic design principles to counter them. The application of the envisioned elements can extend a search engine towards a software capable of complex research tasks. With the current up-trend of online learning this unlock a new way of using them. The surplus resides not only in the dynamic and vivid interface, it prepares a whole new tier of online search solutions. The process of learning can be preserved and shared with others. One can come back at any time, jump right into the saved search process and reconstruct the development of certain knowledge. With this tool chain at hand learning becomes a social and an integrative part of the WWW. The next step in deploying dynamic elements into search user interfaces would be prototyping them. Design snippets need to be tested for usability and acceptance in the real world.
Starting as overlays and additional feature of existing search engines may develop and emerge into independent solutions.
Acknowledgement Part of the work is funded by the German Ministry of Education and Science (BMBF) within the ViERforES II project (01IM10002B).
SearchPanel: A browser extension
for managing search activity
Simon Tretter
University of Amsterdam Amsterdam, The Netherlands
Gene Golovchinsky FX Palo Alto Laboratory, Inc.
3174 Porter Drive
Palo Alto, CA gene@fxpal.com
Pernilla Qvarfordt FX Palo Alto Laboratory, Inc.
3174 Porter Drive
Palo Alto, CA pernilla@fxpal.com ABSTRACT People often use more than one query when searching for information; they also revisit search results to re-find information. These tasks are not well-supported by search interfaces and web browsers. We designed and built a Chrome browser extension that helps people manage their ongoing information seeking. The extension combines document and process metadata into an interactive representation of the retrieved documents that can be used for sense-making, for navigation, and for re-finding documents. 1. INTRODUCTION
Broder et al. [
From a theoretical perspective, it is not clear whether refinding is a di↵ erent kind of search activity or an orthogonal dimensions. Regardless, while major web search engines o↵ er simple and e cient interfaces for navigational and transactional searches, relatively little support is available for more complex informational search or re-finding.
These seemingly neglected activities are not unimportant,
however: Teevan et al. [
We created a web browser extension that enriches common web search engine interfaces and addresses important deficits with respect to open-ended (exploratory) search and re-finding. Our extension visualizes search results to help users find the right document or documents by visualizing metadata of the retrieved pages.
Following Golovchinsky et al. [
The browser plugin enhances the searcher’s ability to use process metadata to understand their search results and to plan subsequent activity by displaying surrogates for the current set of retrieved documents. We represent prior retrieval state, whether a document was opened, and whether it was bookmarked in an integrated overview that appears at the side of the browser window. We also make it possible for searchers to examine multiple documents without returning to the search results or using multiple tabs.
The remainder of this paper is organized as follows: we review the relevant related work, describe the browser extension, and conclude with a discussion of the design space.
RELATED WORK
There are two broad categories of related work: the management of search history and the representation of search results. Refinding has received increasing attention recently.
While the browser implements some history mechanisms,
these are typically not well-suited to users’ needs [
Information overload due to large numbers of results is
a common problem in information seeking [
One potential issue with the systems above is that the overall organization of the interface itself may induce usability problems. Complex interfaces allow more individual settings to be specified by a user, but simple interfaces allow a broader spectrum of users to use them. This tradeo↵ is not trivial to handle, and as we see nowadays, most Web search interfaces tend to be quite simple.
Supporting the searcher’s decision making process can be
crucial for e↵ ective search performance for complex
information needs. This support can take the form of enhanced
surrogates for documents. One type of information often
used for this purpose is document metadata (author, date,
images of the document, etc.). Even et al. [
Spoerri [
To compensate for the deficiencies of SERPs we created a browser extension called SearchPanel. This extension combines document and process metadata in a visual representation of search results to help people manage their information seeking. We chose the browser extension approach rather than creating a proxy for several reasons. While both o↵ er the potential of parsing and augmenting SERP and document pages, a browser extension has some advantages.
It scales better with respect to storing user history data. It ensures a higher level of data privacy, since data that might potentially reveal user interests (e.g., query keywords, selected URLs, etc.) can be logged as hashed values. Finally, it has access to bookmarks and local browsing history. 3.1
Design space
When performing search tasks, searchers may need di↵ erent kinds of information to support their information seeking. We represent the design space as consisting of three categories of activities: search activity, navigation activity, and organization activity.
Historically, web UI support for the search process, or search activity, has been focused on query formulation and understanding the current query. Web browsers o↵ er limited support for comparing current results set with earlier activity by marking the visited status of documents.
When engaged with a search task, users need to shift their attention between the SERP and the retrieved pages. In some cases, the searcher does not find the desired information in a retrieved document, but rather in links to other documents containing relevant information. This navigation activity can be an important part of the information seeking process.
When searchers find useful web pages, they may wish to save those documents for future access. More specialized search engines sometimes support this capability directly, but it is most often supported only by the browser’s bookmarking capability.
We can consider these search and sense-making activities in light of the kinds of information required to satisfy them.
In particular, Table 1 shows when document and process metadata might be pertinent for the di↵ erent categories of search activities. A representation of the number of visits to a retrieved result (process metadata) could be used by a searcher to decide how to interact with that result. In a refinding sub task, for example, searchers might want to ignore newly-found documents or pages that were not opened.
The purpose of the search panel is to complement the SERP and to be available when exploring search results; we wanted the design to be simple and unobtrusive but still convey useful information. Some features (e.g., organizating bookmarks) listed in Table 1 are too complex to be integrated into the extension. Others, such as favicons, while seemingly trivial, may still provide useful information for navigating search results.
SearchPanel displays automatically on the right side of the browser window when it is enabled (Figure 1). The right side of the content page has been chosen because this location is frequently free of document content. In cases of overlap, its vertical position can be adjusted manually to accommodate page content that may be occluded.
SearchPanel displays immediately after a search has been performed on a supported web search engine (currently, they are Google, Google Scholar, Yahoo, Bing and Microsoft Academic Search). SearchPanel remains visible even if the searcher follows links from retrieved documents. In addition, searchers can return directly to the original query, or re-run it on a di↵ erent search engine.
A short tutorial page is displayed at installation, and can also be reached through the option menu. This page also allows logging (see 3.2.4) to be disabled, and can be used to delete the recorded history. 1The Wayback Machine is a service that provides access to archived and historical versions of web sites.
SearchPanel displays several kinds of document metadata.
Documents are represented by bars arranged in order correa newly-found page; 3 favicon representing the site from which the page was retrieved; 4 bar representing page that has been visited; 5 highlighted bar based on cursor position; 6 bookmark indicator; 7 currently-selected page. sponding to the retrieved list; clicking on a bar is equivalent to clicking on a link on the SERP. Almost all websites have icons (favicons) to help re-identify the web page quickly; these icons are shown to the right of the bar (see Figure 1, item 3 ). A tooltip with the title of the document is added to each bar as well. We considered identifying other metadata such as document MIME type, but that would incur the overhead of a separate HTTP request for each document.
At least initially, we chose not to pursue this strategy. 3.2.2
Process metadata
Process metadata is also incorporated into SearchPanel.
First, the icon of the search engine that ran the search is highlighted in the top bar (item 1 ). Other icons represent available comparable search engines. Clicking on one of these icons re-runs the query with the selected search engine. Search engines are grouped into two categories (web search and academic research) and only the relevant ones are shown. The current selection (highlighted with a black border) links back to the search result page if the user navigates to one of the retrieved documents.
Each bar can have one of three di↵ erent colors, depending on the link history. If a link has never been retrieved before, the state of the link is ”new” and the color will be teal. Results that have been retrieved by prior queries but have not been clicked on are colored blue. Visited links are colored violet. The local browser history is examined to retrieve the link status. This allows us to incorporate page views that occurred before SearchPanel was installed.
Each bar’s length reflects the frequency of retrieval of the corresponding page. The more frequently a page has been retrieved, the shorter the bar gets (item 3 ). The retrieval history is stored locally in the browser for privacy reasons and can be deleted through SearchPanel’s option page.
In SearchPanel, the bookmarking function serves two purposes (item 6 in Figure 1). First, searchers can click on the star to bookmark the corresponding page. Second, previously bookmarked documents in the SERP will show a yellow star next to them. This allows to re-find a web page quicker, as the user does not need to navigate to a document to know if they have previously bookmarked it.
The selection indicator (see item 7 in Figure 1) indicates the currently-selected result page. If a link on a result page is clicked, the page indicator will stay on the last retrieved document page to indicate that navigation started with it.
Hovering over the result highlights the associated bar (item 5 ), and also highlights the corresponding snippet in the SERP (Figure 2); the SERP is scrolled as necessary to bring highlighted snippet into view. Conversely, when the mouse is over a snippet on the SERP, the related bar jiggles leftright to reinforce the connection between the two.
When the user navigates o↵ the SERP to a search result, SearchPanel remains active. Clicking on bars navigates among the retrieved documents, bypassing the intermediate step of reloading the search results. When the mouse is over a bar in SearchPanel, the SERP snippet of that result will be shown. This can be seen in Figure 3, where a preview of the Wolfram Alpha snippet is shown. If the snippet is not available, a tooltip with the document title is shown instead. Both of these features should make it easier and more e cient to navigate the search results without necessarily creating a large number of tabs in the process.
The extension was created to study people’s information seeking behaviors. The goal of the project is to understand how people use the web when looking for information to improve their search experience. Therefore logging of user activity was necessary. To encapsulate it from the basic functionality it was designed as plugin that could be connected or disconnected from SearchPanel. It collects information related to the use of SearchPanel for the purposes of statistical analysis of patterns of behavior.
To maximize searchers’ privacy, no personally-identifying information is saved. Queries and found URLs are recorded as MD5-hashed values only. This allows us to identify recurring queries and documents, without being able to read the content of the query or to observe which pages people view. Specifically, the following information is recorded: • The IP address and the time the event was logged • When a search result was clicked and where this hap
pened (SearchPanel or SERP) • Hash strings that represent the queries and found web
pages. • Time spent with the mouse on di↵ erent interface parts
(SearchPanel vs SERP) • Various actions related to the extension (adding book
marks by clicking the start, moving it, etc.).
After an in-house pilot deployment, SearchPanel has been made available through the Google Chrome store. The goal of the deployment is to understand whether the extension helps people with their search tasks, and to assess the relative utility of document vs. process metadata. We also expect to collect a dataset that characterizes people’s browsing and searching behaviors in terms of patterns of retrieval and re-retrieval, search result navigation, etc.
Web search engines are used for many di↵ erent kinds of search tasks. While navigational and transactional uses of search engines are well-supported by current interfaces and algorithms, searchers are left to their own devices for more open-ended information seeking and re-finding. We created a Google Chrome browser extension to help people manage their search activity. We explored the design space of document and process metadata related to the wide range of activities searchers may engage in during information seeking. The extension keeps track of retrieval, page visits, and bookmarking, and integrates traces of these activities with document metadata to give people a more complete impression of their search activity. An upcoming deployment will explore the e↵ ect that this extension has on how people interact with search results. M. Atif Qureshi*†!, Arjumand Younus*†!, Colm O’Riordan*, Gabriella Pasi!, Nasir
Touheed† *Computational Intelligence Research Group, Information Technology, National University of Ireland,
Galway, Ireland !Information Retrieval Lab, Informatics, Systems and Communication, University of Milan Bicocca,
Milan, Italy †Web Science Research Group, Faculty of Computer Science, Institute of Business Administration,
Karachi, Pakistan muhammad.qureshi,arjumand.younus@nuigalway.ie, colm.oriordan@nuigalway.ie, pasi@disco.unimib.it, ntouheed@iba.edu.pk Traditional search engines fail to capture the notion of “perspective” in their search results and at times present the results skewed towards a particular topic. Under most of these cases even query reformulation fails to retrieve desired search results and the underlying reason for such failure is often the bias within the document collection itself (e.g., news articles). A perspective-aware search interface enabling users to look into search results for some “perspective” terms may be of great use for certain information needs. In this paper we describe such a system.
Categories and Subject Descriptors H.1.2 [User/Machine Systems]: Human factors; H.3.3 [Information Search and Retrieval]: Search process General Terms Human Factors, Performance Keywords Perspective, Wikipedia, Bias 1. INTRODUCTION AND RELATED WORK
It is often the case that when using a search engine for
information seeking users have an underlying intent [
Presented at EuroHCIR2013. Copyright !c 2013 for the individual papers by the papers’ authors. Copying permitted only for private and academic purposes. This volume is published and copyrighted by its editors.. terrorism in most of the cases. This prompts the user to explicitly evaluate how much Islam is related to terrorism in the returned search results. • Consider the case of a user who wishes to find out about roles and rights of women in Islam but the search engine returns articles that contain a high amount of terms highlighting oppression against women instead of women rights and roles. In this case the user is prompted to check the correlation between women and oppression within the search results that have been returned.
Note that the perspective given by most search results (Islam in our motivating example (1) and oppression in our motivating example (2)) may or may not be aligned with the user’s query intent. In case of search results not being aligned with his/her query intent he/she may be interested in observing the amount of perspective tendencies in various news reports.
This paper proposes the concept of a “perspective-aware”
search interface that enables the user to explicitly analyse
search results for information from a particular
perspective with respect to an issued query. To the best of our
knowledge, previous research within Human-Computer
Interaction and Information Retrieval has failed to capture
the notion of “perspective” within the information retrieval
process. Early research related to Interactive Information
Retrieval by Belkin [
Recently the information retrieval community has turned
attention to diversification of search results which aims to
tackle the issue of query ambiguity on the user side [
This section presents the essential details of the proposed
perspective-aware search interface along with the underlying
implementation details. We keep the interface as simple as
possible on account of research suggesting users’ reluctance
in switching from a simple search form [
The underlying perspective detection algorithm makes use of the encyclopedic structure in Wikipedia; more specifically the knowledge encoded in Wikipedia’s graph structure is utilized for the discovery of various perspectives in documents returned by the search engine. Wikipedia is organized into categories in a taxonomy-like3 structure (see Figure 2).
Each Wikipedia category can have an arbitrary number of
subcategories as well as being mentioned inside an arbitrary
number of supercategories (e.g., category C4 in Figure 1 is
a subcategory of C2 and C3, and a supercategory of C5, C6
and C7.) Furthermore, in Wikipedia each article can belong
to an arbitrary number of categories, where each category is
a kind of semantic tag for that article [
The underlying perspective detection algorithm within our system requires the perspective term/phrase to match the title of a Wikipedia article. This may seem to impose a cognitive load on the user at search time. However, this is not the case: as shown in Figure 3 the entered text automatically turns green when a certain user-specified perspective term matches the title of a Wikipedia article, and symmetrically the entered text automatically turns red in case of a mismatch.
Once the perspective term is entered correctly the system fetches the Wikipedia article corresponding to the perspective term referred to as Seed Perspective Article (PAseed) along with the categories to which it belongs and we use 3We say taxonomy-like because it is not strictly hierarchical due to the presence of cycles in the Wikipedia category graph.
DISCUSSION “terrorism” is shown in Figure 4. As evident from the top search result, there is a high perspective of terrorism within the returned document and perspective terms that our algorithm fetches are as follows: a) the war on terrorism, b) ayman al zawahiri, and c) osama bin laden.
PC04 to refer to these categories. After fetching of Wikipedia categories in PC0, the system retrieves sub-categories of PC0 until depth 2 i.e., PC1 and PC25 and collectively these categories related to PAseed are referred to as PC (where PC is union of PC0, PC1 and PC2.). Next, the set of all articles within the Wikipedia category set PC is retrieved and we refer to this set as Expanded Perspective Article Set (PAexpanded). The system then retrieves all categories associated with the set PAexpanded which we refer to as WC ; note that PC is a subset of WC. Finally, the intersection between PC and WC is retrieved which is a set of categories representative of the domain of the perspective term originally input by the user, we refer to this set of representative categories as RC.
After building the Wikipedia category sets as defined above6 i.e., PC, RC and WC we match variable-length n-grams within a document with articles in the set PAexpanded, and we check for cardinality of RC and WC. The cardinality scores along with n-gram frequencies are used to compute a perspective score for each document.
There have been many efforts in the information retrieval research to present to users information regarding the relationship between the query and the answer set and the query and document collection. Capturing this information during the retrieval process provides the user with much valuable information (e.g. whether a term is overly specific, or whether a term is ambiguous etc.). Various attempts have been made to tackle this problem, ranging from the definition of snippets to the definition of approaches to cluster search results (Clusty.com), to the presentation of diversified search results in the first position of the ranked list offered to the users.
Recently there has been a resurgence of interest in defining visualization techniques of search results that offer an effec2.2 Search Results Presentation tive and more informative alternative to usual and scarcely informative ranked lists. Pioneer visualization systems are
The perspective scores computed in section 2.1 are dis- represented by Tilebar [
Low, and d) Neutral. Moreover, in case of documents with This additional information can help the user in their high, medium and low scores we also report the top-scoring search task (e.g. allowing them to navigate the collection perspective terms that were extracted using the Wikipedia more easily or providing evidence to allow the user to reforgraph structure as explained previously. A sample search mulate their query more efficiently). corresponding to search query “india pakistan relations” and Our proposed system, although related in that we also attempt to give the user an insight into the answer set and its 4These are basically perspective categories at depth zero. relation to the query, differs in a fundamental manner. Our t5wToh.ese are basically perspective categories at depth one and system, we posit, allows the user to gain insight into the an6The set building phase is performed through a cus- swer set and its relation to the query, but moreover, allows tom Wikipedia API that has pre-indexed Wikipedia to the user to gain an insight into a perspective inherent in data and hence, it is computationally fast. For details the answer set. Our system uses an external and collectively http://www3.it.nuigalway.ie/cirg/prj/WikiMadeEasy.html created knowledge resource (which is less likely to be biased in a given direction) to obtain extra terms to represent the perspective of interest to the user. This knowledge (perspective term and related terms) does not modify the query (as would an additional query term), but is instead used to highlight the presence of a perspective in the answer set.
In this paper we have proposed a novel approach for capturing the relationship between a user’s query and the returned answer set. We do not rely on evidence in the document collection or the query stream, but rather instead extract terms from an external source of evidence to help users quickly see the presence of a particular perspective in the document collection and answer set.
Having built the system and undertaken preliminary user evaluations7, we aim at undertaking a complete and systematic review of the approach. This will comprise a number of separate user evaluation tasks. The initial experiments will involve comparing our search approach with and without the perspective-aware component over a number of tasks to see if the additional context and information provided by our perspective aware system aids the users in a range of information-seeking tasks. Our second planned experiments will be focussed on persons seeking information from newspaper articles, a domain wherein a degree of bias often exists.
We wish to explore the users’ experience with regards to any perceived bias in the considered corpora.
REFERENCES