EuroHCIR2012,was,the,second,workshop,in,the,Europea n,series,focusing,on,the, combined,aspects,of,Human=Computer,Interaction,and,Information,Retrieval, (HCIR).,The,MUMIA,WG3,supported,event,saw,significant,growth,in,interest, from,the,first,year,,attracting,over,30,submissions.,9,key,research,and,position, papers,were,accepted,for,Oral,Presentation,,while,a,further,13,demos,and, posters,were,accepted.,All,are,included,in,these,proceedings,so,that,they,can,be, accessed,by,those,unable,to,attend,the,event,in,Nijmegen. ,
Proceedings+of+the+! nd
Supported!as!a!MUMIA!WG3!event!
Page"3">"" Using"Card"Sorts"to"Understand"how"Users"Think"of"Personal"Information"
Paul%Thomas%and%David%Elsweiler" Page"7">"" Using"Semantic"Differentials"for"anE"valuative"View"of"theS"earch"Engine"as"an" Interactive"System"
Frances%Johnson" Page"11">"" The"Fault,"Dear"Researchers,"is"not"in"Cranfiedl,"But"in"ourM"etrics,"that"they" are"Unrealistic."
Mark%D.%Smucker%and%Charles%L.%A.%Clarke% Page"13">"" A"Model"of"Consumer"Search"Behaviou"r
Tony%Russell@Rose%and%Stephann%Makri% Page"17">"" Revisiting"User"Information"Needs"in"Aggregated"Sear"ch
Shanu%Sushmita,%Martin%Halvey,%Robert%Villa%and%Mounia%Lalmas% Page"21">"" Improving"Search"Experience"on"Distributed"Leisure"Events"
Richard%Schaller,%Morgan%Harvey%and%David%Elsweiler% Page"25">"" Opinion"Mapping:"Information"Visualization"Approaches"for"Comparative" Sentiment"Analysis"
William%Hsu%and%Praveen%Koduru% Page"29">"" Search"System"Functions"for"Supporting"Search"Mod"es
Thomas%Beckers%and%Norbert%Fuhr% Page"33">"" Ingredients"for"a"User"Interface"to"Support"Media"Studies"Researchers"in"Data" Collection"
Marc%Bron,%Frank%Nack,%Maarten%De%Rijke%and%Jasmijn%Van%Gorp"
Page"37">"" Exploring"Italian"Wine:""aCase"Study"of"Aesthetics"and"Interaction"in"a"
Generative"Information"Visualization"Method"
Luca%Buriano"" Page"41">"" From"Task>based"Evaluation"to"Featur>ebased"Evaluation"in"Personal"Searc"h
Seyedeh%Sargol%Sadeghi,%Mark%Sanderson%and%Falk%Scholer"" Page"43">"" Visualization"of"Clandestine"Labs"from"Seizure"Reports:"Thematic"Mapping"and" Data"Mining"Research"Directions"
William%Hsu,%Mohammed%Abduljabbar,%Ryuichi%Osuga,%Max%Lu%and%Wesam%Elshamy"" Page"47">"" Towards"Detecting"Wikipedia"Task"Context"s
Hanna%Knäusl,%David%Elsweiler%and%Bernd%Ludwig"" Page"51">" CUES:"Cognitive"Usability"Evaluation"Syste m"
Matthew%Pike,%Max%L.%Wilson,%Anna%Divoli%and%Alyona%Medelyan"" Page"55">"" Collaborative"Environment"of"the"PROMISEIn"frastructure:"an""ELEGantt""
Approach" Marco%Angelini,%Claudio%Bartolini,%Gregorio%Convertino,%Guido%Granato,%Preben%Hansen% and%Giuseppe%Santucci" Page"59">"" Search"User"Interface"Design"for"Children:"Challenges"andl"uStoions"
Tatiana%Gossen,%Marcus%Nitsche%and%Andreas%Nuernberger"" Page"63">" EyeGrab:"A"Gaze>based"Game"with"a"Purpose"to"Enrich"Image"Context"
Information"
Tina%Walber,%Chantal%Neuhaus%and%Ansgar%Scherp"" Page"67">"" Using"Wordclouds"to"Navigate"and"Summarize"Titwter"Search"Results"
Rianne%Kaptein"" Page"71">"" Do"Users"Benefit"from"Controlled"Vocabularies"in"Search"Interfaces?"
Ying@Hsang%Liu,%Paul%Thomas,%Jan@Felix%Schmakeit%and%Tom%Gedeon"" Page"75">"" User>Centred"Design"to"Support"Exploration"and"Path"Creoanti"in"Cultural" Heritage"Collections" Paula%Goodale,%Paul%Clough,%Nigel%Ford,%Mark%Hall,%Mark%Stevenson,%Samuel%Fernando,%
Nikolaos%Aletras,%Kate%Fernie,%Phil%Archer%and%Andrea%De%Polo% Page"79">"" Supporting"Serendipitous"and"Focused"Searc"h
Junte%Zhang"" Page"83">"" Vague"Query"Formulation"by"Desig"n
Marcus%Nitsche%and%Andreas%Nuernberger%
Using card sorts to understand how users think of personal information
Paul Thomas
CSIRO, Canberra paul.thomas@csiro.au
David Elsweiler University of Regensburg david@elsweiler.co.uk ABSTRACT Understanding how users think of personal information, and how they mentally categorise or classify the objects they work with, should inform the design of personal information management (PIM) or personal retrieval systems. However, most investigations of this topic predate widespread multimedia, websites, and social media—objects that a contemporary PIM or retrieval system should work with.
We describe a pilot study that has used a variant of card sorts to elicit categories for personal information such as files, email, tweets, and websites. Our early results suggest that there are common categorisations which are not yet supported by PIM software, but which might reward further work. Our results also suggest that—with some caveats— card sorts are useful for understanding users’ categories.
Categories and Subject Descriptors: H.5.2 [Information Interfaces and Presentation]: User Interfaces General Terms: Human Factors Keywords: Facets, classification, card sorts 1. INTRODUCTION
Tools for personal information management (PIM) and
search support the archival, retrieval, and management of
“personal” data: the files, email, photos, videos, and other
digital objects a person creates or uses [
We are interested in how users think of the wide range of digital objects they interact with—the objects conventionally considered by PIM tools, objects less commonly considered such as websites and applications, and newer objects such as messages from social media. There are three linked questions: 1. What properties do users think “personal” digital objects have? That is, in which ways do users think of the objects they use? Presented at EuroHCIR2012. Copyright © 2012 for the individual papers by the papers’ authors. Copying permitted only for private and academic purposes. This volume is published and copyrighted by its editors. 2. Can we expose these properties in a PIM or search tool?
Can the properties of an object be determined algorithmically? How should the properties be presented? 3. Assuming we can expose some or all of these properties, would we expect that to make management or retrieval easier?
At present, we are considering the first question. In particular, in this work, we have experimented with card sorts to elicit users’ own descriptions of personal information. 2.
CLASSIFICATION AND TOOLS
Past work has investigated the properties users assign to files, and elicited categorisation schemes. This work has not however considered as wide a range of object types as we do here; we may expect that with di↵ erent types, sources, and quantities we would see di↵ erent categorisations. Existing PIM and file management systems also support, or impose, particular faceting schemes. 2.1
Classifications
There is a rich tradition, in information science and information behaviour, of studies that try to understand how people organise, classify, and think about their information— that is, how people understand their information independent of any particular software capabilities or restrictions.
Three studies of note are by Cole, Kwasnik and Case.
Cole [
Other research relating to our work has tried to learn about
how people think about digital information by investigating
how they behave with information in practice. For example,
people have been shown to organise email messages and files
based on projects [
More recently, Gonc¸alves and Jorge [
It was discovered that time, location, and purpose of the document were the most common attributes used in stories.
Similarly, Blanc-Brude and Scapin [
Our aim here is to add to and complement this previous work by using a technique that can deal with rich variety of information objects we interact with today; be tool agnostic; and allow insight into how di↵ erent documents can be associated in di↵ erent ways. We would also like to understand whether this has any impact on the design of PIM tools. 2.2
Tools
Tools for desktop search typically expose not just
filesystem attributes such as name, size, and timestamp, but also
extracted metadata. For example, Phlat [
More elaborate PIM tools have exposed other attributes to
support di↵ erent interactions. Some, such as Lifestreams [
Here, objects are organised according to tags for the
context in which they are used [
These systems o↵ er variety of projections, across a number of media and storage types, but it is hard to know whether these match the way people naturally think of their objects.
Alternative presentations that were natural for users, and easy to implement, would be worth further thought.
METHOD
In this work, we have experimented with repeated singlecriterion card sorts to elicit users’ mental categorisations.
Cards represented digital objects on each participant’s computer. 3.1
Card sorts
Repeated single-criterion card sorts—or just “card sorts”— are a common technique for eliciting users’ categorisations (see e.g. Rugg and McGeorge [22] for an overview). Compared with interview-based techniques, card sorts are less flexible but are very lightweight: in our experience participants grasped the idea very quickly, many found it enjoyable, and the entire protocol took little time. Coding card sorts for later analysis is also relatively straightforward.
In a typical exercise, each participant is given a number of cards, each representing an object or a concept. They are asked to partition these cards according to any criteria they like; the criteria used for the sort, the categories (piles or sets), and the cards in each category are recorded. This is repeated several times, with participants suggesting a di↵ erent criteria each time. For example, given cards labelled as follows:
1. pig; 2. chicken; 3. snake; 4. horse; 5. spider a participant may sort cards according to the criterion “raised on a farm”, with cards 1, 2, and 4 in category “yes” and cards 3 and 5 in category “no”. A second sort, according to the criterion “where eaten”, might have cards 1 and 2 in category “almost everywhere”, card 3 in category “Asia”, card 4 in category “Asia and Europe” and card 5 in category “don’t know”.
Records of the sorts may then be analysed with qualitative or quantitative methods. 3.2
Our approach
For this early experiment, we recruited a convenience sample of ten participants from two institutions. All were heavy computer users.
As preparation for the experiment participants were asked to select several information objects they had seen, used or created in the recent past. An “information object” was defined by giving as examples computer files, emails, websites, tweets or Facebook updates, documents or articles read, photographs or images, videos, music, and computer applications. However, participants were not restricted to these objects and could choose anything they wanted using these as a guideline. We encouraged participants to label 10 to 15 cards, which we believe balances the need for broad coverage with practical limits on participants’ time.
While choosing objects, participants were asked to create index cards with the name of each (or some other reminder of its identity or contents).
With cards made, each participant was introduced to sorts using a set of cards with pictures of buildings; they were taken through some example sorts which included criteria clear from the pictures themselves (colour, material), criteria which were not immediately clear (insulation), criteria which were subjective (good place for a party). “Can’t tell” or “don’t know” categories were included in these examples.
The participants’ own cards were then used for repeated sorts. Participants were asked to make piles according to a criterion of their choosing, and we noted the criterion (sometimes this was implicit), the categories used, and the cards in each category.
After collecting individual classifications from all
participants, the full dataset, i.e. the criteria used to associate
information objects, was analysed qualitatively using an
a nity diagramming technique. This is a group-based
process, which allows the discovery and validation of patterns in
the data [
RESULTS OF THE SORTS
The results from this pilot are promising. Card sorts elicited a variety of criteria; some of these di↵ er from those seen before, and many are not well supported by PIM or retrieval tools. document lifecycle events object’s form object’s a↵ ective qualities object’s cost and value object static/dynamic distinction people and community properties of associated tasks topics covered work/leisure distinction (three other object-related groups)
Participants
General observations
Our ten participants provided 64 sorts, a median 5.5 sorts each (first/third quartile 5.0/8.0 sorts each). At the leaves of our hierarchy, there was in general little overlap: 13 of 25 criteria were used by only one participant. However, 12 were used by two or more participants, 8 by three or more, and one criteria (discussed below) was used by every participant for at least one sort. Figure 1 summarises how many participants were represented in each top-level group, that is each group at the top of our hierarchical coding.
The single most common criterion was a distinction between objects used for work and objects used for leisure—all our participants used this criteria, and typically early on.
Following this there were four common groups: to do with the object itself, especially the form (data type and other surface features), which 7 participants used at least once; the topics an object is connected with (5 participants); the properties of tasks associated with an object (4); and criteria describing people and community (4). A striking finding is the diversity in the criteria derived by the participants.
Although we were able to group the criteria into 13 distinct high-level cateorgies, only two of these, work/leisure and form, were named by more than half the population.
After our initial grouping, which was based on labels proffered by participants and not on any statistics of the sorts themselves, there were no clear correlations between criteria— that is, “work” did not look the same as “important” or “Word files” as “makes me angry”.
On our analysis, five of Kwa´snik’s seven groups were represented in our data: situation, document, time, value, and cognitive state. However, they were very unevenly distributed: criteria we classified as “situation” were used by all ten participants, for one or two sorts each; “document” was used by nine participants, for a median 3.5 sorts each; while at the other end of the scale, “time” was used by only three participants and “cognitive state” was used by two participants, once each in each case. 4.2
Criteria, groups, and tool support
In many previous studies two groups of criteria—form and topic—were found to be central, and our data reinforces this.
There are also, however, notable contrasts.
Work/leisure. To the best of our knowledge, previous studies of classification behaviour have not found a work/leisure distinction. However, every participant in our sample used this criteria. This may be because we most of the objects in our study were digital, not paper documents—it is very easy to mingle work- and leisure-related objects online—but it is clearly important and is not explicitly supported by PIM tools.
One participant reported that he used two top-level folders in his file system, and two email accounts, to keep work and leisure information separate. No other participants reported as clear a distinction, however. It should be possible in a PIM/search tool to tag files, or e.g. learn a classifier, to help maintain this distinction. Distinguishing work from leisure contexts might also allow di↵ erent technologies to be used in each case.
People. Our participants did associate their objects in terms of specific people, but not in the way we might have expected.
Rather than linking objects to particular, specified people, our dimensions relate to relationships with the community: “popular with many people”, for example, “things I will/won’t talk about”, or “involvement of other people”. Unfortunately it is not clear how a PIM or search tool could support this sort of classification.
Task, time, and workflow. Users in our study did not group
objects by particular tasks—objects related to task A, to
task B, etc—but four users did group objects according to
whether an object had an associated task, and by properties
of that task (state, importance, and cost or di culty). This
could be used to extend the work of Jones and his colleagues
[
Time was mentioned by four participants. However rather than categorise objects according to time of use (or receipt), as supported by a number of tools, three participants derived criteria from the lifecycle of an object. Criteria such as “when I need to act on this” or “when this is important” will change over time for each object. This is related to Cole’s “level of information” dimension. Only one participant used objects’ importance to an event, at a particular time, as a criteria.
Tools which support an explicit notion of document or task lifecycle, or approximate this e.g. by recording patterns of use, would presumably suit these participants. 5.
DISCUSSION OF THIS APPROACH
The results above suggest that card sorts, in this variant, are useful for eliciting criteria: it does seem possible to gain some insight into how users think of personal objects, and how we might support this. This pilot has, however, highlighted some limitations.
The objects represented by each participant’s cards were familiar—that is, they tended to choose objects they had used recently or frequently. They were also selected for sharing, since although we did not record the card titles we did see them. We cannot be sure that the chosen objects represent the sorts of things users may search for in a PIM system, and of course they are not representative of unfamiliar objects. We could instead choose objects from a participant’s computer, for example by choosing randomly from the file system and labelling cards with file icons and names, and similar. There is a tradeo↵ , however: if participants did not recognise these objects, the only possible criteria would be file icons and names, and we would learn little. By allowing users to choose their own objects the cards hopefully acted as prompts for other, richer, associations.
On a related point, some media types, such as video or audio, are di cult to represent on cards. It is not clear what this means for eliciting criteria. We are possibly unlikely to get criteria such as “out of focus” (for photos) or “scratchy bit in the middle” (for audio), but participants’ familiarity with the objects may mitigate this to some extent.
There are of course properties that are not captured by
this method: links between documents, for example (except
implicit links of the type “sorted into the same pile”). It is
also possible that our presence, and the apparatus we used,
made it hard for participants to think naturally. They may
have been inspired to create other categories if prompted
(as in Gonc¸alves and Jorge [
We also note that the objects chosen varied greatly from participant to participant, and this may have played a role in the criteria that were chosen—although we did see some overlap, possibly there would have been more if the objects were more similar. We are considering constraining participants more in future, for example by prompting them to make a certain number of cards for each media or perhaps having subpopulations sort a shared set of cards, e.g. emails, web pages etc. they have all seen or received.
CONCLUSIONS
We hope to extend this work by scaling to a larger group of participants, but we will consider some methological changes: constraining the objects chosen, for example, or careful prompts to elicit more classifications. Nonetheless card sorts, in the variant here, have proved useful for starting to understand how users think of “personal” digital objects from a wide range of sources and media. Some classifications were both common and expected, but we did observe interesting di↵ erences both with the criteria found in earlier studies and with the criteria exposed in PIM and search tools.
ACKNOWLEDGEMENTS We would like to thank our participants for their time.
Using semantic differentials for an evaluative view of the search engine as an interactive system
Frances Johnson Department of Languages, Information & Communications
Manchester Metropolitan University
Geoffrey Manton ABSTRACT In this paper, we investigate the use of semantic differentials in obtaining the evaluative view held by users of the search engine.
The completed scales of bipolar adjectives were analysed to suggest the dimensions of the user judgment formed when asked to characterize a search engine. These were then used to obtain a comparative evaluation of two engines potentially offering different types of support (or assistance) during a search. We consider the value of using the semantic differential as a technique in the toolkit for assessing the user experience during information interactions in exploratory search tasks.
Categories and Subject Descriptors H3.3 [Information search and retrieval]; Search process. H.5.2 [User interfaces]: Evaluation/methodology General Terms Measurement, Performance, Design, Human Factors 1. INTRODUCTION The design of interfaces to support exploratory search seeks to provide users with the tools for and the experience of an interactive and engaging search. This is a departure from the classic model of information retrieval wherein the user submits a keyword query to the system and scans the list of retrieved results for relevance, either stopping with relevant results or refining the query to get results that are closer to the information need.
Exploratory search does not necessarily assume that the user has a well defined information need (at least one that can be articulated as a keyword query) or indeed that the query will be ‘static’ and thus satisfied by a single list of retrieved results.
Presented at EuroHCIR2012. Copyright © 2012 for the individual papers by the papers' authors. Copying permitted only for private and academic purposes. This volume is published and copyrighted by its editors.
Accordingly, search engine developments have focused on
providing query assistance drawing on contextual aspects to the
search, such as personal history and/or current context [
The context for evaluation is predicated on White and Roth’s [
The challenge for the evaluation of exploratory search is the assumption that the user is willing or able to make an evaluative judgment throughout the search or that valid measures can be found through their actions, for example of usage of query terms.
In general, evaluation draws from established HCI measures of effectiveness (can people complete their tasks?) efficiency (how long do people take?), an assessment of the user’s overall satisfaction or other affective responses. Where possible, and increasingly so, the user actions are observed and recorded as dependent on the system and/or its interface. In this study we focus on an attempt to obtain the user’s evaluative view of the search engine, based on criteria which may be affected by the developments for new and richer interactive designs. It is assumed that this would be part of an assessment which when taken with others will build a picture of the ‘user experience’ of the system used in exploratory search.
USER EVALUATION In developing an instrument to collect the user assessment effort goes into ensuring that the evaluation is made in the task context.
It means little to know that the user is ‘satisfied’ with the interface without gaining insight into why this assessment has been formed.
A variety of questionnaires have been developed for assessing
usability of interactive systems, such as search engines. Two well
known are the SUS (System Usability Scale) developed at the
Digital Equipment Corporation [
Questionnaires, such as SUS, are used in an experimental set up
when an explanation of the user’s overall assessment is sought.
However, the limitations of the questionnaire to capture and
provide insight into the complexity of the user’s assessment has
lead to alternative tools, for example Microsoft’s Product
Reaction Cards in the "Desirability Toolkit". This invites
participants on a usability test to select as many, or as few, words
from a list of 118 which best describe their reaction and/or
interaction with the system they have just used. Benedek and
Miner [
Given the potential scope of the users’ response (represented in the reaction cards with some 100+ terms) this study sets out to investigate the value in assembling these into a framework (of sorts) for the collection of the users’ evaluative judgment of an interactive system based on the technique known as ‘semantic differentials’. Specifically the aim of this small preliminary investigation was to begin to determine the extent to which users hold an evaluative view of a ‘search engine’ and, what are the dimensions (traits or criteria) on which we form this view. If it can be found that this view is strongly held (that is, an attitude is formed which may influence how we behave and interact with the search engine) then it may be feasible to investigate the influence, if any, of a design for information interaction on the evaluative view. In this study the technique of semantic differentials is used to best describe the evaluative view held by its participants. This is then employed to assess two quite different search engines following the completion of two query based searches.
SEMANTIC DIFFERENTIALS
Semantic Differentials (SDs) originate from the work of Osgood
[
The assumption made here, in the use of SDs on ‘search engines’ is that users hold an evaluative view which is formed when using the engine to find and/or explore information. The SD is used to investigate the adjectives that best ‘conceptualise’ the search engine, from the user perspective. Factorial analysis is also used to identify the dimensions of the judgment, in a sense the packaging of the components of the judgment into smaller units of meaning reflecting what is important when responding to the concept ‘search engine’.
The design of the SD aims to allow a degree of abstraction in the evaluation so that participants can reflect the complexity of their response. In this study, the adjectives to include on the SD scale were chosen from Microsoft’s Product Reaction Cards, these having been collected in previous research, usability studies and in the marketing of web sites and systems. The majority of the terms formed pairs on some continuum and 40 terms (20 pairs) were selected to present in the SD. The selection was subject to the judgment of the researcher. This is a limitation of this exploratory study, however some steps were taken to formalise the selection. A loose grouping of the adjective pairs was made as relating to appearance (such as ‘attractive’ ), judgment (‘relevant’), emotive (‘bor ing’) and use (‘fast’). Five pairs from each of these groupings were made. The pairs were mixed on the SD to avoid having all the positive terms on one side of the scale and only intervals were shown on the scales with the numerical values used only for data entry. This allowed participants to focus on how an adjective pair related to the engine and its characteristics, rather than on ‘scoring’ it in some way. 3.1
Implementation The study was conducted on our undergraduates studying BSc Web Development and on a postgraduate cohort studying on MA Library and Information Management or the MSc Information Management. A total of 89 students participated in the study. At the start of the class each participant was asked to think about a search engine, and adjectives they would use to describe the engine, (in other words, “what it means to them” ). Each participant was then given the SD to complete. This is referred to as the ‘baseline’ and the data we re analysed to gauge user perceptions of search engines.
In the following lab sessions (about one hour later) each participant was required to perform two search tasks on each of the two search engines - Google, an engine we can assume some familiarity and, a second clustering engine (Yippy, formerly Clusty). The two tasks were as follows These were selected to give the participants experience of using the engines for a closed question (find symptoms) and on a more open ‘informational’ type of query (on the ‘nature nurture’ debate). A measure of search success was not taken as the aim was simply to get the participants using the engines. The order of use of the two sites was randomized so that approximately half of the participants worked on Google first and half on the clustering engine. All were told to spend no longer than 10 minutes searching on each engine and to complete the SD for each engine immediately after each use. 4.1
FINDINGS
Evaluative views The responses to the baseline (think of an engine) were entered into SPSS with the scales coded (7-1) so that the positive adjectives corresponded to the higher numbers. Descriptive statistics of mean, mode and standard deviation were calculated for each of the adjectives. Those with a mean greater than 4 or less than 3 were taken to suggest the adjective pairs that best characterise the participants’ view, as follows attractive powerful valuable relevant satisfying fast predictable intuitive easy unattractive simplistic not valuable irrelevant frustrating slow unpredictable rigid difficult Factor analysis investigates the correlations among subsets of the responses to the bipolar pairs and groups the correlated variables such that each group is largely independent of the others.
Exploratory factor analysis was employed to identify the groups
which might explain most of the variance in the data. With 20
pairs of adjectives to perform Principal Components Analysis
(PCA) in SPSS it is recommended that a minimum of 100
responses are obtained, whilst others recommend that a sample
requires approx 5-10 times the number of people as scale pairs
[
Factor 1 label USE – Utility effective, valuable, satisfy, relevant, predictable,
intimidating, inspiring, stimulating Factor 2 label QUALITY – Affective
engaging, fun, connected Factor 3 label QUALITY - Appearance
high quality, personal, meaningful, rigid, attractive Factor 4 label USE – Efficient
easy, intuitive, fast, powerful Factor 5 label USE - Control controllable 4.2 Comparative evaluations Using the same SDs, participants scaled their responses post search using Google and the clustering search engine. These were entered into a worksheet to obtain basic statistics. The mode for each adjective is shown Figure 1 with a note of those with mode >4 and < 3 suggesting a positive or negative response.
Google (mode > 4 or < 3) & in bold where mean is also > 4 1attractive - , 6valuable - , 8relevant - , 15satisfying - , 16fast - , 17predictable - ,18controllable -, and (where mode < 3) 19rigid
Clustering search engine (mode > 4 or < 3) & in bold where mean > 4 or < 3 14engaging - , 19intuitive – and (where mode < 3) 13intimidating - , 17 – unpredictable Using the suggested dimensions or aspects of the user evaluation from the factor analysis of the ‘baseline’ data we can compare the participants’ responses on the high or low scoring adjectives across the engines. On QUALITY – Appearance Google was rated rigid and attractive and whereas Google was neutral on the factor QUALITY- Affective, the clustering search engine obtained a positive score towards the adjective engaging.
On the factor labeled USE- Utility Google was scored as predictable, valuable, relevant and satisfying, whereas the clustering engine as unpredictable and towards intimidating. On USE-Efficient Google was rated as fast and the clustering engine appears more intuitive. Google was also rated as controllable.
DISCUSSION This is an exploratory study and it has its limitations. It is questionable whether the selection of the adjectives to use in the SD influenced the results. In particular there is uncertainty in the results that intuitive to rigid is on some continuum. Also there is some unease at accepting a factor with 8 out of 20 pairs and one with only one. Perhaps the sample size was too small to attempt factoring. The results also raise questions on how some of the adjectives were interpreted by the participants. These withstanding, the participants in this study did appear to hold an evaluative judgment of the concept ‘search engine’ and the traits represented in the scale were grouped to suggest the aspects on which an assessment may be formed. It is of particular interest that upon using the search engine Google to conduct a search task the ratings on the SD, on the whole, altered only in the factors of ‘controllable’ and USE –efficient (easy, intuitive and powerful). Perhaps we can assume that Google was the typical engine when asked to think of an engine in the baseline and, when it came to use Google, users shifted their perception with regards to some of the adjectives. Perhaps this is not surprising but it may suggest that we hold an implicit view of search engines, and that this view will be influenced by actual use (and the experience).
Our participants may have had less familiarity with the clustering engine, and in the evaluation this appears to have prompted an ‘affective’ response in finding the engine to be ‘engaging’ whilst also indicating shifts in the ‘use’ factors (towards an assessment of the engine as ‘unpredic table’). Again the infallibility of some of the terms is highlighted where an ‘unpredictable’ system may be regarded to be a negative judgment, but if the system is also considered to be engaging the assessment could be highly desirable depending on the user’s goals. This study of the use of semantic differentials indicates that it is worth running the test with a new cohort of students to determine the extent to which a consistent view is obtained. As an exploratory study it also suggests that further research on user’s perceptions and mental models of search engines is worthwhile. With regards to the challenge of providing an evaluation of the exploratory search, this study falls short as no behavioural data was obtained.
However, perhaps, with further design of the SD and use in an experimental set up with honed tasks, a user assessment of the interface may be obtained as dependent on the search interface development and design. _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ unattractive personal fun simplistic connected not valuable low quality relevant ineffective meaningful confusing inspiring empowering engaging frustrating slow unpredictable uncontrollable rigid eas The fault, dear researchers, is not in Cranfield, But in our metrics, that they are unrealistic.
Mark D. Smucker Department of Management Sciences
University of Waterloo, Canada mark.smucker@uwaterloo.ca
Charles L. A. Clarke
School of Computer Science University of Waterloo, Canada claclark@plg.uwaterloo.ca 1. INTRODUCTION
As designers of information retrieval (IR) systems, we need some way to measure the performance of our systems.
An excellent approach to take is to directly measure actual
user performance either in situ or in the laboratory [
The Cranfield approach to IR evaluation provides low-cost, reusable measures of system performance.
Cranfield-style evaluation frequently has been criticized as
being too divorced from the reality of how users search, but
there really is nothing wrong with the approach [
As such, we should really be thinking of the Cranfield approach as the application of models to make predictions, which is common practice in science and engineering. For example, physics has equations of motion. Civil engineering has models of concrete strength. Epidemiology has models of disease spread. Etc. In all of these fields, it is well understood that the models are simplifications of reality, but that the models provide the ability to make useful predictions.
Information retrieval’s predictive models are our evaluation metrics.
The criticism of system-oriented IR evaluation should be redirected. The problem is not with Cranfield — which is just another name for making predictions given a model — the problem is with the metrics.
We believe that rather than criticizing Cranfield, the correct response is to develop better metrics. We should make metrics that are more predictive of human performance. We should make metrics that incorporate the user interface and realistically represent the variation in user behavior. We should make metrics that encapsulate our best understanding of search behavior.
In popular parlance, we should bring solutions, not
problems, to the system-oriented IR researcher. To this end,
we have developed a new evaluation metric, time-biased
gain (TBG), that predicts IR system performance in
human terms of the expected number of relevant documents
to be found by a user [
Presented at EuroHCIR2012. Copyright (C) 2012 for the individual papers by the papers’ authors. Copying permitted only for private and academic purposes. This volume is published and copyrighted by its editors. 2.
TIME-BIASED GAIN
HCI has a long history of automated usability
evaluation [
Ja¨rvelin and Keka¨l¨ainen produced a significant shift in
evaluation metrics with their introduction of cumulated
gainbased measures [
Cumulated gain can be plotted vs. time to produce a gain curve and compare systems. The curve that rises higher and faster than another curve is the preferred curve. While we can plot gain curves of one system vs. another, it is wellknown that users do not endlessly search; different users stop their searches at different points in time for a host of reasons. Given a probability density function f (t) that gives the distribution of time spent searching, we can compute the expected gain as follows:
E[G(t)] = ! ∞ 0
G(t)f (t)dt,
(1) where G(t) is the cumulated gain at time t. Equation 1 represents time-biased gain in its general form, i.e. timebiased gain is the expected gain for a population of users.
While it is natural for us to talk about cumulated gain over time, the traditional cumulated gain measures have substituted document rank for time and implicitly model a user that takes the same amount of time to evaluate each and every document. By making time a central part of our metric, we gain the ability to more accurately model behavior.
For example, in a document retrieval system, longer
documents will in general take users longer to evaluate, and if the
retrieval system presents results with document summaries
(snippets), we know that users can use summaries to speed
the rate at which they find relevant information [
Another significant advantage of using time directly in our retrieval metric is that we now make testable predictions of human performance. Our predictions are in the same units as would be obtained as part of a user study. To our knowledge, this alignment between the units of Cranfield-style metrics and user study metrics has not previously existed.
Time-biased gain in the form of Equation 1 makes no
mention of ranked lists of documents, for it is a general purpose
description of users using an IR system over time. To
produce a metric suitable for use in evaluating ranked lists, we
followed a process common to development of new
simulations [
3. Validation of model.
Our first step in model creation was to adopt the standard model of a user that works down a result list and move Equation 1 to a form common to cumulated gain measures: ∞ ! gkD(T (k)), k=1
(2) where gk is the gain at rank k, T (k) is the expected time it takes a user to reach rank k, and D(t) is the fraction of the population that survives to time t and is called the decay function.
Our model for the time it takes a user to reach rank k, T (k), takes into consideration a hypothetical user interface that presents results to the user in the form of document summaries. A click on a document summary takes the user to the full document. We model both the probabilities of clicking on summaries given their NIST relevance and the probability of then judging a viewed full document as relevant. We separately model the time to view summaries and full documents. For the time spent on a full document, we modeled longer documents taking longer with an additional constant amount of spent. We treated duplicate documents as zero length documents. We then calibrated T (k) using data from a user study, and finally we validated that our T (k) provided a reasonable fit to the user study data. Likewise, we modeled D(t) as exponential decay fit to a search engine’s log data.
In contrast, older evaluation metrics such as mean average
precision [19, p. 59] cannot be calibrated and have only
been validated after their creation. For example, the work
of Hersh and Turpin [
The Cranfield approach to IR evaluation is merely
another name for the development and use of predictive
models, which is a fundamental part all science and engineering
fields. In particular, IR evaluation fits nicely into the
framework of simulation where models are created, calibrated, and
validated before being used to make predictions. We have
presented time-biased gain as an example of what we believe
the correct direction is for IR system evaluation. We are not
the only ones to be working on better metrics or taking a
simulation based approach [
Our position is that system-oriented IR research is useroriented IR research given its use of evaluation metrics that model users. If HCIR researchers can produce better models than exist today — by better, we mean more predictive of human performance — then we can help system development to focus on changes that help users better search. 4. ACKNOWLEDGMENTS
This work was supported in part by the NSERC, in part by GRAND NCE, in part by Google, in part by Amazon, in part by the facilities of SHARCNET, and in part by the University of Waterloo. Any opinions, findings and conclusions or recommendations expressed in this material are the authors’ and do not necessarily reflect those of the sponsors. 5. REFERENCES
A Model of Consumer Search Behaviour
In order to design better search experiences, we need to
understand the complexities of human information-seeking
behaviour. In previous work [
In this paper, we extend this work to the site search context, examining the needs and behaviours of users of consumeroriented websites and search applications.
We found that site search users presented significantly different information needs to those of enterprise search, implying some key differences in the information behaviours required to satisfy those needs. In particular, the site search users focused more on simple “lookup” activities, contrasting with the more complex , problem-solving behaviours associated with enterprise search. We also found repeating patterns or ‘chains’ of search behaviour in the site search context, but in contrast to the previous study these were shorter and less complex. These patterns can be used as a framework for understanding information seeking behaviour that can be adopted by other researchers who want to take a ‘needs first’ approach to understanding information behaviour .
Categories and Subject Descriptors H.3.3 [Information Search and Retrieval]: Search process; H.3.5 [Online Information Services]: Web-based services General Terms Human Factors.
Keywords
Site search, enterprise search, information seeking, user
behaviour, search modes, information discovery, user experience
design.
1. INTRODUCTION
Classic IR (information retrieval) is predicated on the notion of
users searching for information in order to satisfy a particular
'information need'. However, it is now accepted that much of what
we recognize as search behaviour is often not informational per
se. For example, Broder [
Presented at EuroHCIR2012. Copyright © 2012 for the individual papers
by the papers' authors. Copying permitted only for private and academic
purposes. This volume is published and copyrighted by its editors.
In this paper, we examine the needs and behaviours of individuals
across a range of site search scenarios. These are based on an
analysis of user needs derived from a series of customer
engagements involving the development of customised site search
applications. In so doing, we extend and validate a model of
information behaviours derived from a previous study of
enterprise search users [
The model is based on a set of ‘search modes’ that users employ to satisfy their information search and discovery goals. It extends the IR concept of information-seeking to embrace a broader notion of discovery-oriented problem solving, addressing a wider range of information interaction and information use behaviours.
The overall structure of the model reflects Marchionini’s [
We investigate the degree to which the model extends to
accommodate the domain of site search (i.e. consumer-oriented
websites and search applications) and discuss some of the
differences between the needs and goals of enterprise search users
versus those of site search. We conclude by exploring the ways in
which these modes combine to form distinct chains or patterns,
and reflect on the value this offers as a framework for expressing
complex patterns of behaviour.
2. MODELS OF INFORMATION SEEKING
The framework investigated in this study is influenced by a
number of existing models. For example, Bates [
Cool & Belkin [
Ellis and his colleagues [
In addition to the research-oriented models outlined above, we
should also consider practitioner-oriented views. Spencer [
In this paper, we use the characteristics of the models above as a
lens to interpret the behaviours found in a new source of empirical
site search data. We also explore the combinatorial nature of the
modes, extending Ellis’s [
These information needs take the form of ‘micro-scenarios’, i.e. a brief narrative that illustrates the end user’s goal and the primary task or action they take to achieve it, for example:
Find best offers before the others do so I can have a high margin.
Get help and guidance on how to sell my car safely so that I can achieve a good price.
Understand what is selling by area/region so I can source the correct stock.
See year-on-year ad spend trends for TV and online to supply to the Head of Global Media.
The scenarios were collected as part of a series of requirements workshops involving stakeholders and customer-facing staff from the respective client organisations. They were generated by participants in individual breakout sessions, and then moderated by the workshop facilitator in a group session to maximise consistency and minimise redundancy or ambiguity. They were also prioritised by the group to identify those that represented the highest value both to the end user and to the client organisation.
This data possesses a number of unique properties. In previous
studies of information seeking behaviour (e.g. [
Moreover, the scope and focus of these scenarios represents a
further point of differentiation. In previous studies, (e.g. [
Finally, the data is unique in that is constitutes a genuine
practitioner-oriented deliverable, generated expressly for the
purpose of designing and delivering professional site search
systems. As such, it reflects a degree of realism that interview data
or other research-based interventions might struggle to replicate.
3.2 Data Analysis
These scenarios were analyzed using the model derived previously
for the domain of enterprise search [
Although the original study involved three separate analysts, the behaviours this time were identified by the first author alone. The current analysis approach is therefore much more subjective.
However, the first author was also the facilitator at each of the requirements workshops at which the scenarios were generated, and was able to again a deep insight into the needs, goals and motivations of the participants. This allowed him to be as confident as possible in his understanding of the users’ information needs and consistent in his interpretation of the information behaviours required to satisfy a particular need.
A number of the scenarios focused on needs that did not involve
any explicit information seeking or use behaviour, e.g. “Achieve a
good price for my current car”. These were excluded from the
analysis. A further number were incomplete or ambiguous, or
were essentially feature requests (e.g. “ Have flexible navigation
within the page”), and were also excluded. This process resulted
in further confirmation and validation of the nine search modes
identified in the original study, but with revised definitions to
reflect a broader scope:
1. Locate: To find a specific (possibly known) item, e.g. “ Find my
reading list items quickly”. This mode encapsulates the
stereotypical ‘findability’ task that is so commonly associated
with site search, consistent with (but a superset of) Spencer’s [
In some ways the boundaries of this mode are somewhat less
prescribed than the others, but what the instances share is the
characteristic of open ended, opportunistic search and browsing in
the spirit of O’Day and Jeffries [
Just as new definitions were needed to accommodate the new domain, new patterns of occurrence were identified in the data.
Typically these consisted of chains of length two or three, of which the following were most frequent:
Insight-driven search: (Explore->Analyze-> Comprehend): This patterns represents an exploratory search for insight to resolve an explicit information need, e.g. “ Assess the proper market value for my car” (45 instances) Opportunity-driven search: (Explore-LocateEvaluate): In contrast to the explicit focus of the pattern above, this sequence represents a less directed exploration in the prospect of serendipitous discovery e.g. “ Find useful stuff on my subject topic”(31 instances) Qualified search (Locate-Verify) This pattern represents a variant of the stereotypical findability task in which some element of immediate verification is required, e.g. “ Find trucks that I am eligible to drive” (29 instances) A deeper insight into these patterns can be obtained by presenting them in diagrammatic form, as a network (Figure 1). This diagram illustrates the three sequences outlined above plus other commonly found patterns. It also reflects an outcome of the pervious study, in that certain modes tend to function as “terminal” nodes, i.e. entry points or exit points to a scenario. For example, Explore typically functions as an opening, while Comprehend and Evaluate function in closing a scenario. Analyze typically appears as a bridge between an opening and closing mode. 4.1 Site search vs. Enterprise Search The sequences described above also allow us to reflect on some of the differences between the needs of site search users and those of enterprise search. One of the most fundamental differences is an emphasis on simpler “lookup” modes such as Locate and Verify: these were relatively rare in the enterprise search data, but prominent in site search (120 and 39 instances respectively).
Enterprise search, by contrast, emphasised higher-level “investigate” behaviours such as Analyze and Evaluate (modes which also appeared frequently in site search, but not as prominently: 58 and 61 instances respectively). However, in neither case was the stereotype of ‘search as findability’ borne out: even in site search (where it was the most common mode), Locate was accountable for no more than a quarter of all instances.
But perhaps the biggest difference was in the composition of the chains: while enterprise search was characterised by a wide variety of heterogeneous chains, site searched focused on a small number of common trigrams and bigrams. Moreover, these chains displayed little evidence of the composite nature observed in enterprise search, in which certain chains were seen to be embedded within others to create larger, more complex sequences of behaviour. 5. DISCUSSION A key feature of the current model is its emphasis on the combinatorial nature of search modes, and the value this offers as a framework for expressing complex patterns of behaviour. Such an approach is not unique: the second author, for example, has also previously explored the concept of mode chains to describe information seeking behaviours observed in naturalistic settings.
However, his approach was based on the analysis of complex tasks observed in real time, and as such was less effective in revealing consistent patterns of atomic behaviour such as those found in the current study.
Conversely, this virtue can also be a shortcoming: the fact that simple repeating patterns can be extracted from the data may be as much an artefact of the medium as it is of the information needs it contains. These scenarios were expressly designed to be a concise, self-contained deliverable in their own right, and applied as a simple but effective tool in the planning and prioritisation of software development activities. This places a limit on the length and sophistication of the information needs they encapsulate, and hence a natural boundary on the scope and extent of the patterns they represent. Their format also allows the analyst to apply perhaps an unrealistic degree of top-down judgement and iteration in aligning the relative granularity of the information needs to existing modes; a benefit that is less readily available to those whose approach involves real-time, observational data.
A further caveat is that in order to progress from understanding an information need to identifying the information behaviors required to satisfy those needs, it is necessary to speculate on the behaviours that a user might perform when undertaking a task to satisfy the need. It may transpire that users actually perform different behaviours which achieve the same end, or perform the expected behavior but through a combination of other nested behaviours, or may simply satisfy the need in a way that had not been envisaged at all.
Finally, the process of inferring information behaviour from selfreported needs can never be wholly deterministic, regardless of the consistency measures discussed earlier. In this respect, further steps should be taken to operationalize the application of the framework and apply some independent measure of stability or objectivity in its usage. 6. CONCLUSIONS In this study we have investigated a model of information seeking behaviour derived from the domain of enterprise search, and validated its extensibility to users of consumer-oriented websites and search applications. In so doing, we explored a novel, goaldriven approach to eliciting user needs, and identified some key differences in user behaviour between the two domains.
In addition, we have demonstrated the value of the model as a framework for expressing complex patterns of behaviour, extending the IR concept of information-seeking to embrace a broader range of composite information interaction and use behaviours. Moreover, we propose that our method can be adopted by other researchers who want to take a ‘needs first’ approach to understanding information behaviour.
Revisiting User Information Needs in Aggregated Search
Shanu Sushmita
University of California LA shanusushmita@ucla.edu
Robert Villa
University of Sheffield r.villa@sheffield.ac.uk
Martin Halvey Glasgow Caledonian
University Martin.Halvey@gcu.ac.uk
Mounia Lalmas
Yahoo! Labs Barcelona
mounia@acm.org
ABSTRACT
Aggregated search interfaces are a common way to present
web search results, mixing di↵ erent types of results into one
single result page. Although numerous e↵ orts have been
made to infer users’ information needs in “standard” search,
we know little about users’ information needs within the
context of aggregated search. This paper presents the outcomes
of a survey of 117 respondents, investigating users’
preferences for their type of search result (image, news, video)
and their type of information need (informational,
navigational and transactional). The survey reveals that users’
result preferences di↵ er based on their underlying information
needs, suggesting that the taxonomy provided by Broder [
Throughout the evolution of web search, users’ interaction
with search results has been studied by many to improve the
quality of the search results and the search experience.
Efforts were (and are still being) made to understand users’
information seeking process, based upon which several
taxonomies describing users’ behaviours have been proposed [
For instance, in 2002, Broder [
Others such as Lindley et al. [
While there is a substantial body of work on understanding users’ information needs and browsing activities in “standard” search, far less is known about these within the context of aggregated search. For instance, it is not clear if the existing taxonomies on information needs for “standard” search hold in an aggregated search scenario. In aggregated search, search results may originate from di↵ erent media (e.g., images, maps) or may be of di↵ erent genres (e.g., news, blogs).
This may have an e↵ ect on the way users interact with the
results, and a↵ ect their preferences for the types of results. A
study in [
The focus of this short paper is, therefore, two-fold: (1)
to investigate the preference of search results sought by the
users; and (2) to investigate the existing frameworks of web
activities within the context of aggregated search. For this
purpose, users’ preferences for results of several media types
and genres are investigated. Furthermore, since Broder’s
taxonomy has been heavily used (e.g. [
We present the results of a survey that investigated users’
preferences for results of di↵ erent media types and genres,
as answers to informational, navigational and transactional
queries.
2. STUDY
A survey containing sixteen questions (4 background
questions and 12 search task questions) was distributed on
various social networks. The survey allowed us to reach a large
and diverse enough number of users, and is a common way to
elicit user perceptions and preferences [
The respondents’ age varied between 20-59 years (mean 29).
Geographically, respondents were distributed across the US and Canada (3%), Europe (34%), Asia (62%) and Africa (1%). Most respondents were familiar with search engines and used them frequently. 2.1
Task The aim of the survey was to elicit users’ preferences for the types (media, genres) of search results for informational, navigational and transactional search tasks. To this end, we designed four search topics2 for each of these three categories. The list of topics for each category is listed in Table 1.
In total, there were twelve questions for each respondent to answer. The orders of the questions were rotated to minimise ordering bias.
We designed topics that could be understood universally (e.g, global warming, checking emails, buying dvd, software download). Furthermore, the topics were devised to fit the informational, navigational and transactional categories.
Therefore, we did not manipulate topics to suit specific media or genre. For instance, for the topic global warming, some people may want to read the latest news about global warming, some others may want to view pictures of melting icebergs, while some others may want to watch a documentary on global warming. Therefore this topic does not have an implicit type intent (e.g. image) but requires the gathering of information (informational search task) from many web pages; it is expected that users will look for multiple results to satisfy the corresponding information need. However, it will depend on users which result types (image, news, video, etc) they prefer to view – only news articles, few pictures, or a combination of both. 2.2
Procedure
For each search topic, the respondents were given five choices,
namely, web, news, image, video and other results3. The
re1http://www.dmoz.org/
2A search topic describes a search task scenario. The
concept of a search task scenario was inspired from [
Figure 1, shows the screenshot of an example question with the preference options. Next, the outcomes of the survey are presented. As the data obtained from the survey was non-parametric, we report medians and the interquartile range for the preference scores. The results are reported in Table 2, which shows the median rank of each vertical by information need. Friedman tests were performed to estimate the significance of preference for the results types, among and across the three categories (navigational, informational and transactional).
Finally, multiple Wilcoxon-tests were run in the post-hoc analyses while adjusting the p-values using the Bonferroni method. The outcomes from the post-hoc pair wise comparisons for navigational, informational and transactional categories are shown in Tables 3, 4 and 5 respectively. Each row in these tables indicates whether a particular result type was preferred over each of the other result types.
As can be seen in Table 2, most respondents indicated the ‘web page’ as the most preferred type of results, when
Navigational Topics 1. When you wish to book tickets with British Airways, which results would be useful for you? 2. When you wish to find an address from yellow pages, which results would be useful for you? 3. When you wish to check courses of a University, which results would be useful for you? 4. When you wish to check your email (e.g, gmail, hotmail, msn, etc), which results would be useful for you? Informational Topics 5. When you wish to learn about salsa dance, which results would be useful for you? 6. When you wish to gather information about global warming, which results would be useful for you? 7. When you wish to learn on how to make a pancake, which results would be useful for you? 8. When you wish to know about 2011 budget, and how it e↵ ected farmers, which results would be useful for you? Transactional Topics 9. When you wish to download a free software, which results would be useful for you? 10. When you wish to download a song for your iTunes library, which results would be useful for you? 11. When you wish to file a property complaint, which results would be useful for you? 12. When you wish to buy a DVD online, which results would be useful for you? compared to the other four types (image, video, news and others). The di↵ erence was found to be significant for navigational, informational, and transactional cases (rows 1-4 in Tables 3, 4 and 5 ); thus suggesting that “standard” web results are the prime source of information sought by most users. After web results, news was the second most preferred type of results when compared to image, video and others (6th row in Table 2). For the navigational category, news results were significantly preferred over image, video and others results (rows 6, 8 and 9 in Table 3). However, video was equally preferred to news for informational and transactional categories (row 8 in Tables 4 and 5).
Finally, there is a trend for image and video results to come third in preference from respondents for most categories (4th and 5th rows in Table 2). However, post-hoc analyses suggest a significant di↵ erence of preference for video and image over ‘other results’ for all three categories (rows 7 and 10 in Tables 3, 4 and 5). In addition, video results were significantly preferred to image results for informational and transactional cases (row 5 in Tables 4 and 5), while no significant di↵ erence was observed for the navigational case (row 5 in Table 3 ). Therefore, it is possible that users may prefer image results instead of video results in some cases, and video results in other cases. In addition, image and video being the third preference indicates that providing image and video results for all queries may not be appreciated by users.
In Tables 3 to 5, in only two occasions were the ranking of result types not significantly di↵ erent: image-video for navigational, and news-video for informational information needs. This indicates that for navigational needs, neither image or video results are judged as important to users, backing up the results in Table 2, where both are ranked bottom. For informational information needs, both news and video were judged equally important to the search tasks, second only to web (Table 2).
DISCUSSION The aim of our study was to investigate, via a survey, users’ results preference for navigational, informational, and transactional search topics.
Overall, three key observations can be made from this
survey. First, for all query categories, web results continue to
be the prime source of information sought by users – 90%
for navigational, 54% for informational and 85% for
transactional – suggesting that for an aggregated search result
page, web results should always be provided. This echoes
the findings of [
Second, there appears to be a di↵ erence between the result preferences for navigational and transactional queries.
From Broder [
Third, users’ preferences for the ‘type’ of results vary with
the query category. For instance, for navigational and
transactional search topics, web and news results seem to be
preferred. The preference is more mixed for informational
search topics, with image results least preferred. In itself,
it is not surprising that users’ preferences vary with query
categories. However, concrete knowledge regarding which
‘types’ of sought results are preferred would allow for more
appropriate aggregation of the di↵ erent verticals under
consideration. Similar investigations were carried out in [
CONCLUSION AND FUTURE WORK 4http://www.slideshare.net/rankabove/com-scorerankabove-final
We presented the analysis of a survey of 117 respondents’ preferences regarding the di↵ erent types of results for navigational, informational, and transactional information needs.
Although small in terms of the number of users and
acknowledging the limitation of an online survey, interesting insights
emerged from our investigation. The outcomes of the
survey support the aggregated search paradigm, showing that
users’ preferences are for a diverse range of result types. The
analysis also indicates a need to revisit the definition of the
three categories of information needs [
Figure 4a. Visual aids (rectangle: action area, here: “not like”, circle: gaze cursor).
Figure 4b. No visual aids. 4. EVALUATION DESIGN In order to evaluate EyeGrab, 24 subjects (7 female) played the game. The subjects’ age was between 15 and 32 years (mean: 24, SD: 3.9). 19 subjects were students, 2 research assistant, one pupil and 2 had other professions. Most of the players had experience in gaming (mean: 3.5, SD: 1.31). Only a few were familiar with eye tracking, this is indicated by 19 subjects rating the question concerning their eye tracking experience with one (mean: 1.63, SD: 1.38). The subjects were randomly divided into two groups A and B. Group A had no visual aids during the game, whereas group B did. 8 users were wearer of glasses or contact lenses (4 in each group). There were no problems using the eye tracker for those subjects.
To avoid distractions, the game was played in our eye-tracking lab providing a chair, a desk with an eye tracker, and a standard monitor. The first step was a calibration of the eye tracker. After this was done, the game was started without further instructions and was played with 30 images in each of the three rounds. The data from the first round is not used in the later analysis, because it has only served for getting the subjects acquainted with the usage of the eye tracker as input device. At the end of the experiment, every user filled out a questionaire, including personal information and questions about the performance of the game. The answers were given on a 5-point Likert scale. 5. EVALUATION RESULTS 5.1 Satisfaction The questionnaires show that the subjects enjoyed playing the game. On average, the statement “It was fun playing the game.” is rated 3.46 (SD: 0.93) considering all 24 subjects. 20 of the 24 users agreed to this statement. One of the following questions was if the participants felt like the interaction with the eye tracker increases the fun of the game. 14 subjects agreed or strongly agreed to this statement (mean: 3.5, SD: 1.25). Also, most of the subjects did not feel disturbed by the eye tracker (mean: 2.25, SD: 1.5). 5.2 Effectiveness and Efficiency One round of the game comprises 30 images and it takes about two minutes including the introduction and the input form. Each level has a different pace at which the images fall down the screen. Thus, the classification per image takes between 2.6 and 4 seconds.
In total, 1440 pictures were shown to the subjects within the game (30 pictures per category “house” and “mouse” times 24 subjects).
Only in 42 cases, the image passed without classification, resulting in a total of 1398 classified images. 1162 images were correctly classified (83%). Thus, only 236 images were incorrectly classified. Overall we had 897 true-positive classifications, 128 false-negative and 108 false-positive classification, which leads to a precision of 89% and a recall of 88% over all users. For the group with the better results (the group without visual aid, see next section) we obtain a precision of 92%. 5.3 Visual aid The subjective perception of the users in group B (the group that was provided with visual aids) was that the visual aids supported them in the classification tasks. The question regarding the visual highlighting of the active areas was rated as very helpful with an average of 4.67 (SD: 0.49). The subjects also answered that displaying the gaze point was very helpful and scored this question on average with 4.5 (SD: 0.67). However, to our surprise, the following statistical analysis of the data shows that group B with the visual aids misclassified significantly more images than group A did.
Group A correctly classified 296 images for category “house” whereas group B correctly assigned 264 images for this category.
Regarding the category “mouse”, in total 317 correct assignments were made by group A whereas group B correctly assigned 287 images. Regarding the misclassified images, group A misclassified 59 images for category “house”, whereas group B wrongly assigned the image category in 81 cases. For the category “mouse”, the number of incorrect assignments is 37 for group A and 57 for group B. We compared the values for correct and incorrect assignments for group A and B in a 2x2 Chi-square test for both categories. The differences are significant regarding a significance level of = 0.05 with 2 (1, N = 700) = 5.14, p = 0.023 for the category “house” and 2 (1, N = 698) = 5.6, p = 0.018 for “mouse”. In group B 31, images passed without classification, in group A only 11 images were not classified.
These results indicate that the visual support is not improving the classification. Despite the good impression of the visual support that group B expressed, the following question might be an indicator that this group felt less comfortable with the eye trackerbased interaction than group A did: we asked the subjects to state if they preferred a mouse-based interaction instead of the eye tracker-based one. On average, subjects of group A scored this question with 2.17 (SD: 1.47) and group B with 3.25 (SD: 1.48).
Using a two-tailed Mann-Whitney U-Test, a weakly significant difference was determined stating a preference of group B over group A to use the mouse to play EyeGrab (U = 43, z = -1.719, n1 = n2 = 12, p = .085). 6. FUTURE WORK For the current version of our EyeGrab game, we have used preclassified images in order to verify the classification performance of the subjects. We plan to use images without annotations in future extensions of the game.
Also the detailed analysis of the gaze information will be performed in a next step. In a small sample of 5 images classified by one user, we received 231 gaze points on the images. An example of a gaze path visualization is shown in Figure 5. We expect a sufficient number of fixations and correct classification to allow a detailed analysis.
We received 897 ratings for the shown images. 556 of them were
positive. The quality of these ratings has to be investigated in a
future experiment, e.g., by repeating the ratings in another context
with the same users or by using a ground truth set with images,
often liked by a big number of other users. However it has to be
clear, that a subjective rating can never be “correct” or not. These
investigations can only provide an indication of the worth of the
rating. Overall, this detailed analysis will allow us to identify the
regions that correspond to the category given in the EyeGrab
game. Such region-based annotations will allow for a better
retrieval of the images in the future.
7. SUMMARY
We have introduced the gaze-based game with a purpose
EyeGrab to classify images using an eye tracker. We have shown
that the game has the potential to entertain the players and that the
classification results are good enough to advance beyond the gaze
analysis. This analysis is the first step in the direction of
extending image context information with information gained in
an eye tracking game. The next step will be the analysis and
evaluation of the gained information and to use it for improving
image search tasks.
8. REFERENCES
[
Walber, T. and Scherp, A. and Staab, S. 2012. Identifying Objects in Images from Analyzing the Users’ Gaze Movements for Provided Tags. Advances in Multimedia
Modeling, 138-148.
[
Malik, J. 1999. Blobworld: A system for region-based image indexing and retrieval. Visual Information and Information
Systems. Using Wordclouds to Navigate and Summarize Twitter Search
Results Rianne Kaptein
Oxyme Amsterdam, The Netherlands This paper describes an application in which wordclouds are used to navigate and summarize Twitter search results. A search on Twitter can return thousands of relevant tweets. By just looking at the first few result pages you will not get an overview of what is discussed in all search results. Our application summarizes sets of tweets into wordclouds, which can be used to get a first idea of the contents of the tweets. Also the application provides the option to zoom in on a certain part of the search results to inspect them in more detail.
The application has not been formally evaluated, but we do provide some insights and points for discussion. 1
Introduction One of the most common problems in Information Retrieval is information overload: there is simply too much relevant information available for the users to process. Therefore applications are needed to help users deal with large amounts of data. In this paper we describe an application which was developed for this purpose. The use of wordclouds in the application serves two purposes: 1. To summarize 2. To aid navigation This application was developed with the following two user scenarios in mind: 1. General Twitter search
Nowadays many people express their opinions about products, services and companies on Twitter. When you want to get a broad overview of what people are tweeting in general about a company or event, it does not suffice to read the first few pages of search results. You want to get a feeling for the most frequently discussed topics overall, and dive into particular subtopics of special interest, such as product recommendations.
Presented at EuroHCIR2012. Copyright c 2012 for the individual papers by the papers’ authors. Copying permitted only for private and academic purposes. This volume is published and copyrighted by its editors. 2. Searching fragments of categorized data
Besides Twitter there are many more places on the Web where people express their opinions. These opinions can be collected and annotated with labels such as sentiment, source, market etcetera. When you have a large amount of annotated data available, it is interesting to see for example what are the different topics discussed in positive and in negative messages.
In this paper we will focus on the first user scenario: General Twitter search, since Twitter data is abundant and publicly available.
Humans have a great capacity to notice terms which are out of the ordinary. When looking at a wordcloud there will always be some unexpected terms which catch your attention and are good pointers for further investigation. In tweets about public transport you can expect for example tweets about delays, but you might not expect certain tweets about recent events such as a new colour of the trains.
What we try to do in the wordclouds is to emphasize the words that are noteworthy from a statistical point of view, and leave it up to the user to decide which messages to explore further.
Although the usefulness of tagclouds for navigation is still a topic of
debate [
The search results that we are investigating in this paper have three characteristics: • A search result is a short textual message. By design a Twitter
message cannot contain more than 140 characters. • The number of search results is large. If this would not be the case, since the results are short texts, you could simply read through all of them. • There are many, equally relevant search results. In web search there are usually not more than a handful highly relevant search results. Many of the search results contain copied or redundant information, or only mention the search words occasionally. Although Twitter search results also contain redundant information, i.e. repeated tweets and retweets, the set of relevant tweets can still consist of thousands of equally highly relevant tweets.
In the next sections of this paper we will present our approach (Section 2), a case study (Section 3), and finally our conclusions (Section 4). The application consists of two screens. The first screen handles the input, the second screen displays the results based on your input.
On the first screen the system offers a number of selections that can be made to make sure you generate the wordclouds that are best representing your data and your analysis purpose. The input is collected using textfields, radiobuttons and checkboxes. The first part of the inputscreen is shown in Figure 1.
The following selections can be made: • File selection, a tab separated text file is required as input. • Text selection, which column in the dataset to use as textual
input for the wordcloud generation. • Category selection, based on a value in any column of your dataset your data can be categorized. It is also possible to create categories based on the presence of words in the contents of your data, e.g. to create a category for all tweets containing the term ‘happy’. • Language, used for the removal of standard stopwords. • Optionally, additional stopwords can be specified. These
words will not occur in any of the wordclouds. • Stemming, currently available only for English. The Krovetz stemmer is used, because this stemmer always stems words into other valid English words. • Exclude numbers, when your data includes many numbers such as product prices it can be desirable to exclude these numbers from the wordcloud. • Exclude retweets / repeated posts, when your data contains a tweet that is retweeted very frequently, this one tweet will dominate the wordcloud which can be undesirable. • Include only usernames, for Twitter data only, keep only the
usernames, i.e. all the words starting with @. • Include only hashtags, for Twitter data only, i.e. all the words
starting with #.
The second screen shows the output, which consists of wordclouds for the categories you have specified, as well as a wordcloud for all the search results.
Wordclouds for categories are generated using a parsimonious
language model. This model compares the frequency of words in a
set of documents to the average term probability in a background
collection containing similar documents to extract the most
noteworthy terms. In this case the background collection are all the
retrieved search results. Terms that are only mentioned occasionally
in the set of documents and terms which have a similar or higher
probability of occurrence in the background collection will not be
included in the parsimonious language model [
The parsimonious language model [
(1) (2) Pmle(ti|S) = t f (ti, S) Â t t f (t, S) where S is the set of documents, and t f (t, S) is the text frequency, i.e. the number of occurrences of term t in set of documents S. Subsequently, parsimonious probabilities are estimated using Expectation-Maximisation:
E-step: M-step:
(1 l )P(t|S) et = t f (t, S) · (1 l )P(t|S) + l P(t|C)
e Ppars(t|S) = t , i.e. normalize
 t et where C is the background collection model. In the initial E-step, maximum likelihood estimates are used for P(t|S). We set the smoothing parameter l to 0.9. In the M-step the words that receive a probability below a threshold of 0.001 are removed from the model. The iteration process stops after a fixed number of iterations.
In the next section we present a case in which the generated output of the application is presented. 3
Case Using an example search we will demonstrate how we use wordclouds in our application to navigate and summarize the search results. We executed a search on Twitter using the Twitter search API1 for the query ‘#london2012’ over the last 5 days, saved all the 30,504 search results in a .csv file and load this file into our application. Looking at the wordcloud over all the results that is shown in Figure 2, we see the term ‘torch’ is frequently used, and we zoom in on this aspect of the ‘#london2012’ search. By clicking on the word ‘torch’ a list of messages is shown that all contain the term ‘torch’, so these messages can be inspected in more detail.
This list of messages is still quite long however, consisting of 1,046 tweets. We can zoom in further on these tweets by going back to the input screen and specifying ‘torch’ as a category. Now, a parsimonious wordcloud is created from the 1,046 tweets that contain the term ‘torch’. The resulting wordcloud is shown in Figure 3.
The figure is a screenshot of the screen that is displayed when the word ‘Sheffield’ is clicked, showing the tweets containing the word ‘Sheffield’.
Words which occur frequently in all of the ‘#london2012’ messages, such as ‘#london2012’, ‘2012’, and ‘olympics’, receive a lower score from the parsimonious model, and almost none of these
1https://dev.twitter.com/docs/api/1/get/search words occur in the ‘torch’ wordcloud. Also general words that occur frequently in all of the messages, such as ‘get’, and ‘will’ are filtered out. Instead the cloud contains words that occur more frequently in the subset of messages that contain the word ‘torch’, for example some of the cities that the torch passes through such as Sheffield, Leeds and Manchester. Every result in this cloud by definition contains the word ‘torch’, therefore it takes a prominent place in the wordcloud. You can choose to not show the word ‘torch’ in the wordcloud by specifying it as a stopword on the input screen.
Clicking on a term in the wordcloud has the same effect as query expansion, i.e. adding that term to your query and retrieve another set of results. When you use the Twitter API to search Twitter without query operators, only results will be returned that contain all of the search terms in the Tweet, username or hyperlink. This means adding a term to your query will not lead to more search results.
Only if you remove the original query terms, other results will be returned.
Observations We have not had the chance to evaluate our application through means of a user study. However, we do want to point out the following observations. Given the nature of our data, i.e. a collection of tweets, there might be some improvements possible that exploit this particular type of data. Tweets can contain special elements in the text, namely usernames, hashtags, links, and emoticons. We make the following observations: • Usernames and hashtags are currently considered in the sense that we remove all punctuation except the characters ‘@’ and ‘#’ which are the indicators of usernames and hashtags respectively. There is an option to generate wordclouds containing only usernames, or only hashtags. In the default settings usernames and hashtags are included as is in the wordcloud. For future work we want to discuss and investigate two open issues: 1. Can a word with a hashtag be considered as the same word without the hashtag? While a hashtag term does not always have to be a real word, e.g. #london2012, in many cases it is, e.g. #london. For the wordcloud should the terms ‘london’ and ‘#london’ be merged? Sometimes usernames are used in a similar way as hashtags to address companies, e.g. in this tweet: ‘Ambush marketing at the Olympics! Well played, @Nike.
bit.ly/N4zAUc #London2012’. 2. A related issue is the importance or term weights of usernames and hashtags. Is a hashtag a stronger signal, and should it therefore be featured more prominently in the wordcloud? Similarly for usernames, but usernames could also be considered a weaker signal, so should they be featured less prominently? Both of these questions can also be considered when you want to optimize a retrieval algorithm. • Besides the ‘@’, and ‘#’ all other punctuation is removed during text preprocessing. This means all emoticons like ‘:)’ are removed. Sometimes these emoticons are used as indicators of sentiment, i.e. tweets containing ‘:)’ are classified as positive messages, and tweets containing ‘:(’ as negative messages. In this sense the emoticons do indeed represent valuable information that could be included in the wordcloud.
When an emoticon appears in the wordcloud, clicking on it can give you all the messages associated with for example a positive emoticon.
Feedback from users is required to determine the most useful improvements for the application. 4
Conclusions
results. Wordclouds are a quick way to summarize and get a first
overview of large amounts of data. Using human observation skills
it is easy to zoom in on a group of messages in which you are
interested, i.e. all messages that contain a specific term from the
wordcloud. In future work we would like to evaluate the usefulness of
wordclouds for navigation and summarization of search results in a
user study.
5
[
Tag clouds for summarizing web search results. Proceedings of the 16th international conference on World Wide Web WWW 07, 196:1203, 2007.
Do users benefit from controlled vocabularies in search interfaces?
Ying-Hsang Liu School of Information Studies
Charles Sturt University Wagga Wagga, NSW 2678,
Australia yingliu@csu.edu.au
Paul Thomas
CSIRO
GPO Box 664 Canberra, ACT 2601, Australia paul.thomas@csiro.au
Tom Gedeon Research School of Computer
Science Australian National University Canberra, ACT 2601, Australia tom@cs.anu.edu.au
Jan-Felix Schmakeit Research School of Computer
Science Australian National University Canberra, ACT 2601, Australia
janfelix.schmakeit@anu.edu.au ABSTRACT Search providers in domains from medicine to news have long labelled documents with controlled vocabularies, to help users explore their collections. These vocabularies are expensive to build and use, however, and seem to be useful mostly for domain experts.
This paper describes an on-going gaze-tracking study which asks whether users notice controlled vocabularies when they are exposed in a search interface; whether they make use of them; and whether this improves search. We also hope to learn what e↵ ect several standard search interfaces have on the use of controlled vocabularies.
Categories and Subject Descriptors H.3.3 [Information Search and Retrieval]: Search process; H.5.2 [User Interfaces]: User-centered design—performance measures General Terms Experimentation, Human Factors Keywords Search results presentation, individual di↵ erences, gaze behaviour, MeSH terms 1. INTRODUCTION
It has been recognised that people engage with di↵
erent kinds of searching behaviours, but current information
Presented at EuroHCIR2012. Copyright c 2012 for the individual papers
by the papers’ authors. Copying permitted only for private and academic
purposes. This volume is published and copyrighted by its editors.
retrieval (IR) systems are primarily designed for specified
search [
To this end researchers have advocated “natural” search
user interfaces, arguing they are easier to use and require less
user training [e.g. 9, 20]. It is however challenging to design
natural interfaces because of the complexity of information
problems and associated searching behaviours. For instance,
user studies have demonstrated that user queries are typically
very short representations of complex information needs [
One way to support query formulation is with a controlled indexing language, where each document is assigned terms from an predefined list or hierarchy of indexing terms. Examples include Medical Subject Headings (MeSH) terms and Library of Congress Subject Headings (LCSH). The usefulness of MeSH terms in biomedical searching is especially important because of the extreme popularity of the PubMed database1, the publicly accessible version of MEDLINE on the web.
Controlled vocabularies are expensive to build, use, and
maintain, and they may contribute to clutter in a search
interface. There is some evidence that domain experts benefit
from controlled vocabularies, but results have been mixed
for ordinary users (e.g., [
1http://www.ncbi.nlm.nih.gov/pubmed
This paper describes an on-going eye-tracking study of user gaze and search behaviours searching clinical search topics, with particular reference to the user’s attention to and use of the document surrogates (i.e., MeSH terms, title and abstract). The specific research questions are: 1. What components of document surrogates do searchers look at when reformulating their queries? Do searchers even notice MeSH terms in standard search interfaces? 2. If they do notice them, how do searchers use the dis
played MeSH terms in their search processes? 3. If they are used at all, do MeSH terms lead to better
search performance and e ciency?
RELATED WORK
Past work has considered system designs to support query
reformulation. From a system perspective, researchers have
proposed visualizing document inter-relationships [21],
explicit term distribution information [
Recent HCI and IR research has focused on users’ cognitive
aspects in search interactions by measuring the gaze patterns,
an indicator of searcher attention (see e.g. Dumais et al. [
We are conducting a user experiment to assess the e↵ ect of displayed MeSH terms on search behaviors and performance.
The search task is to perform searches on clinical information for other patients, and find the best query to obtain as many relevant documents as possible. Our recruits are undergraduate and postgraduate students with search engine experience but without advanced academic background in the biomedical domain. Each user searches 8 topics in total, with a 7-minute limit for each topic, and the experiment takes about 90 minutes in total.
Participants are given brief instructions about the search task and system features, followed by a practice topic and then the searches proper. User interaction data is recorded: we are noting all queries, mouse clicks, retrieved documents, time spent, and eye movements. Electroencephalogram (EEG) readings are also captured. (a) Screenshot of Interface “B”, suggestions per-query and displayed at top. (b) Screenshot of Interface “D”, suggestions per-document and displayed with the document.
Participants search on four di↵ erent search interfaces using a single search system. The four search interfaces are distinguished by whether the MeSH terms are presented and how the displayed MeSH terms are generated: Interface “A” mimics web search and other search systems with no controlled vocabulary. This interface has a brief task description at top; a conventional search box and button; and each result is represented with its title, authors, publication details, and abstract where available.
Full text is not available, so the results are not clickable. Users must judge their success on the titles and abstracts alone.
Interface “B” (Figure 1(a)) adds MeSH terms to the interface. After the user’s query is run, MeSH terms from all results are collated; the most frequent ten are displayed at the top of the screen. This mimics the per-query suggestions produced by systems like ProQuest2.
MeSH terms are introduced with “Try:” and are clickable: if a user clicks a term, their query is refined to 2For example, see http://www.proquest.co.uk/en-UK/ products/brands/pl_pq.shtml Imagine that you are 63-year-old male with acute renal failure probably 2nd to aminoglycosides/contrast dye.
You would like to find information about acute tubular necrosis due to aminoglycosides, contrast dye, outcome and treatment.
include the MeSH term and then re-run. We hope that the label, and the fact they work as links, will encourage users to interact with them.
Interface “C” uses the same MeSH terms as “B” but displays them alongside each document, where they may be more (or less) visible. It is a hybrid of interfaces “B” and “D”.
Interface “D” mimics EBSCOhost3 and similar systems that provide indexing terms alongside each document.
As well as the standard elements from interface “A”, interface “D” displays the MeSH terms associated with each document, as part of that document’s surrogate (Figure 1(b)).
Again, terms are introduced with “Try:” and are clickable.
Each interface is labelled with a simple figure—a square, circle, diamond, or triangle—which we refer to in our exit questionnaire. 3.2
Design
This experiment is a 4 ⇥ 4 factorial design with four search
interfaces and four topic pairs. We are using a 4 ⇥ 4
GraecoLatin square design [
We expect to enroll 32 participants from the campus of a large university, which will give good statistical power (when N = 32, ANOVA < 0.01 for “medium” e↵ ect of = 0.75).
Entry and exit questionnaires are collecting demographic information and information on participants’ cognitive styles and their perception of the search process. We also ask participants’ opinions of the tasks and the interfaces. 3.3
Topics
Search topics used here are a subset of the clinical topics
from OHSUMED [
We selected topics to cover a range of di culties: we sorted the topics according to the number of judged relevant documents and selected two topics, at random, from each quartile. These eight topics were then randomly paired o↵ to produce four pairs of topics. A final topic, the same for all participants, is used for training. 3.4
Software and hardware
The search system is built on Solr4, with the search results ranked by default relevance score. The MeSH terms are not specifically weighted. 3http://www.ebscohost.com/ 4http://lucene.apache.org/solr/
Gaze tracking uses FaceLab5 software and hardware. We use Eyeworks software6 for recording and basic analysis.
EEG data is recorded with an Emotiv headset7.
Where do people look?. Recordings will be analysed to see how often there are fixations in di↵ erent parts of document surrogates, and therefore how often people have looked at each part. In particular, for interfaces B, C and D we will consider how often participants look at the controlled vocabularies (“Try:. . . ”). Any e↵ ect on gaze patterns due to interface would tell us which interfaces make the extra information easiest to discover.
Our exit questionnaire also asks whether users noticed the controlled vocabularies: we would not be surprised if there were di↵ erences between the self-reported data and the gaze data, for example if participants were trying to please us.
Do they use the controlled vocabulary?. Our software records all clicks on terms from the controlled vocabulary, so it will be easy to note how often it is used and whether there is any correlation with interface, task, sequence, or user. Again, an e↵ ect due to interface would suggest which style of interface makes features like the controlled terms most attractive.
Participants who merely read and re-type the controlled vocabulary may be picked up in query logs.
Again, we intend comparing these recordings with selfreports.
If so, does it help?. Assuming some participants do make use of the MeSH terms, we anticipate four ways to address this question. First, as before, we will consider self-reports of task di culty to see whether these correlate with the use of controlled vocabulary features. Second, since participants’ final queries on each topic should be the ones the like best, we can check how many of these use MeSH terms. Third, the judgements associated with OSHUMED topics will allow us to measure the actual e↵ ectiveness of queries with and without controlled terms. Finally, if participants do not use all their allocated time for each task, variations in completion time may be interesting.
FIRST RESULTS AND NEXT STEPS
We have conducted a small-scale pilot to test our design and instruments.
Our participants did glance at MeSH terms: 8% of fixations were on MeSH terms in interfaces B to D, which compares to 6% on document titles and 12% on abstracts. However, they were very seldom used – only one query, of 44 queries issued on these interfaces, used any MeSH terms at all. There are also some indications of a per-interface e↵ ect, with the MeSH terms at the top of interface D receiving little attention We will shortly be recruiting for the full-scale experiment.
We hope this will o↵ er some insight into the relationship 5http://www.seeingmachines.com/product/facelab/ 6http://www.eyetracking.com/Software/EyeWorks 7http://www.emotiv.com/ between interface, reading patterns, search behaviour, and search e↵ ectiveness.
ACKNOWLEDGMENTS
Ying-Hsang Liu has been supported by the School of Information Studies Research Fellowship from Charles Sturt University and working as Visiting Fellow at Research School of Computer Science, The Australian National University. User-Centred Design to Support Exploration and Path
Creation in Cultural Heritage Collections 1 1 1 1 1 Paula Goodale, Paul Clough, Nigel Ford, Mark Hall, Mark Stevenson, Samuel Fernando, 1Nikolaos Aletras, 2Kate Fernie, 3Phil Archer, 4Andrea de Polo 1
1University of Sheffield, Sheffield, United Kingdom p.goodale│p.d.clough│n.ford│m.mhall│m.stevenson│s.fernando│naletras1@sheffield.ac.uk 2MDR Partners, United Kingdom; 3iSieve Technologies, Greece; 4Alinari 24 ORE Florence, Italy 2kate.fernie@mdrpartners.com, 3phil@philarcher.org, 4andrea@alinari.it ABSTRACT In this paper, we present the results of the user requirements and interface design phase for a prototype system, designed to enhance interaction with cultural heritage collections online through means of a pathway metaphor. We present a single user interaction model that supports various work and information seeking tasks undertaken by both expert and non-expert users within the context of collection exploration and path creation. The user interaction model is shown to enable seamless movement between interaction modes, with the potential over time to encourage deeper engagement and learning.
Categories and Subject Descriptors
H.5.m [Miscellaneous]: Interaction framework
General Terms
Design, Human Factors, Theory.
1. INTRODUCTION
Large-scale projects for the digitisation of cultural heritage (CH)
have become commonplace in recent years, and yet complex
issues arise with regard to information access. Specialist metadata
and the often variable quantity and quality of object descriptions
make it difficult for users to navigate vast, structured and often
very scholarly collections. It is therefore difficult to locate
resources of interest, especially for those without advanced levels
of subject and domain knowledge [
Guided tours and activity trails are commonplace offerings to aid
visitor orientation at physical cultural heritage sites, and offer a
Presented at EuroHCIR2012. Copyright © 2012 for the individual
papers by the papers' authors. Copying permitted only for private and
academic purposes. This volume is published and copyrighted by its
editors.
range of opportunities for immersive and more highly engaged
visitor experiences [
Online paths and trails are seen as a means of aiding navigation,
exploration and learning [
Expert users regularly engage in both simple fact-finding and
more complex information gathering tasks, amongst others, with
the latter having multiple variations and components such as topic
searches, exploration, collecting/combining [
For known-item or fact-finding searches, some knowledge of the
metadata and collection structure is imperative, but such
knowledge is much less likely to be used effectively, if at all by
non-expert users than expert users [
Exploratory search extends the idea of basic lookup into the areas
of learning and investigation, which in turn incorporate extended
information processing, evaluation and annotation [
Develop a detailed understanding of the characteristics and needs of potential users across four primary domains: heritage, education, professional/commercial, and general/leisure.
Explore the meanings and potential applications of the path metaphor in the context of digital CH.
Gain an understanding of the path-creation process and the types of paths that might be created.
Determine the current availability and functionality of path-creation tools in CH collections.
In order to achieve these goals, mixed methods were employed, gathering a variety of complementary qualitative and quantitative data. First, an online user survey was used to collect data from 79 expert and non-expert users, comprising questions about their personal and cultural participation characteristics, and information behavior and use in the CH context. This was complemented by in-depth semi-structured interviews conducted with 22 expert users, which focused on exploring the meaning of the path metaphor in CH environments, and understanding the process of development and use of paths in this context.
Secondary data was used to scrutinise the features of published paths from various sources, to ascertain their core elements.
Similarly, a comparative analysis of general and cultural-heritage specific systems offering path-creation functionality was conducted, to identify common features and standard approaches to the proposed core functionality. These findings were validated via user participation in path-creation tasks, utilsing low- and medium-fidelity techniques.
Analysis of these various complementary data enabled the development of detailed domain and role-specific information user profiles; a user interaction model supporting four key modes of interaction; and, use cases illustrating some of the primary user interaction scenarios. From these we extrapolated detailed user requirements, and in turn, interface designs and functionality for the first PATHS prototype. The resulting system is intended to support all elements of the interaction model, allowing users to move seamlessly between modes of use. 4. RESULTS Given the breadth and depth of data, this paper focuses on the findings relating to paths and their uses in CH, and in turn, how related user tasks are incorporated within a single user interaction model, to be implemented in the prototype system. 4.1 Existing Path Forms Analysis of existing paths and trails found that online and offline paths both have similar characteristics. Nodes are the essential building blocks of all paths, representing collection objects. Each node has associated metadata and primary content (e.g. descriptions, images) relating to the object. Connections between nodes enable navigation through the path and often represent meaningful relationships between objects. In the online environment, additional features of paths included navigation tools (e.g. path overviews and back/forward arrows), annotations added by the path creator to give context and guidance for use, and occasionally links to other related content, both within the same collection, and/or in external web sites. These findings largely support the initial vision for PATHS and can all be seen in the first prototype design.
In addition, it was found that most existing online paths are static and pre-published by an author, linear in form, rather than a more complex map or network structure, and standalone, without interconnections with other paths. These findings fall somewhat short of the PATHS vision, limiting the possibilities for exploration and discovery, although for pragmatic reasons, they form the core functionality of the first prototype, with more advanced variations of paths coming later. 4.2 The Path Metaphor in Cultural Heritage Interviews with potential expert users in the heritage, education and professional domains found a strong affinity with the path metaphor, revealing a range of different interpretations of what it means in the CH context, and similarly about what form paths might take, and how they could be employed in an online environment to engage with key audiences. Eight interpretations of the path metaphor emerged:
Path as information literacy journey / learning process
Path as transaction process
The first three of these are closest to the idea of hypertext trails
[
Path as search history Path as information seeking journey Path as linked metadata Path as a starting point or way in Path as a route through Path as augmented reality path-creation tools currently available. Linearity is rarely seen as the best option for maximizing the potential of paths as exploration devices. Allied to this is the belief that starting and end points for paths should be mutable rather than fixed, allowing different users to explore a path in different ways according to their preferences and needs.
In the absence of linearity, some form of organization is still required to aid the accessibility and navigation of the path. The most popular option is for path content to be aligned to themes, with other alternatives including date, location, narrative and author, where the latter might present multi-layered paths offering the differing perspectives of several path-creators on the same topic. An over-arching conceptual framework for the path is also desirable, in order to tie together the themes and other ideas.
As a way-finding or navigational aid, paths are seen to support both guided and exploratory behavior, with the latter seen as the more desirable goal for user interaction. Features that are needed to enable way-finding include path overviews, navigational context in the form of next/last and nearby nodes, branching opportunities where paths converge and diverge, visualization, e.g. in the form of timelines or maps, and some degree of object level information at the node and overview display.
Path content must be carefully selected or ‘curated’ by the path creator, with the addition of context and interpretation so that the objects within the path convey a narrative or meaning. Content may be derived from one collection, but there are significant benefits from including objects from diverse collections, along with other materials from external web sites. It may also be beneficial for interpretation of the path content to be extended by user-generated content and/or annotations of various kinds.
Many of these characteristics are seen in existing path systems, but limitations arise from the linearity that is commonplace.
Exploration and deeper levels of engagement within collections requires more complex path structures, carefully curated content, interpretation and narrative, and interconnectedness of paths and other content within and outside of the system. The fact that most of these more advanced characteristics are rare, and that linearity prevails also suggests that these are complex issues yet to be adequately resolved. 4.4 Potential Applications of Paths Many opportunities for the use of paths in CH were suggested.
Two major themes emerging from these are the use of paths to achieve learning, and to support exploration and browsing. For learning to occur there needs to be strong contextual information, along with questions and other exercises to structure the learning process. Exploration and browsing activities implicitly enable meaning making and learning to take place, as users become more familiar with a topic and select or interpret the objects they encounter.
Specific instances of learning activities that may be delivered via paths are collection or subject familiarization, story-telling, individual or collaborative inquiry-based learning utilising path creation, modeling the research process, and comparative analysis of differing view-points on a topic of interest.
In addition to learning,, paths may also serve to deliver
entertainment and an enjoyable interaction experience for more
general audiences. In practical terms, paths may simply be used as
a means of introducing people to a collection and its stories, and
in due course, encouraging them to venture further in a more
independent fashion. Paths facilitate topic-based information
retrieval typified by the berry-picking mode of interaction [
Findings from the qualitative data collected via interviews and path-creation tasks revealed a set of five core activities relating to the creation, use and sharing of paths; developing a concept for a path; collecting items in include in a path; creating a path from items collected; communicating about paths found and about paths created; and, consuming (following or exploring) paths created by others. All elements of the model may be undertaken by expert and non-expert users, in any sequence, and with varying degrees of iteration, according to the user’s preferences and behavioural traits.
Figure 1. PATHS user interaction model.
Initially, we expect users to begin by Consuming paths created by others, using them as a means of exploration and familiarization with the collection and the system. Collecting items of interest when exploring a collection is a natural behaviour in berrypicking mode, and is implicit in the process of creating a path, or as a by-product of a user’s information seeking history. When the path creation activity is purposeful, it is likely that an overarching Concept is devised, which may come from activities undertaken outside of the system, but also may be developed via a process of exploration within the collection and any pre-existing paths. The concept may also evolve alongside the collection and path creation activities, through a process of iteration and meaning-making. A path is Created once a number of appropriate items have been collected, and this activity may include ordering the items into a narrative, and adding contextual information and/or metadata. In a web 2.0 environment, it is also important to allow for Communication activities in support of the interaction experience. These may include sharing paths that have been created or discovered, both within and outside of the system (e.g. via social media), commenting on and rating content, and adding narrative to personal paths as a means of making meaning.
It is imperative in an adaptive web environment that systems do not prescribe modes of interaction or enforce sequences of activities. During the design of PATHS we have uncovered four primary interaction modes, all of which are supported by the user interaction model, but each with a somewhat different typical interaction flow.
Path consumers are the most passive users, and likely to be in the majority. By using paths as a guided tour or means of simple exploration of the collection and its content, we expect users to become more interested in communicating their discoveries with others and exploring further within the main collection. Over time we would expect some of them to move onto collecting and creating paths of their own, as they develop into more independent and active users of the system.
Path creators will likely be in a minority in the early stages, and primarily expert users such as curators and educators, and perhaps a few more independent non-expert users. In expert path-creation mode we believe interaction will be purposeful and systematic, with a goal of creating a path about a defined topic. Topics and styles of paths may vary by domain, and we expect that educators are more likely to adapt ideas from existing paths, whilst CH experts will try to develop something novel, showcasing elements of a collection or subject expertise. In contrast, non-expert path creators are more likely to develop their concept as they explore the collection, and their paths may be more idiosyncratic, evolving over time, or in the education domain, may even be directed in the task by an expert in a path facilitator role.
Path facilitators are most likely to be found within educational
settings, where inquiry-based learning is prevalent. These users
may not create paths themselves, but may curate a broad
collection of objects from which a group of non-expert users are
encouraged to create their own paths. (for instance, as a
homework project). Facilitators are more interested in enabling
deeper engagement with CH materials, and in fostering
communication and reflection on the activity and the content of
the paths created in this way.
6. CONCLUSIONS
We have presented the findings of our user requirements study on
the creation and use of paths as a means of aiding information
access and exploration in CH digital collections. It has been
shown that paths support many of the needs for exploratory
information behavior, and have applications for diverse users
across multiple domains. Users interactions with paths comprise
five core elements, integrated into a single user interaction model
and can be used in varying sequences, illustrated by four primary
modes of interaction.. An initial prototype has been developed
from the user interaction model, which is currently being
evaluated within a task-based user-centred evaluation setting.
7. ACKNOWLEDGMENTS
The research leading to these results has received funding from
the European Community’s Seventh Framework Programme
(FP7/2007-2013) under grant agreement no. 270082. We
acknowledge the contribution of all project partners involved in
PATHS (see: http://www.paths-project.eu).
8. REFERENCES
[
Supporting Serendipitous and Focused Search
Junte Zhang Meertens Institute, Royal Netherlands Academy of Arts and Sciences
Amsterdam, the Netherlands ABSTRACT People with complex information needs are for example Humanities researchers, who need advanced search engines to investigate their research questions. Much can be gained by combining research datasets, reusing tools and serendipitously discovering new insights for further research. Humanities researchers have di↵ erent (large-scale) research datasets and tools, which are described di↵ erently with metadata.
We present a highly interactive advanced search engine for Humanities researchers that semantically converges di↵ erently structured metadata records from di↵ erent collections and institutions. It has features that support serendipitous and focused search in context based on the structure of the metadata used. This single system serves Humanities researchers by allowing them to search interactively across yet unexplored (research) data, discover patterns, locate relevant data for new insights, and find existing tools that could provide novel use cases.
Categories and Subject Descriptors H.3.3 [Information Search and Retrieval]: Search process; H.3.7 [Digital Libraries]: Systems issues, user issues; H.5.2 [Information interfaces and presentation]: Graphical user interfaces (GUI) General Terms Design, Human Factors Keywords information retrieval, metadata, user interfaces, ehumanities 1. INTRODUCTION
The Common Language Resources and Technology
Infrastructure (CLARIN) initiative seeks to establish an
integrated and interoperable research infrastructure of language
Presented at EuroHCIR2012. Copyright c 2012 for the individual papers
by the papers’ authors. Copying permitted only for private and academic
purposes. This volume is published and copyrighted by its editors.
resources and its technology.1 Descriptive metadata is used
to characterize large number of (legacy) research data
resources (collections) and tools (e.g. Web services) to
facilitate their management and discovery. The Search & Develop
(S&D) project within CLARIN in the Netherlands uses the
Component MetaData Infrastructure (CMDI; [
How to use the CMDI framework with ISOcat to search
for data and services, which can be understood by both
people from varying disciplines and machines? The challenge is
that the data is heterogenous both in content and
structure, and can be massive in amount. In [
USING CMDI FOR FOCUSED AND SE
MANTIC ACCESS
CMDI has grown out of the need to facilitate access,
reuse, and interoperability using metadata [
The DCR was established by the ISO Technical Committee
37, Terminology and other language and content resources
based on the ISO 12620:2009 standard. Because multiple
elements may refer to the same DC, semantic interoperability
can be achieved across di↵ erent datasets. A specification
using the DCR and projected for example in an XML schema
is called a metadata profile and can be (re)used for
describ1See
http://www.clarin.eu/external/index.php?page=aboutclarin
(a) Query autocompletion based on the count that a query (b) The selection widget that allows users to keep overview of
occurs in a tag within the result set. By default the query box the search trail and change it, while updating the result list.
is content-centric, but searching directly in a tag is possible Here, the query stored is “periode” (period) within the tag
with Advanced Search (can be collapsed with a click). Users time coverage!description. Interesting terms are suggested
can express queries using the metadata or only the fulltext by presenting the top TF⇤ IDF terms, which people can use
of the document by discarding autocompletion. to start a parallel search episode.
(c) To further support query expansion and serendipitous in- (d) The distribution of retrieved time-referenced documents
formation seeking, a dynamic tag cloud is generated based (given the tags Century of Publication and Year of
Publicaon the last retrieved result list and used metadata label with tion) are visualized in bar or line charts. Users can click in
keyword highlighting. Moreover, retrieved geo-referenced the charts to narrow down the result set. The distribution of
documents are projected on a map and clustered by markers. results in tags collection and schema profile always appear.
(a) Retrieved list of results with the display of the list of
results with ‘fixed’ contextual information, snippets and
keywords in context within the last searched metadata label and
the presentation of all used keywords in context given the
fulltext. There is links to the fulltext of the metadata record
and the actual resource in the digital archive.
(b) For each retrieved result in the list, there is a
recommendation (when available) of related results based on the
content similarity of the last used metadata label. A
recommendation consist of a link to the record, the collection it
belongs to, and a snippet (can be collapsed with a click).
ing datasets and for eventual access. Moreover, RELcat [
We have indexed 246,728 CMDI files from 18 di↵ erent
profiles consisting of 143 di↵ erent types of elements in a single
stream, which shows our indexing method for CMDI files is
robust enough to deal with complex data [
When a user with no a priori intentions interacts with a
node of information and acquires useful information, then
serendipitous information retrieval occurs [
Since the CMDI MI Search Engine should serve Humanities researchers, we design it to support serendipitous search and be highly interactive. The system has been designed to maximize the user’s ability to explore. This is our focus.
The user interface of the system is depicted in Fig 1. It uses
the JavaScript library AJAX Solr2, which has been
heavily modified and extended by us with JQuery. It allows for
faceted search [
A user can improving the search episode (session) by effectively reducing the information space step by step. These steps are stored as part of the search trail, so the overview is kept. There are di↵ erent search strategies possible. Users can search by fulltext by entering a query. This makes sure users can always search in everything. The query get highlighted in context given the fulltext, but the dynamic tag cloud widget that supports query expansion is not activated, see Fig.1(a). Users can also do a focused search request by using structure, i.e. within the content of a specified tag, and get the content of these tags returned. This can be content-centered, as users enter a keyword and the autocompletion widget returns a list consisting of keyword plus field name and hit count. It can also be structure-centered (using the Advanced Search option) by looking up a tag and then entering a keyword also with the autocompletion feature. When the last two options are used, then the keyword highlighting also occurs within the context of the retrieved snippets of the searched tag, see Fig.2(a).
A challenge is how we can support serendipitous search given the diversely structured metadata in CMDI. Hence, we introduce and propose the concept of serendipitous search in context. We can use the heterogeneous structure of di↵ erent collections to provide context to the user in a single search engine. We propose the following contextual system features that aim to support serendipitous and focused search.
• Help users by automatically completing the query that the user is entering while simultaneously and directly giving the hit count for the suggested queries in conjunction with a tag, see Fig.1(a). • Provide inline suggestions (Did you mean...) based on
a spell checker whenever applicable. • Suggest a new parallel search episode (You could also look for...) by presenting interesting terms based on the content of the first few retrieved results after each used query, see Fig.1(b). This increments and becomes more focused as a search episode gets more queries. • O↵ er di↵ erent overviews of the retrieved results and allow for query expansion by directly presenting a dynamic tag cloud of the aggregated content within the metadata label used and highlighting the query entered in this context, see Fig.1(c). • Preserve the overview of a search episode by storing the search selection (see Fig.1(b)), and the overview on collection level by the result type, e.g. the metadata profile ‘lied’ (song) in the Dutch Song Database, and the collection a document belongs to (see Fig.1(d)). • Aggregate and visualize collection-specific search features in extra widgets, such as projecting and clustering the list of retrieved geo-referenced resources on a map (see Fig. 1(c)), and displaying the date ranges of the documents in charts that can be clicked to narrow down a result set (see Fig. 1(d)). • Entice users to explore further by recommending related resources using the content similarity by presenting a link to the metadata record and a snippet of a recommendation, see Fig.2(b).
So the context consists of di↵ erent modalities and features existing in the structure of the metadata of a collection, and used in the retrieval and visualization of information. This can be displayed on a aggregated level based on the set of retrieved results. And it can be displayed with di↵ erent displays of the result types given the metadata profile. Eventually, the user finds the links to the resources in the digital archive using the metadata, and can use the found resources for further research or development. However, there is no real definite end of the search episode as people still can continue searching using the above proposed system features.
We have presented a working proposition for serendipitous and focused search by describing the CMDI MI search engine. The novelty is that it provides semantic access to diversely structured language and digital heritage resources with di↵ erent metadata schemas for users such as researchers with very specific and complex information (research) needs.
The search engine provides faceted search and has serendipitous features that maximize the user’s ability to explore any metadata in CMDI in context, such as query autocompletion, tag clouds, and recommendation of related resources, while keeping track of the search trail. It is a tool that provides interactive and focused access to heterogeneous metadata, gives new perspectives on legacy (research) data and tools, and provides new insights for research and development. It has been released as live, and can be used at www.meertens.knaw.nl/cmdi/search.
ACKNOWLEDGMENTS
This work is part of the Search & Develop project at the Meertens Institute, and funded by CLARIN-NL.
Marcus Nitsche Faculty of Computer Science, Otto-von-Guericke-University,
Germany marcus.nitsche@ovgu.de
Andreas Nürnberger Faculty of Computer Science, Otto-von-Guericke-University,
Germany andreas.nuernberger@ovgu.de When users search for information in domains they are not familiar with, they usually struggle to formulate an adequate (textual) query.
Often users end up with repeating re-formulations and query refinements without necessarily achieving their actual goals. In this paper we propose a user interface that is capable to offer users flexible and ergonomic interaction elements to formulate even complex queries simple and direct. We call this principle vague query formulation by design. By this formulation we like to point out its design-driven origin. The proposed radial user interface supports phrasing and interactive visual refinement of vague queries to search and explore large document sets. The main idea is to provide an integrated view of queries and related results, where both queries and results can be interactively manipulated and influence each other. Changes will be immediately visualized. The concept was implemented on a tablet computer and the usability was stepwise evaluated during a formative and a summative user study. The results reveal high usability ratings, even if the concept was completely unknown to our test users.
Keywords Search User Interface, Query Reformulation, Query Refinement, Information Retrieval.
Categories and Subject Descriptors H.3.3 [Information Storage and Retrieval]: Information Search and Retrieval.; H.5.2 [Information Interfaces and Presentation]: User Interfaces.
General Terms Design, Human Factors, Management.
INTRODUCTION
When users try to handle complex information needs they often
end up in conducting exploratory searches [
In this work we like to tackle this problem of querying
appropriate queries by offering users dynamic user interface (UI) elements
that they can manipulate directly by touch gestures to give them
a feeling for a certain query configuration that matches a certain
result set. Thereby learning and exploring aspects will covered as
well [
User-specific context aware data filtering is not a new challenge.
In the following we show two tools, that can also be used for these
application domains. The VIBE-system [
This system follows the principle of dust-and-magnet [
But in contrast to our proposed solution it is divided into different UI elements and different views. It is less integrated in a single view. Therefore user’s work load might be higher since he needs to face various mode switches. Commercial systems, like the Vis4you concept1, are more focused on visualization than on interaction via direct manipulation. Furthermore, this system is designed to be used on desktop computers with a mouse (single point and clickprinciple), no multi-touch-support. In the next section we like to present our concept in more detail.
CONCEPT & DESIGN
Due to the increasing amount of data and complexity, it is
necessary to apply and improve the concepts of visual information
filtering and retrieval. This goes along with the underlying methods and
tools. Considering clustering algorithms (e.g., k-means [
The concept follows the idea that more relevant data are centred. Note, this is equivalent to filtering an overcrowded desktop, cf. Fig. 3 (left picture)2, where the more centralized documents are possibly more important (highlighted in the right picture).
The system was designed to be a multi-user system. Therefore
a number of multiple users need to be supported at the same time,
also considering security aspects [
We offer users a dimension merging according specified weights, similar to the result listing of search engines, where also different 1http://www.vis4you.com/vis4you/ (accessed on 04.07.2012) 2http://lawprofessors.typepad.com/ (accessed on 04.07.2012) weights can be linked to specific query terms (Fig. 2). Data points represent the data space. Query objects, so called widgets, can be entered via a virtual keyboard and can also be dragged by the user to formulate more complex or vague queries. Selecting a specific data point supports the user with additional information on this data point and highlights all related data points.
The distance of a certain term is directly connected to its
importance for the user. In other words, if a user thinks a specific term is
more relevant to its actual filter-/search-task, she or he positions the
corresponding UI-element more to the center, which influences the
weight of this term when computing its Term Frequency / Inverted
Document Frequency (TF/IDF)-value [
Result elements are placed near to corresponding query elements.
The formula for calculating the relevance of a SearchResult object (result dot) is shown in Fig. 5.
Figure 5: Relevance of a SearchResult object.
The calculated relevance determines the distance to the center, considering further result objects.
To address various types of end devices such as multi-touch desktops or mobile interfaces with large displays, we use direct manipulation as a central interaction paradigm. Only the relative distance of an UI element to the center is relevant for the system. Thus, we provide users with a direct linking to the data they like to filter.
By this interaction concept, we propose to achieve more precise results. Additionally, we support users with the concept of Whatif -queries, which supports a fault-tolerant interaction system, using a ghosting technique: Dragging an element and holding it on a specific position triggers the system to show the user how many items are in the center point of interest (POI) after releasing the element.
Thereby, users are able to explore the impact of possible next steps.
Changes of the query configuration also effect the data points
to provide the user with a direct link to the data (interactive
visualization). By the underlying metaphor of magnets, we offer an
integrated feedback, comparable to Dust-and-Magnet [
Features
The UI supports direct feedback since the relevance value is simultaneously shown while users interact with the widget (Fig. 7).
Results, corresponding to a specific query object are visually highlighted and grouped to each other (Fig. 8).
Detailed information on particular result objects, like a website preview, is provided after clicking on the result dot (Fig. 9).
Since this contribution is basically driven by fields of human
factors and user interface design, we are using common methods
from these research areas. Such as user centred design (UCD)
processes [
To proof the concept of the proposed user interface, a prototype
was implemented. This was done by using an Apple iPad.
ThereFigure 9: Prototypical search result popover as a website
preview feature, here a result for ’Labrador Retriever’.
fore the application was written in ObjectiveC using the xCode3
environment. The backend architecture is the CARSA system [
The evaluation concept followed a formative evaluation
principle where several usability testings were conducted. Also in
parallel to the development process: To identify at least 85% of all
usability issues this mock-up was evaluated according to Nielsen
and Landauer [
DISCUSSION & OUTLOOK
We described a newly designed user interface for filtering, exploring and managing data via direct manipulation supporting multiple reference systems to support context sensitive interaction techniques. We proposed a UI concept for visual filtering, that is • flexible: parameters can be adapted or enhanced by users • context-sensitive: initial parameters are extracted from the
current use case • easy to learn: through work environment metaphor and direct
manipulation In near the future a more detailed and larger user study will be conducted to identify further improvements of our tool and the overall concept. Also a plan to re-design slightly is already in place. 6.
ACKNOWLEDGEMENT
Part of the work is funded by the German Ministry of Education and Science (BMBF) within the ViERforES II project (no. 01IM10002B). We also thank Martin Schemmer for the implementation of the presented concept during his diploma thesis.