Search providers in domains from medicine to news have long labelled documents with controlled vocabularies, to help users explore their collections. These vocabularies are expensive to build and use, however, and seem to be useful mostly for domain experts. This paper describes an on-going gaze-tracking study which asks whether users notice controlled vocabularies when they are exposed in a search interface; whether they make use of them; and whether this improves search. We also hope to learn what e ect several standard search interfaces have on the use of controlled vocabularies.
H.3.3 [Information Search and Retrieval]: Search process; H.5.2 [User Interfaces]: User-centered design|performance measures Search results presentation, individual di erences, gaze behaviour, MeSH terms
It has been recognised that people engage with di
erent kinds of searching behaviours, but current information
Presented at EuroHCIR2012. Copyright c 2012 for the individual papers
by the papers’ authors. Copying permitted only for private and academic
purposes. This volume is published and copyrighted by its editors.
retrieval (IR) systems are primarily designed for speci ed
search [
To this end researchers have advocated \natural" search
user interfaces, arguing they are easier to use and require less
user training [e.g. 9, 20]. It is however challenging to design
natural interfaces because of the complexity of information
problems and associated searching behaviours. For instance,
user studies have demonstrated that user queries are typically
very short representations of complex information needs [
One way to support query formulation is with a controlled indexing language, where each document is assigned terms from an prede ned list or hierarchy of indexing terms. Examples include Medical Subject Headings (MeSH) terms and Library of Congress Subject Headings (LCSH). The usefulness of MeSH terms in biomedical searching is especially important because of the extreme popularity of the PubMed database1, the publicly accessible version of MEDLINE on the web.
Controlled vocabularies are expensive to build, use, and
maintain, and they may contribute to clutter in a search
interface. There is some evidence that domain experts bene t
from controlled vocabularies, but results have been mixed
for ordinary users (e.g., [
This paper describes an on-going eye-tracking study of user gaze and search behaviours searching clinical search topics, with particular reference to the user's attention to and use of the document surrogates (i.e., MeSH terms, title and abstract). The speci c research questions are: 1. What components of document surrogates do searchers look at when reformulating their queries? Do searchers even notice MeSH terms in standard search interfaces? 2. If they do notice them, how do searchers use the displayed MeSH terms in their search processes? 3. If they are used at all, do MeSH terms lead to better search performance and e ciency?
Past work has considered system designs to support query
reformulation. From a system perspective, researchers have
proposed visualizing document inter-relationships [
Recent HCI and IR research has focused on users' cognitive
aspects in search interactions by measuring the gaze patterns,
an indicator of searcher attention (see e.g. Dumais et al. [
We are conducting a user experiment to assess the e ect of displayed MeSH terms on search behaviors and performance. The search task is to perform searches on clinical information for other patients, and nd the best query to obtain as many relevant documents as possible. Our recruits are undergraduate and postgraduate students with search engine experience but without advanced academic background in the biomedical domain. Each user searches 8 topics in total, with a 7-minute limit for each topic, and the experiment takes about 90 minutes in total.
Participants are given brief instructions about the search task and system features, followed by a practice topic and then the searches proper. User interaction data is recorded: we are noting all queries, mouse clicks, retrieved documents, time spent, and eye movements. Electroencephalogram (EEG) readings are also captured. (a) Screenshot of Interface \B", suggestions per-query and displayed at top. (b) Screenshot of Interface \D", suggestions per-document and displayed with the document. Interface \A" mimics web search and other search systems with no controlled vocabulary. This interface has a brief task description at top; a conventional search box and button; and each result is represented with its title, authors, publication details, and abstract where available.
Full text is not available, so the results are not clickable. Users must judge their success on the titles and abstracts alone.
Interface \B" (Figure 1(a)) adds MeSH terms to the interface. After the user's query is run, MeSH terms from all results are collated; the most frequent ten are displayed at the top of the screen. This mimics the per-query suggestions produced by systems like ProQuest2. MeSH terms are introduced with \Try:" and are clickable: if a user clicks a term, their query is re ned to 2For example, see http://www.proquest.co.uk/en-UK/ products/brands/pl_pq.shtml Imagine that you are 63-year-old male with acute renal failure probably 2nd to aminoglycosides/contrast dye. You would like to nd information about acute tubular necrosis due to aminoglycosides, contrast dye, outcome and treatment.
include the MeSH term and then re-run. We hope that the label, and the fact they work as links, will encourage users to interact with them.
Interface \C" uses the same MeSH terms as \B" but displays them alongside each document, where they may be more (or less) visible. It is a hybrid of interfaces \B" and \D".
Interface \D" mimics EBSCOhost3 and similar systems that provide indexing terms alongside each document. As well as the standard elements from interface \A", interface \D" displays the MeSH terms associated with each document, as part of that document's surrogate (Figure 1(b)).
Again, terms are introduced with \Try:" and are clickable.
Each interface is labelled with a simple gure|a square, circle, diamond, or triangle|which we refer to in our exit questionnaire. 3.2
This experiment is a 4 4 factorial design with four search
interfaces and four topic pairs. We are using a 4 4
GraecoLatin square design [
Entry and exit questionnaires are collecting demographic information and information on participants' cognitive styles and their perception of the search process. We also ask participants' opinions of the tasks and the interfaces. 3.3
Search topics used here are a subset of the clinical topics
from OHSUMED [
We selected topics to cover a range of di culties: we sorted the topics according to the number of judged relevant documents and selected two topics, at random, from each quartile. These eight topics were then randomly paired o to produce four pairs of topics. A nal topic, the same for all participants, is used for training. 3.4
The search system is built on Solr4, with the search results ranked by default relevance score. The MeSH terms are not speci cally weighted.
Gaze tracking uses FaceLab5 software and hardware. We use Eyeworks software6 for recording and basic analysis. EEG data is recorded with an Emotiv headset7. 3.5
With the design above, we expect to answer the three questions from Section 1.
Where do people look?. Recordings will be analysed to see how often there are xations in di erent parts of document surrogates, and therefore how often people have looked at each part. In particular, for interfaces B, C and D we will consider how often participants look at the controlled vocabularies (\Try:. . . "). Any e ect on gaze patterns due to interface would tell us which interfaces make the extra information easiest to discover.
Our exit questionnaire also asks whether users noticed the controlled vocabularies: we would not be surprised if there were di erences between the self-reported data and the gaze data, for example if participants were trying to please us. Do they use the controlled vocabulary?. Our software records all clicks on terms from the controlled vocabulary, so it will be easy to note how often it is used and whether there is any correlation with interface, task, sequence, or user. Again, an e ect due to interface would suggest which style of interface makes features like the controlled terms most attractive.
Participants who merely read and re-type the controlled vocabulary may be picked up in query logs.
Again, we intend comparing these recordings with selfreports.
If so, does it help?. Assuming some participants do make use of the MeSH terms, we anticipate four ways to address this question. First, as before, we will consider self-reports of task di culty to see whether these correlate with the use of controlled vocabulary features. Second, since participants' nal queries on each topic should be the ones the like best, we can check how many of these use MeSH terms. Third, the judgements associated with OSHUMED topics will allow us to measure the actual e ectiveness of queries with and without controlled terms. Finally, if participants do not use all their allocated time for each task, variations in completion time may be interesting. 4.
We have conducted a small-scale pilot to test our design and instruments.
Our participants did glance at MeSH terms: 8% of xations were on MeSH terms in interfaces B to D, which compares to 6% on document titles and 12% on abstracts. However, they were very seldom used { only one query, of 44 queries issued on these interfaces, used any MeSH terms at all. There are also some indications of a per-interface e ect, with the MeSH terms at the top of interface D receiving little attention We will shortly be recruiting for the full-scale experiment. We hope this will o er some insight into the relationship 5http://www.seeingmachines.com/product/facelab/ 6http://www.eyetracking.com/Software/EyeWorks 7http://www.emotiv.com/ between interface, reading patterns, search behaviour, and search e ectiveness.
Ying-Hsang Liu has been supported by the School of Information Studies Research Fellowship from Charles Sturt University and working as Visiting Fellow at Research School of Computer Science, The Australian National University.