Information Retrieval (IR) has bene ted from standard evaluation practices and re-usable software components, that enable comparability between systems and experiments. However, Interactive IR (IIR) has had only very limited bene t from these developments, in part because experiments are still built using bespoke components and interfaces. In this paper we propose a exible workbench for constructing IIR interfaces that will standardise aspects of the IIR experiment process to improve the comparability and reproducibility of IIR experiments.
Information Retrieval (IR) has bene ted from standard evaluation practices and re-usable software components. The Cran eld-style evaluation methodology enabled evaluation programmes such as TREC, INEX, or CLEF. At the same time provision of re-usable software components such as Lucene1, Terrier2, Heritrix3, or Nutch4 have enabled IR researchers to focus on the development of those components directly related to their research. However, Interactive IR (IIR) as had only very limited bene t from these developments.
Typically IIR research is still conducted using a single system in a laboratory setting in which a researcher observed 1https://lucene.apache.org/ 2http://terrier.org/ 3https://webarchive.jira.com/wiki/display/Heritrix/Heritrix 4http://nutch.apache.org/
Presented at EuroHCIR2013. Copyright 2013 for the individual papers
by the papers’ authors. Copying permitted only for private and
academic purposes. This volume is published and copyrighted by its
editors.
and interacted with a participant [
In [
A number of attempts at developing a con gurable,
reusable IIR evaluation system have been made in the past.
In 2004, Toms, Freund and Li designed and implemented
the WiIRE (Web-based Interactive Information Retrieval)
system [
To achieve the goal of developing a system that ful ls
these requirements, we propose a system design that is based
around a very lean core into which the researcher can plug
the IIR components they wish to include in their experiment.
We have implemented this design in our web-based
evaluation framework ( g. 1), which complements the larger IIR
experiment support system presented in [
Web Frontend is handles the interface between the participant's browser and the evaluation workbench and is implemented using a combination of client-side and server-side functionality.
Message Bus handles the inter-component communication and forms the core of the system. It is responsible for passing messages from the Web Frontend to the IIR components con gured to be listening for those messages and also for passing messages directly between the components.
Session handles loading and saving the components' current state for a speci c participant, hiding the complexities of web-application state from the individual components. [SearchResults] handler = application.components.SearchResults name = search_results layout = grid-9 vgrid-expand connect = search_box:query
When the researcher sets up the workbench for their experiment, they can freely con gure which components to use, how to lay them out, and which components to connect to which other components. Based on this con guration the Web Frontend generates the initial user-interface that is shown to the participants. Then, when the participant interacts with a UI element ( g. 2), the resulting UI event is handled by the Web Frontend, which generates a message based on the UI event. This message is passed to the Message Bus, which uses the con guration provided by the researcher to determine which components to deliver the message to. The components that are listening for that message update their own Session state based on the message and then mark themselves as changed. After message processing has been completed for all components, the Web Frontend then updates the UI for each of the changed components.
An example of the con guration used to set-up the experiment is shown in gure 3 (from the experiment in gure 4), specifying the con guration of the \search results" component. It speci es that the component should be displayed 9 grid-cells wide (the application layout uses a 12-by-12 cell grid layout) and should expand vertically to use as much space as is available. The component is con gured to be connected to the \search box" component via the \query" message. It is this ability to freely plug components together that, we believe, makes the framework su ciently exible to support the wide range of IIR experiments, while remaining simple to set-up and use.
The core system provides only the framework into which the IIR components can be plugged. This allows the researcher to build any custom IIR UI they wish to test, while at the same time being able to take advantage of the standardised session and log handling functionality. As IIR UIs frequently include required elements that are not the focus of the study the researcher wishes to undertake, an optional set of default components for core IR UI elements is provided to reduce set-up time. This has the additional advantage that as their behaviour is consistent across experiments, the comparability of experiments using the framework is improved. 3.1
Logging provides a standardised logging interface that allows the components to easily attach logging information to the UI event generated by the participant.
The Search Box component ([
The Standard Results List component ([
The Pagination component ([
The Category Browsing component ([
The Saved Documents component provides an area where the participant can save things that they have found interesting, to support them in their current task. Documents are added through a save_document message. The Saved Documents component supports an optional tagging feature enabling the participant to tag the document with values speci ed by the researcher. This can be used to let the participant specify why they have chosen that document or how much it helps them in their current task. 3.6
The Task component provides a static display of the task information to show to the user. Two versions of this component are provided, one that displays a static text set in the con guration, and one that can fetch a task description from the database, based on a parameter passed to it.
The evaluation work-bench has so far been used to build two IIR experiments, very di erent in their nature, clearly demonstrating the work-bench's exibility.
The rst experiment ( g. 4) re-uses the standard Task, Search Box, Pagination, and Saved Documents components, and extends the Standard Results List to work with the speci c search backend. This set-up re-creates what is essentially a relatively standard search UI con guration, that is being used to investigate query session behaviour.
The second experiment ( g. 5) demonstrates a much richer interface, with more modi cations to the components and an experiment-speci c component. It re-uses the Task and Category Browsing components, extends the default Search Box, Pagination, Standard Results List, and Saved Documents components, and adds a new Item View component. The message-passing nature of the system made it possible to quickly integrate the new component, so that when the participant clicks on a meta-data facet in the Item View, a query message is sent to the Standard Results List to nd items with the same bit of meta-data. The interface was used to investigate un-directed exploration behaviour in a large digital cultural heritage collection. 5.
The stated aim of this paper was to present a novel, pluggable, extensible, and con gurable IIR interface work-bench, that supports our wider aim of improving IIR experiment comparability. The work-bench is su ciently exible to support the wide range of web-based IIR experiments that are undertaken, while being su ciently simple and light-weight to encourage wide-spread use of the workbench.
To enable this wide-spread use, the system has been released under an open-source license6. We are also moving to engage with the wider research community to determine to what degree the work-bench satis es their needs for an evaluation system and what needs to be done to achieve the wide-spread use needed to improve IIR experiment comparability. 6.
The research leading to these results was supported by the Network of Excellence co-funded by the 7th Framework Program of the European Commission, grant agreement no. 258191.