This paper presents the experiences from the INEX iTrack experiments conducted over a period of seven years. The purpose is to present the infrastructure of the experiments with the aim to identify its potential for re-use in new experiments. The paper discusses the terminology, research design, methodology, resources and reporting from the Inex iTrack in light of this.
The Initiative for Evaluation of XML retrieval (INEX) started in 2002
as a set of experiments following the Cranfield model. The purpose
of INEX was initially to test the potential of XML elements as items
for retrieval, as an alternative to full text documents, document parts
and document passages. The INEX interactive track (iTrack) was
run as a subtrack from 2004 to 2010 [
In this paper we shall investigate the methodological approach used in INEX iTrack. The intention is to explore its potential for re-use and the experience that can be of value for establishing a common methodology for interactive information retrieval (IR) experiments. The paper is structured in the following way; the ifrst part contains the method, we present iTrack infrastructure, i.e. the terminology, research design, methodology, resources and reporting used. Thereafter follows a discussion of challenges, before the final part with summary and conclusions.
In order to identify the infrastructure of the INEX iTrack we investigate the reports published in the proceedings from 2004 to 2010. The structure of the iTrack reports was kept fairly consistent across
During the iTrack years, the terminology used went through some changes. In particular, the first year (2004) stands out with an idiosyncratic terminology. Table 1 shows the distribution of central terms used over the period, compared according to their intended use, i.e. the concept (infrastructure element) they represent. This means, e.g., that from 2005 to 2010 the term "document corpus" was used consistently to refer to the collection of documents used in the experiments, whereas the term "Tasks" was used consistently from 2008 to 2010.
Table 2 provides an overview of central concepts, definitions, and the terminology where term use have changed over time. This does not represent an exhaustive overview, only concepts used over several years of experiments are included.
Although term use has changed over time, it is easy to identify the common infrastructure elements from the proceedings report. Most confusing is the diferent uses of the term "Task", which was used to refer to diferent experimental tasks in 2005 and 2006. In 2006, e.g., three diferent tasks were described as "Task A - Common Baseline System with IEEE Collection", "Task B - Participation with Own Element Retrieval System" and "Task C - Searching the Lonely Planet Collection", respectively. 3.2
The research design used in the iTrack experiments has been stable. A generic representation of the experimental procedure can be described in the following way: Concept Definition Distribution Task The task(s) assigned to Topics (2004), participants, what they tasks/topics (2005), are asked to find infor- search tasks (2006), mation about and its tasks (2008-2010) context Search sys- The system(s) designed System (2004, 2005), tem to be used in the exper- search system (2006iments 2010) Document The documents search- Document corpus (2005corpus able in the search sys- 2010)
tem Experimental The procedure used for Experimental protocol procedure performing the experi- (2004), experimental ment procedure (2005-2010) (1) General questionnaire. The participant fills out a questionnaire on background knowledge, demographic data etc. Questionnaires were on paper (2004-2006) or online (2008-2010) (2) Training task. The participant is given a training task to introduce them to the system’s design and functionalities. (3) Task 1 (a) Task specific questionnaire. The participant fills out a questionnaire on task specific knowledge (b) Search session. The participant interacts with the system in order to perform the task. (c) Post task questionnaire. The participant fills out questionnaires related to the experience with the system, dificulty in solving the task etc. (4) Additional tasks performed as described in step 3. (5) Post experiment questionnaire. The participant fills out a questionnaire to provide feedback about the search system. In addition to a common experimental procedure, the participating groups had the opportunity to perform their own experiments. In 2005 and 2006 it was explicitly organized so that research groups could use their own systems and compare their results to the system developed for the experiments as a baseline.
Very little analysis was performed as part of the iTrack work.
Studies performed on iTrack data and reported in journal articles and
conference proceedings papers have used transaction log analysis,
statistical analysis of questionnaire data, screen capturing and
eyetracking. The studies have, e.g., investigated users preference with
respect to element granularity [
The initial purpose of the iTrack was twofold: "to investigate the
behaviour of users when interacting with components of XML
documents, and secondly to investigate and develop approaches for
XML retrieval which are efective in user-based environments". In
the first two years, the iTrack was closely connected with the INEX
ad hoc-track, using the ad hoc-track’s document corpus and
topics/tasks. The tasks have been formulated as simulated work task
situations [
B Very useful and Fairly specific C Very useful and Marginally specific D Fairly useful and Very specific E Fairly useful and Fairly specific F Fairly useful and Marginally specific G Marginally useful and Very specific H Marginally useful and Fairly specific I Marginally useful and Marginally specific J Contains no relevant information 2004 2005 2006 2008 2009 2010
General/Challenging The "general" challenges were designed as simpler than the "more complex" challenging tasks Types: Decision mak- The tasks were coming; Fact finding; Infor- bined on two dimenmation gathering Struc- sions: type and structure: Hierarchical; Par- ture. allel Fact finding/Research
The tasks were designed to represent information needs typical for Wikipedia users, finding facts, such as the "biggest airport" or perform research to write a paper.
Broad/Narrow/Semi Broad tasks represented self-selected needs that lead to thematic exploration. Narrow tasks represented relatively narrow topical information needs.
Explorative/Data gath- The tasks were deering/Semi self-selected signed to represent diferent stages in information seeking processes.
Table 3: iTrack task categories In 2005 the author noted concerns that the 2004 scale “was far too complex for the test persons to comprehend”, thus choosing the simple scale in 2005. In 2006 and 2008 a two-dimensional scale with ifve possible scores was used, with the following definitions: Relevant, but too broad, contains relevant information, but also a substantial amount of other information. Relevant, contains highly relevant information, and is just the right size to be understandable. Relevant, but too narrow, contains relevant information, but needs more context to be understood. Partially relevant, has enough context to be understandable, but contains only partially relevant information. Not relevant, does not contain any relevant information that is useful for solving the task. 3.3.5 Analysis methods. iTrack data analysis has been performed using a combination of transaction logs and questionnaire data. Studies have been performed investigating the types of transactions taking place, typical transaction patterns, and factors influencing transaction patterns.
The INEX iTrack evolved from 2004 to 2010. In the first years, it complemented the research goals of the ad hoc-track, re-using topics, with some modifications, from the ad hoc-track, with the intention to identify how end-user react to element-based IR systems. The software used for the search system, which was developed at the University of Duisburg-Essen, gradually developed and interface design was kept consistent. Questionnaires were also kept fairly consistent, addressing the same background factors from year to year.
The iTracks proceeding reports document the study design. The software is documented at the web sites. The questionnaires are not well documented. The biggest issue is the availability of transaction logs and questionnaire data. These are not openly available at the time of writing. The intention of the iTrack was that the data should be available only to the research groups for a limited period and then become available for others upon request. Unfortunately, the iTrack web sites are no longer available, which leaves us with the track reports as the main oficial documentation.
The experiences from the INEX iTrack have been manifold. With the collaborative efort of several research groups collecting data in a standardized manner, the iTrack resulted in large interactive IR datasets. The maximum number of participating research groups were 11 (in 2004 and 2005), with 119 searchers taking part in the 2005 experiment. The data can be compared across countries and, to a certain degree, across diferent user groups (although the majority of participants have, however, been students in computer science and library and information science). In addition, rich background data on many searchers have been collected.
The major challenges of the experiments are the design of tasks.
These should be relevant for the participants and tailored following
Borlund’s simulated work task situation method [
Other challenges include the identification of factors that influence interaction. We need to be able to identify the degree in which we can make valid analysis based on the data.
Specific challenges related to re-use and data sharing in interactive IR include establishing standardized ways of documenting experiments, which is what the BIIRRR workshop addresses. It is also necessary to establish a forum for discussions and coordination of IIR experiment eforts
The INEX interactive track organized collaborative interactive information retrieval experiments from 2004 to 2010. In all, the iTrack initiated six rounds of experiments with changes in tasks, collections and search systems. The experiments resulted in data in the form of transaction logs and questionnaires. All experiments were documented in the INEX proceedings.
Although experiments evolved throughout the period, with significant impact on elements such as task types and relevance scales, the documentation is fairly consistent. The data are, however, at present not publicly available and the systems that were used are only partially available. This raises the following questions and challenges for securing re-use of Inex iTrack experiments, which will also be of value for resuse of interactive IR experiments in general: • the need for a data repository for preservation of research designs, including transaction logs and questionnaires along with code books and necessary documentation for re-use • a common repository for document corpuses and search systems • a discussion on the need for standardized questions in questionnaires in order to compare across experiments
I would like to thank Norbert Fuhr and Thomas Beckers for valuable information about the current status of the iTrack systems and data. Workshop on Barriers to Interactive IR Resources Re-use at the ACM SIGIR Conference on Human Information Interaction and Retrieval INEX iTrack Revisited: Exploring the Potential for Re-use