This paper describes our experiences with the interactive IR tracks organized at CLEF from 2013-2016 and aggregates the lessons learned with each consecutive instance of the lab. We end with a summary of practical insights and lessons for future collaborative interactive IR evaluation exercises and for potential re-use scenarios.
After the INEX (Initiative for Evaluation of XML Retrieval)
Interactive Track ended in 2010 [
This paper provides a chronological overview of the development and history of these two IIR initiatives and their outcomes. We focus on the lessons learned for future collaborative IIR evaluation exercises and for potential re-use scenarios. We start by chronicling the timeline of the diferent interactive labs that were organized in Sections 2-6. We then highlight the most important lessons learned for the configuration of IIR evaluation experiments. We conclude by discussing consequent activities and insights for the re-use of IIR resources. 2 2.1
The EU-funded PROMISE1 project (Participative Research
labOratory for Multimedia and Multilingual Information Systems
Evaluation) ran from 2010-2013 with the goal of providing a virtual and
open laboratory for research and experimentation with complex
multimodal and multilingual information systems [
For the ‘Unlocking Cultural Heritage‘ (CH) use case, a workshop
at the 2011 CLEF conference was organized in order to review
existing information access use cases in the CH domain and then
develop retrieval scenarios that could be used for evaluating CH
information access systems [
At the 2012 CLEF conference, a pilot evaluation exercise was
organized for the CH domain, progressing work from the workshop
format to an evaluation lab [
Although the 2011 CHiC workshop had already emphasized that a focus on user interaction patterns was an important evaluation aspect for the CH domain, this first CHiC lab in 2012 had no interactive tasks. Instead, it utilized a document collection based on Europeana and used queries harvested from Europeana logs to construct information needs. The vision was to extend the ad-hoc style retrieval evaluation with interactive and other evaluation scenarios (particularly result presentations and alternative methods for relevance assessments) in the next phases.
The Europeana document collection, albeit a real-world collection, turned out to be very challenging. While an efort was made to normalize the provided metadata by wrapping it in a special XML format and removing certain metadata fields, the content in the metadata had very diferent descriptive qualities, depending on the original content provider. Both the data sparseness and multilinguality of the content posed serious challenges for the participants. Image data, such as thumbnails of graphical material in Europeana, could not be provided due to copyright reasons.
Some of the provided topics were not suitable for relevance assessment, because information needs could not always be unambiguously inferred from the provided queries. The topics mostly contained short queries of 1-3 words and only half of them had short descriptions added, which did not help much when the topic was vague. For the CH use case, IIR studies focusing on interaction patterns were needed, so an additional interactive task was proposed for the next round. 3 3.1
The Interactive Track5 at the CHiC 2013 lab at CLEF (iCHiC) aimed
at building a bridge for IIR and behavior researchers to work in a
TREC-style evaluation environment. The idea was to develop a data
collection of IIR evaluation data, which could be re-used and built
upon. This task intentionally used a subset of the document
collection used in the other CHiC ad-hoc retrieval experimental tasks to
allow for later triangulation of results. Based on approximately 1
million metadata records from the English Europeana collection
and representing a broad range of CH objects, a simple search
interface was envisioned that would allow for browse and search
interactions with the metadata records for the IIR experiments [
The same experimental infrastructure, which hosted the
webbased interfaces, documents and logged the interactions [
The most important lesson learned from iCHiC and the reason why it was merged with the INEX Social Book Search lab (see Section 4) was that the provided metadata records were not “rich” enough in content to provide an interesting case study for casual browsing and search. The sparseness of the document collection had already been a problem for the ad-hoc retrieval tests, and real users did not like them any better. The actual purpose of iCHiC—to study users’ interactions with the content—was hampered by the lack of interesting content.
The experimental set-up and questionnaire instruments represented a significant efort for the participants to complete. However, the collected data was deemed necessary for further analysis.
An original plan for the set-up of this task was to provide the metadata collection, simulated work tasks, and the experimental setup (questionnaires, logging protocol) to the participating research groups and have them provide their own infrastructure for data gathering. After discussions, the organizers concluded that having diferent groups each building infrastructures would add too much variability and also pose a large barrier to entry especially for groups that did not have software or GUI design specialists.
The data gathering at the University of Shefield’s servers had the additional advantage of having a central place where all the data was stored. This also posed a problem in later years, however, when researchers afiliated with the University of Shefield moved to a diferent institutions and neither the preservation and maintenance of the infrastructure and data nor its legal ownership were established.
Four teams participated in the track, but not all of them were able
to recruit the 30 required participants. The uneven contribution
led to some discussion about the fairness of all groups then being
able to use the same data in later analyses. Initial discussions on
who would get to analyze the data with which research questions
in which priority (important for later publications) were never
successfully resolved as the organizers moved on to new tasks.
Some of the organizers published follow-up analyses of the data
[
The participating groups all adhered to research ethics requirements set forth by the University of Shefield, which hosted the platform and the collected data. Diferent ethical requirements (e.g., based on national law) were not considered. The experimental participants were asked to consent to their responses being shared not just with the organizers, but with the wider research community, which allows for re-use of the data. However, processes for enabling the data sharing at a later time were not considered.
The proposal for the interactive task had planned for a two-year
period, where the data gathering (user interaction logging) and
preliminary data analysis would happen in the first year. In year
two, an aggregated data set of all logged interactions was to be
released to the research community in order to inform an improved
system design for data gathering, which would start again in year
two. While the organizers provided an initial analysis of the data
[
Social Book Search (SBS)6 started as a system-centered evaluation
campaign at INEX in 2011 [
Another important initiative was to study the diferent stages of
the search process and how they could be supported by diferent
interfaces [? ]. We considered models of the information search
process [
This led to a three-stage model of browsing, searching and
selection, each with separate interfaces that carry over user choices
when switching between interfaces, based on Goodall [
We were interested in the diference between goal-oriented and
non-goal oriented tasks, also to compare the non-goal oriented task
in the book domain to the same non-goal task in CH as used in
6http://marijnkoolen.com/Social-Book-Search/
iCHiC [
The 2014 iSBS Track did not run as a full evaluation campaign,
because most of the year was used to prepare and set up the
multistage search system, tasks and protocol [
Although the long preparation phase left little time for gathering data, it resulted in a consensus among the large group of organizers about the set of generic research questions that the experimental setup and search systems should be able to address.
The setup did not lead to enough complex interactions to identify stage transitions in the search process and to test the value of multi-stage interfaces. We considered multiple causes: (1) the tasks were relatively simple and did not require complex interactions; (2) the instructions and training task were not suficient to get users familiar with such an interface; and (3) the interface was not self-explanatory enough for users to interact with meaningfully. The questionnaire data suggested the tasks could be completed with little efort. We subsequently discussed whether we should use more complex yet still realistic book search tasks.
There was a conflict between the goal of studying social book
search with realistic tasks and the goal of studying the value of
interfaces for diferent stages in the search process. The models
of Kuhlthau [
The second year of the iSBS track was open to other research groups
and had a longer data gathering period with many more participants
(192 in total) [
We found that the fact that metadata in the book collection was exclusively available in English was a hurdle for several non-native English speaking users. As some participating groups contributed many more users than other groups, with more non-native English speakers, the balance was very diferent than the year before, which makes comparison of cohorts dificult.
Users also spent a lot of time on the goal-oriented task with sub tasks, causing some of them to abandon the experiment after the first of the two tasks. In their feedback, others indicated that the overall experiment took too long. This could mean that the gathered data is biased towards more persistent participants. 6 6.1
In the third edition of the iSBS track we made more significant
changes to the experimental setup. Some modifications were made
to the experiment structure to avoid participants abandoning the
experiment. The main change was that users only had one
mandatory task, but could continue with other tasks as long as they were
willing to continue. We added eight tasks based on book search
requests from the LibraryThing discussion forums to provide as
realistic tasks as possible [
A comparison of the 2015 and 2016 cohorts showed very few diferences in terms of time spent on goal-oriented and non-goal tasks (the 2015 cohort showed no ordering efect between doing goaloriented first and doing non-goal-oriented first), giving a strong indication that the experiment structure and tasks are producing reliable results. This also suggests that the two cohorts could be combined to reduce the impact of individual diferences. One of the hardest struggles in IIR evaluation campaigns is getting a large and diverse enough set of users. Running such campaigns for long periods requires continuity. The same experimental systems need to remain available with at most small changes.
The additional tasks based on requests from the LibraryThing discussion forums resulted in diferent search behaviour from the simulated goal-oriented and non-goal oriented tasks, but also showed large diferences between the LibraryThing tasks themselves, with more subjective, fiction-oriented tasks leading to less interaction than concrete, non-fiction-oriented tasks. This suggests that IIR ifndings may be very sensitive to the specifics of the simulated work tasks used. It may also signal that in order to study information search for reading for one’s own enjoyment, it is important that users have ‘skin in the game’ and feel a personal connection to leisure-focus work tasks.
A problem encountered since running the 2016 iSBS Track is that organizers move between institutions, which causes problems for maintaining experimental systems, websites and repositories when they loose institutional access to servers where the infrastructure is hosted on. This in turn endangers the continuous availability of research data and experiments. A natural solution to this recurring problem could be an independent or inter-institutional platform and repository for these systems and materials. One important lesson learned from the iCHiC and iSBS tracks is the importance of a suitable document collection that is realistic in both size and content variety. The document collection used for iCHiC was based on metadata from Europeana. Even though it represented a broad range of diferent topics, the individual items in the dataset were often sparse in their information content. In the iSBS tracks, the document collection based on Amazon and LibraryThing data ofered richer information that is more suitable for an interesting task for users, but over the course of the diferent iSBS editions the collection grew increasingly out-of-date. We found this negatively afected search behavior as well as user engagement, especially during the open search task. Users were looking for recent book titles and got frustrated that they could only find books that were at least six years old.
While re-use of IIR resources is important for replicability and reproducibility, oftentimes older document collections are simply not interesting anymore for participants—something system-based evaluation sufers from to a lesser degree. How to obtain realistic, engaging, and up-to-date document collections, while at the same time maintaining comparability across evaluation iterations, remains an open question.
Using a live document collection from a production system would not allow for the same number of interactions to be studied and poses dificulty for logging. It is not a simple alternative. Arguably, what matters is not the stability of the set of documents that are searchable, but the extent to which that set is up-to-date. Book search interactions gathered in 2014 can be compared with those gathered in 2019 if in both cases users could search books published in the last five years, despite there being no overlap between the two collections, as long as the type and amount of information about books remains the same. To improve re-usability, it may be more valuable to investigate and describe relevant aspects of document collections, so that IIR studies with diferent document collections can be compared based on their overlapping relevance aspects, e.g., recency, structure, type, and amount of information.
Unfortunately, realistic document collections tend to exhibit a larger degree of variety and complexity. This may make them more engaging and interesting to participants, but it also increases the complexity of the analysis of their behavior. One could argue that to achieve a more detailed and thorough analysis, perhaps simpler document collections would be more suitable, thereby setting up a trade-of between complexity at the experimental and the analysis stages. 7.2
In order to have meaningful impact, IIR studies should be
representative of the real-life variety in domains, system designs, and user
types and needs. One way in which iCHiC and iSBS attempted to
do this was by using a varied and realistic set of simulated work
tasks [
Despite the proven usefulness of simulated work tasks, they are still not the same as a user’s own information needs. We therefore also included work tasks in iCHiC and iSBS that focused on the participants’ own information needs. Non-restrictive tasks, in which users can search whatever and however they want for as long or short as they want, ofer more realistic aspects of information behavior, but they make comparison more dificult. Diferences between users can be due to them having wildly diferent ‘tasks’ in mind. Although we experimented with diferent types of tasks, we feel that we have only scratched the surface here. True information needs can be multilingual and multicultural, making assessment even more challenging.
In addition, by focusing only on single information needs, we
believe that we are ignoring valuable aspects of the entire information
seeking process, both individual and collaborative [
Ideally, an IIR evaluation campaign recruits participants that are a realistic representation of the general target population to avoid the introduction of biases [8, p. 241]. However, in most IIR tracks— including our own—researchers have often relied on recruiting students from participating universities or research groups as participants. Due to the short-term preparations and research cycles, this is often the only way to include enough participants in an IIR experiment. However, students are only one of several user groups that need to be taken into account when dealing with complex search tasks. It needs to be assured that users are selected based on the specific system, feature or task to be tested as ignoring these relationships and dependencies is likely to lead to invalid results. Longer preparation time or access to user databases with potential participants could help overcoming such biases in participant recruitment.
One of our findings in iSBS was that the cultural background makes a significant diference. This is something that is rarely reported in studies, but that appears to be an important aspect to include. This also challenges the assumption that by providing the same infrastructure and tasks but using diferent user group distributions over the years or across national boundaries, measured user interactions can be aggregated across these groups. There were some analyses that clustered users based on certain aspects, but the question remains which users can be viewed in aggregation. Since academic IIR studies often rely on students, perhaps studies can explicitly describe criteria of representativeness of the target user group and add questions to the questionnaire that capture aspects of users that allows mapping them to these aspects of representativeness. 7.4
The search user interface is perhaps the most important aspect to get right for the IIR system used in the experiments as our experience with iCHiC and iSBS tracks has taught us. The ubiquity and popularity of modern-day search engines means that any search user interface has certain minimum expectations to meet in terms of layout and/or functionality. Not meeting these expectations means risking distracting users and has a deleterious efect on their search behavior. It would be beneficial if the IIR system ofered the flexibility of choosing diferent search interfaces to study the efects of the GUI on information seeking behavior. This was used to great efect in the iSBS tracks to examine how diferent interfaces can support the diferent search stages.
This flexibility came at a price, however, as the software
components needed for the infrastructure became increasingly complex.
Both iCHiC and iSBS used a customized infrastructure developed
by one of the organizers, which made this possible [
Another dificulty is that the design of interfaces can be informed by diferent theoretical models of information interaction. In setting up the iSBS track and designing the multistage interface, we discussed the appropriateness of numerous information seeking/search models as well as book selection models and strategies, how they are related to each other and how they correspond to or are supported by aspects of the interface. A further complication is that our choices were also steered by the research questions we wanted to address. These issues add another set of variables to take into account when considering comparison and reuse, and should be described in studies. 7.5
IIR research usually includes several complex components that can afect the quality and success of each experiment. While the importance of some elements such as task development have been extensively discussed, other aspects remain less considered. Only a few studies report on or discuss measures used to analyze or interpret results from IIR experiments. So far, IIR measures are highly contextual varying from experiment to experiment. Measures used span from data on interactions, such as session duration or clicks, to qualitative data derived from questionnaires or interviews. Often several data points are complemented or correlated.
A collaborative IIR study requires that participating research groups pool their gathered data and aggregating this data generates substantial overhead. If institutions gather their own data, aggregation may involve harmonizing inconsistencies. In the iCHiC and iSBS tracks, a single system was used to gather all experimental data, but this system had to be developed and adapted with each iteration. A comprehensive documentation and accurate descriptions of the data gathering tools is crucial for the evaluation and re-use of these aspects in future studies.
Diferent research groups and individuals often want to study
slightly diferent aspects of the problem domain or setup, requiring
diferent questions in the questionnaire, diferent tasks or users, or
diferent search system components. With every change, new users
need to be recruited, and comparisons to previously collected data
becomes harder. The long preparatory discussions among the iSBS
organizers regarding research questions, theoretical frameworks
and research designs suggests that it is possible to some extent to
incorporate a broad set of research questions in the overall research
design to allow a range of studies with the same setup. But often
research questions change or new questions are prompted during and
following the experiments, calling for an iterative development of
the research design. We are not aware of any guidelines on how to
best update designs to allow some backwards comparability. While
there is large variability in research questions and research designs,
the group would have benefited from re-using other researchers’
research design components, as was done with the User
Engagement Scale [
From 2011 until 2016, the various interactive tracks generated a wealth of data, but also went through numerous organizational changes, both in terms of the individuals involved and the institutions that provided infrastructure. iSBS started as part of INEX with some data stored on servers dedicated to INEX activities, other data stored on servers maintained by one of the organizers’ institutions and the search indexes on another set of servers of another organizing institution.
Recurring questions are (1) what happens if organizers leave and own crucial pieces of the data or infrastructure, and (2) what happens when organizers move between institutions, thereby losing access to data or infrastructure? For research data management purposes, it is important that organizers of IIR studies make explicit who is responsible for which part of the data and systems, who owns the data or infrastructure, and what happens when organizers move to other institutions or leave the project, or when new organizers join.
While always intended, the organizers of iCHiC and iSBS could ifnd hardly any re-use of the gathered data for IIR studies or triangulation studies with the related ad-hoc retrieval experiments in CHiC or SBS. One reason may have been the insuficient availability of the research data along with a proper rights clearance.
There are generic platforms for storing and sharing scientific data, such as the Open Science Framework7 and several Dataverse8 instances. These options solve some of the institutional issues, but they lack the flexibility to run experimental systems or to add domain-specific search and access features to datasets that make a repository like RepAST useful to the IIR community. Publicly available repositories for software and software infrastructures also exist (e.g., GitHub), but present similar problems to the research data repositories.
Next to problems of storage and access of IIR research data, there are issues of copyright, privacy and ethics. The questionnaire informs users, which institutions are involved, but how should organizers deal with new researchers and institutions joining? One option is for organizers to agree on ethical guidelines for data gathering, informed consent and data representation. For further data re-use, it is crucial that users also give their informed consent for additional analyses of their data. To create a trustworthy environment, IIR researcher must provide concrete statements on who and for what future purposes the data will be used. This should be available additionally to the research data as part of an archived and documented research design (see Section 7.5). 7.7
IIR research is a highly interdisciplinary field bridging areas of information seeking, interactive and system-centered (ranking, evaluation) IR and user interface design. Accordingly, researchers from diferent disciplines need to collaborate on complex questions and experimental setups. Entering the field of IIR research is still a challenge due to inconsistent or incompatible practices. Even for those that work on IIR problems, no collaboration on systems, tasks, data, participants or research questions can be observed. This might be the case due to time and resource constraints caused by traditional one-year research cycles as well as unawareness of other projects.
In assessing the interest in an interactive track in the SBS Lab during a joint iCHiC and SBS discussion session at CLEF 2013, everyone who stated their interest was involved in the initial discussions in setting up the track, to get an overview of what aspects they wanted to investigate, thereby shaping the track around a broad set of interests. This community input is valuable both in attracting groups to actively participate and in creating a setup with potential for long term community support and interest. A challenge of the desired community input and larger organizer numbers is the required additional overhead for the decision processes. Once again, good documentation and communication is vital as are well-understood guidelines or practices about the consequences of researchers joining or leaving the initiative. Collaborative research 7https://osf.io/ 8https://dataverse.org/ also entails a joint understanding of how research results will be presented (e.g. rules of authorship and priority). This is especially important in large collaborations.
Collaborative research, by its very nature, tries to study aspects which require a large-scale infrastructure, a large number of users or other aspects that need a strong community input. This will necessarily prolong the design and implementation phases of any study, which is a detriment in a fast-paced scholarly context as IIR research, especially within the large evaluation campaigns or research conferences, which run on annual cycles. This type of work would be best supported by a multi-year project or by moving to a slower research output model. 8
Based on previous experiences from the CLEF/INEX Interactive
Social Book Search tracks, the two Supporting Complex Search
Tasks (SCST) community workshops (2015 and 2017) [
The 2018 workshop on Barriers to IIR Resources Re-use (BIIRRR)
switched the focus to the analysis and preparation of requirements
for efective re-use of IIR resources or experiments [
One idea is to develop a platform that would allow researchers from interdisciplinary fields to search for IIR research designs once they have been identified as re-usable and are stored and documented. Building such a repository requires an analysis and implementation of user requirements both for accessing and contributing research designs, the development and agreement on a standardized data infrastructure as well as a maintenance plan coordinated by a stable team of researchers.
Apart from a proper documentation and archiving strategy, this retrospective also pointed towards pre-study aspects, which are instrumental for re-using experimental research data and designs. This includes the establishment of guidelines for cross-national and cross-institutional data collection, informed consent and data distribution. As was declared several times in this paper, the reusability of research designs and other IIR study components strongly depends on the community’s willingness to develop and maintain proper documentation, curation and publication guidelines. While this may not be as rewarding as creating new research data by implementing more IIR studies (and we need more of these as well), it is crucial for the community to standardize in order to move forward as a research discipline.