Two key, but usually ignored, issues for the evaluation of methods of personalization for information retrieval are: that such evaluation must be of a search session as a whole; and, that people, during the course of an information search session, engage in a variety of activities, intended to accomplish different goals or intentions. Taking serious account of these factors has major implications for not only evaluation methods and metrics, but also for the nature of the data that is necessary both for understanding and modeling information search, and for evaluation of personalized support for information retrieval (IR). In this position paper, we: present a model of IR demonstrating why these factors are important; identify some implications of accepting their validity; and, on the basis of a series of studies in interactive IR, identify some types of data concerning searcher and system behavior that we claim are, at least, necessary, if not necessarily sufficient, for meaningful evaluation of personalization of IR.
When people, seeking information in order to accomplish some task, or achieve some
goal, engage with information retrieval (IR) systems, they often, perhaps most
typically, conduct what one can term an Information Seeking Session (ISS). Although the
minimal form of such a session can be a single query put to the system, a response by
the system, a choice by the person of an item from the response, and the end of the
session, both research in information behavior, and observation of behavior in
operational systems, demonstrate that such behavior is not typical. Rather, an ISS often,
perhaps most often, consists of a number of such iterations [
Under this understanding of people’s behaviors in interaction with IR systems, we, as have others [e.g. 7, 18], propose that IR is best construed as a sequence of interactions of the person with the IR system, motivated overall by some external task or goal, with each interaction being itself motivated by some information seeking intention. These interactions can be considered as sub-tasks that arise in the person’s attempt to eventually achieve the overall goal of the ISS, which is to obtain that which is deemed useful in accomplishing the motivating task/goal. This view of IR has important implications for what it means to personalize support for IR interaction, and how to accomplish such personalization, for how to evaluate such support, and, importantly, for what data are required to both to accomplish such support, and to evaluate it. In this position paper, we address, in particular, the issue of the data required to accomplish and evaluate personalization of IR interaction. 2
Implications of the ISS Model of IR for Personalization and
its Evaluation
It has been suggested that the ISS model of IR implies that usefulness, rather than
relevance, is the appropriate criterion of evaluation of interactive IR [
In order to address the question just posed, we first examine just what data have already been collected in the course of the rather substantial record of research in the evaluation of interactive IR, and how it has been collected. We analyze and compare the data types collected in such studies in order to see if they can lead to the understandings which we propose are necessary to evaluate support for personalization in the manner we require. This is the subject of the next section of this paper. On the basis of these results, we then propose a research agenda, which could lead to methods for identification and collection of the data necessary, beyond those which have already been identified, for proper evaluation of at least the aspect of personalization which we have described. 3
Existing Data Sets for IR and Interactive IR Evaluation
There exist a number of data sets for the evaluation of IR and Interactive IR (IIR)
evaluation purposes. Table 1 gives an overview of a selection of existing data sets
with their properties. We divided the data sets roughly in three different groups: (1)
evaluation campaign data sets, (2) real-world data sets, and (3) lab study data sets.
Evaluation campaign data sets normally involve a fixed corpus, given topics, queries
and relevance judgements for all or a set of result documents. The goal here is to
optimize the ranking function of the system based on given topics and the expected
relevant results. Thereby the focus of investigation has opened from the query level
(TREC Web Track [7]), over the session (TREC Dynamic Track [
A second group of evaluation data sets comes from real-world search engines. For
web search there exist e.g. the Yandex Web Search click data [
Evaluation data sets from lab studies examine how real users conduct a certain task type with a given topic. From the data side the desire is to log as much information as possible in order to analyze user behavior from this data. This includes logging all interaction with the system including keyboard and mouse interaction and in addition the eye movements. For subjective measures subjects are interviewed before and after the search session e.g. to learn something about the task’s difficulty and success. Additionally, to identify learning steps and decisions within the search session and certain query segments, a number of user interviews have to be conducted after the search session. Given a task, the documents usefulness needs to be assessed in relation to the overall task or sub task (not to the query). Another important issue is to understand the role of each query segment for the overall task. A current investigation 1 http://www.ir.disco.unimib.it/pirclef2017/description-of-the-laboratory/ is to ask the user for the intention of each individual query segment. The question can be: Was it to find new information or to evaluate already found information objects? In reviewing Table 1, we note several important differences between the different data sets, which are especially significant for evaluation of personalization. The most obvious is that only one data set includes detailed information about the search session at the query segment level (last row of Table 1). Since data at this level would be crucial for evaluation of the aspect of personalization we discuss, it’s clear that the methods used in this, and similar studies, need to be considered for evaluation of personalization. However, this type of study suffers from at least two other problems: a small number of cases (users, tasks, topics); and, controlled, rather than real, tasks and task types. A conclusion that one can draw from this comparison, is that what is required is some means for incorporating, in one general type of study, methods which allow: the collection of (relatively) large numbers of cases, of real tasks, addressed over whole search sessions, segmented and identified by information search intentions. Developing a means for doing this, effectively defines a research agenda for the design of studies which aim to evaluate personalization of support for IR. Below, we provide some examples of the types of data that would need to be collected in such studies, as contemplated for, e.g. PIR-CLEF 2018.
As an example, we could extract various aspects of learning that take place throughout the search. Specifically, we should try to understand how the searcher is learning about the task and the domain as he/she retrieves and assesses information, and how that learning affects his/her ongoing search activities. Some of the questions to ask the searcher or an assessor for eliciting such information are: ! What has been learned from the domain knowledge? ! What has been learned from the content obtained? ! How useful is what has been learned for the sub task? ! How well did the system support learning?
Another important aspect that we find useful to elicit from an IIR study is that of evaluation. Specifically, we believe it is important to discover how searchers (not external judges) evaluate an item for correctness and usefulness, and not just relevance. Searchers are also often comparing several relevant/useful items and picking the best and it would be interesting to know how they make such decisions. Some of the questions that could be asked to the searcher or an assessor to gather such information are: ! Which items were evaluated? ! What was evaluated? ! What were the criteria? ! How useful were the items for the sub task? ! How well did the system in supporting evaluating the items?
These two types of intention of course do not cover all information seeking
intentions that could occur during an ISS (see, e.g. [
We have proposed a view of IR that implies that personalization should be with respect not only to context, but to the various information search intentions that people have during the course of an information seeking session. We have identified some types of data which we claim would be necessary in order to evaluate the effectiveness of such personalization. We suggest that, learning just what data are necessary, and developing methods to gather such data, constitute the basis for a research agenda central to the general task of evaluation of personalization of support for IR. This could also be a starting point for considering the nature of the task for PIR-CLEF 2018.
This work was partly funded by Deutsche Forschungsgemeinschaft (DFG), grant no. MA 3964/5-1; the AMUR project at GESIS; and, by the National Science Foundation, grant no. IIS-1423239.