As designers of information retrieval (IR) systems, we
need some way to measure the performance of our systems.
An excellent approach to take is to directly measure actual
user performance either in situ or in the laboratory [
Cranfield-style evaluation frequently has been criticized as
being too divorced from the reality of how users search, but
there really is nothing wrong with the approach [
As such, we should really be thinking of the Cranfield approach as the application of models to make predictions, which is common practice in science and engineering. For example, physics has equations of motion. Civil engineering has models of concrete strength. Epidemiology has models of disease spread. Etc. In all of these fields, it is well understood that the models are simplifications of reality, but that the models provide the ability to make useful predictions.
Information retrieval’s predictive models are our evaluation metrics.
The criticism of system-oriented IR evaluation should be redirected. The problem is not with Cranfield — which is just another name for making predictions given a model — the problem is with the metrics.
We believe that rather than criticizing Cranfield, the correct response is to develop better metrics. We should make metrics that are more predictive of human performance. We should make metrics that incorporate the user interface and realistically represent the variation in user behavior. We should make metrics that encapsulate our best understanding of search behavior.
In popular parlance, we should bring solutions, not
problems, to the system-oriented IR researcher. To this end,
we have developed a new evaluation metric, time-biased
gain (TBG), that predicts IR system performance in
human terms of the expected number of relevant documents
to be found by a user [
Presented at EuroHCIR2012. Copyright (C) 2012 for the individual papers by the papers’ authors. Copying permitted only for private and academic purposes. This volume is published and copyrighted by its editors. 2.
HCI has a long history of automated usability
evaluation [
Ja¨rvelin and Keka¨l¨ainen produced a significant shift in
evaluation metrics with their introduction of cumulated
gainbased measures [
Cumulated gain can be plotted vs. time to produce a gain curve and compare systems. The curve that rises higher and faster than another curve is the preferred curve. While we can plot gain curves of one system vs. another, it is wellknown that users do not endlessly search; different users stop their searches at different points in time for a host of reasons. Given a probability density function f (t) that gives the distribution of time spent searching, we can compute the expected gain as follows:
E[G(t)] = 0 ∞
G(t)f (t)dt, (1) where G(t) is the cumulated gain at time t. Equation 1 represents time-biased gain in its general form, i.e. timebiased gain is the expected gain for a population of users.
While it is natural for us to talk about cumulated gain over
time, the traditional cumulated gain measures have
substituted document rank for time and implicitly model a user
that takes the same amount of time to evaluate each and
every document. By making time a central part of our
metric, we gain the ability to more accurately model behavior.
For example, in a document retrieval system, longer
documents will in general take users longer to evaluate, and if the
retrieval system presents results with document summaries
(snippets), we know that users can use summaries to speed
the rate at which they find relevant information [
Another significant advantage of using time directly in our retrieval metric is that we now make testable predictions of human performance. Our predictions are in the same units as would be obtained as part of a user study. To our knowledge, this alignment between the units of Cranfield-style metrics and user study metrics has not previously existed.
Time-biased gain in the form of Equation 1 makes no
mention of ranked lists of documents, for it is a general purpose
description of users using an IR system over time. To
produce a metric suitable for use in evaluating ranked lists, we
followed a process common to development of new
simulations [
3. Validation of model.
Our first step in model creation was to adopt the standard model of a user that works down a result list and move Equation 1 to a form common to cumulated gain measures: ∞ k=1 gkD(T (k)), (2) where gk is the gain at rank k, T (k) is the expected time it takes a user to reach rank k, and D(t) is the fraction of the population that survives to time t and is called the decay function.
Our model for the time it takes a user to reach rank k, T (k), takes into consideration a hypothetical user interface that presents results to the user in the form of document summaries. A click on a document summary takes the user to the full document. We model both the probabilities of clicking on summaries given their NIST relevance and the probability of then judging a viewed full document as relevant. We separately model the time to view summaries and full documents. For the time spent on a full document, we modeled longer documents taking longer with an additional constant amount of spent. We treated duplicate documents as zero length documents. We then calibrated T (k) using data from a user study, and finally we validated that our T (k) provided a reasonable fit to the user study data. Likewise, we modeled D(t) as exponential decay fit to a search engine’s log data.
In contrast, older evaluation metrics such as mean average
precision [19, p. 59] cannot be calibrated and have only
been validated after their creation. For example, the work
of Hersh and Turpin [
The Cranfield approach to IR evaluation is merely
another name for the development and use of predictive
models, which is a fundamental part all science and engineering
fields. In particular, IR evaluation fits nicely into the
framework of simulation where models are created, calibrated, and
validated before being used to make predictions. We have
presented time-biased gain as an example of what we believe
the correct direction is for IR system evaluation. We are not
the only ones to be working on better metrics or taking a
simulation based approach [
Our position is that system-oriented IR research is useroriented IR research given its use of evaluation metrics that model users. If HCIR researchers can produce better models than exist today — by better, we mean more predictive of human performance — then we can help system development to focus on changes that help users better search.
This work was supported in part by the NSERC, in part by GRAND NCE, in part by Google, in part by Amazon, in part by the facilities of SHARCNET, and in part by the University of Waterloo. Any opinions, findings and conclusions or recommendations expressed in this material are the authors’ and do not necessarily reflect those of the sponsors. 5.