Information retrieval efectiveness evaluation is often carried out by means of test collections. Many works investigated possible sources of bias in such an approach. We propose a systematic approach to identify bias and its causes, and to remove it, thus enforcing fairness in efectiveness evaluation by means of test collections.
In Information Retrieval (IR) the evaluation of systems is often carried out using test collections. Diferent initiatives, such as TREC, FIRE, CLEF, etc. implement this setting in a competition scenario. In the well known TREC initiative, participants are provided with a collection of documents and a set of topics, which are representations of information needs. Each participant can submit one or more run, that consists in a ranked list of (usually) 1000 documents for each topic. The retrieved documents are then pooled, and expert judges provide relevance judgements for the pooled ⟨topic, document⟩ pairs. Then, an efectiveness metric (such as AP, NDCG, etc.) is computed for each ⟨run, topic⟩ pair, and the final efectiveness metric for each run is obtained by averaging its efectiveness score over the set of topics. Finally, the set of runs is ranked in descending order of efectiveness.
Diferent works investigated possible source of bias for this evaluation model by looking at system-topics correlations. In this work we propose to extend prior work by considering the many dimensions of the problem, and we develop a statistical model to capture the magnitude of the efects of the diferent dimensions. 2.1
The output of the TREC initiative can be represented as an
efectiveness matrix E as in Table 1, where each si is a system configuration
(i.e., run), each tj is a topic, esi,tj represents the efectiveness (with
a metric such as AP) of the i-th system for the j-th topic, Es and Et
represent respectively the average efectiveness of a system (with a
metric such as MAP) and the average topic dificulty (with a metric
such as AAP [
To capture the bias of such evaluation setting, Mizzaro and
Robertson [
t1 es1,t1 es1,t1 · · · Et (t1) · · · · · · · · · . . .
tn es1,tn esm,tn Et (tn )
Es Es (s1) . .
.
Es (sm )
link from a system to a topic expresses how a system thinks a topic
is easy, and each link from a topic to a system expresses how a
topic thinks a system is efective. Then, Mizzaro and Robertson
compute the hubness and authority values of the systems and topics
by running the HITS algorithm [
Soprano et al. [
Breaking down the efect caused by a dimension on a complex
system has been widely studied in IR. A problem of particular
interest is to break down the system efectiveness score (such as
AP) into the efect of systems, topics, and system components, like
for example the efect of stemming, query expansion, etc. For this
purpose, Ferro and Silvello [
In this paper, we propose to extend results from the related work (see Sections 2.1 and 2.2) to define a sound and engineered pipeline to find and correct the bias in the IR efectiveness evaluation setting. More in detail, we plan to investigate how the specific bias of efective systems being recognised by easy topics varies when varying the components of a test collection, such as the system population, pool depth, etc. Finally, we propose to use a GLMM as done in the related work (see Section 2.3) to compute the magnitude of efect of the various components on the bias.
To investigate the efect of the diferent pool depths, we plan to compute, for each efectiveness metric, its value at diference cutofs. Figure 1 shows a preliminary result: the plot shows, for the Precision metric, the diferent cut-of values on the x-axis and, on the y-axis, the bias value represented by the Pearson’s ρ correlation between the hubness measure of systems and their average precision value; this bias represents the fact that efective systems are recognised by easy topics. As we can see from the plot, there is a clear trend suggesting that the bias grows together with the pool depth. The undesired efect that efective systems are mainly those that work well on easy topics becomes stronger when increasing pool depth.
To investigate the efect of the diferent collections, we plan to use diferent TREC collections: Robust 2003 (R03), 2004 (R04), and 2005 (R05), Terabyte 2004 (TB04) and 2005 (TB05), TREC5, TREC6, TREC7, TREC8, and TREC2001. Furthermore, we plan to break down the sub-corpora efect by considering the diferent corpora of the collections.
We will investigate the efect of diferent evaluation metrics in
the model bias, specifically we will consider: Precision (P), AP,
Recall (R), NDCG, τAP, RBP, etc. When dealing with the metric
efect, we can consider two approaches in the normalisation step:
remove the average of system efectiveness and topics ease, as for
example remove MAP and AAP from AP (as done by Mizzaro and
Robertson [
Another efect we can investigate is to consider diferent systems
and topic populations. We can consider, for example, systems
ordered by efectiveness, topics ordered by dificulty, or even consider
the most representative subset of systems / topics selected according
to the strategy developed by Roitero et al. [
From the above equation, we can compute the Size of Efect index
ω2 which is an “unbiased and standardised index and estimates a
parameter that is independent of sample size and quantifies the
magnitude of diference between populations or the relationships
between explanatory and response variables” [
Our contribution is twofold: we propose an engineered pipeline based on network analysis and mixture models that can be used to detect bias and its causes in retrieval evaluation, and we present some preliminary result. We plan to conduct the experiments described, that will allow to better understand the efect and cause of bias and fairness in the retrieval evaluation.