We consider the issue of using fewer topics in the e ectiveness evaluation of information retrieval systems. Previous work has shown that using fewer topics is theoretically possible; one of the main issues that remains to be solved is how to nd such a small set of a few good topics. To this aim, in this paper we try a novel approach based on clustering of topics. We consider various algorithms, metrics, and various features of topics that can be helpful in identifying such a set.
E ectiveness evaluation is of paramount importance in the Information Retrieval
(IR) eld. The e ectiveness evaluation method commonly used by Text REtrieval
Conference (TREC) and other similar initiatives is based on a set of topics,
which are textual descriptions of information needs, and on computing relevance
assessments for these topics in a pooled set of documents. Participants run their
own retrieval systems on the topics provided, and return a ranked list of the
retrieved documents. Those are then pooled and their relevance is judged by
human assessors. The process of creating these relevance assessments is very
expensive in terms of money (circa $30M from 1999 to 2009 according to [
We present a novel approach, based on clustering, to select such a good subset of topics. This paper is structured as follows. Sect. 2 describes the state of the art and the research question, Sect. 3 describes the experiments performed. We conclude in Sect. 4, where we also provide some directions for future work.
The starting point of our research is shown in Table 1. Each row represents a system (or, in TREC terms, run). Each column represents a topic. Each AP(si; tj ) represents the e ectiveness of the system si on the topic tj . AP stands for the Average Precision measure, while MAP stands for Mean AP. . . .
tn
MAP AP(s1; tn) MAP(s1) . .
.
AP(sm; tn) MAP(sm)
Guiver et al. [
It is reasonable to conjecture that similar topics will evaluate systems in a
similar way, and therefore they might be redundant: by removing some of the
similar topics the nal system evaluation should not change much. When
looking for smaller topic subsets a natural strategy is then: (i) to cluster topics into
a set (henceforth we call this set a cluster-set of topics), and (ii) to select a
topic representative of each cluster and form a topic subset (henceforth we call
this set a one-for-cluster -topic subset). Therefore, we inquire into the following
3 The values are justi ed on the basis of previous proposals: [
To answer our research question we build a cluster-set of topics. We use the
columns of Table 1 as vectors, so that each topic is represented with an
mdimensional vector tk = fAP(s1; tk); : : : ; AP(sm; tk)g, where AP(si; tj )
represents the e ectiveness of the evaluation of the system si on the topic tj . We use
hierarchical clustering, with complete method to join clusters, and the cosine
distance. Then, in order to compare the one-for-cluster set with the average set
(similar to the comparisons done in [
We also tried other methods and features to cluster topics: we considered other methods to join clusters (i.e., single, average, Ward); we considered another clustering algorithm, k-means; we changed from cosine distance to other ones (i.e., Euclidean, Manhattan, . . . ). We used also other topics features: we considered the set formed by the identi ers (and the numbers) of the documents (and the relevant ones) in the pooled set, the textual description of topics (only the title eld and the full description), plus a set of experiments in which we considered combinations of all the features and all the parameters. In all these experiments the one-for-cluster topic subset shows the same correlation values (or worse) than the average-topic subset.
These results seem to rule out the possibility of using clustering to select a few good topics, at least when the columns of Table 1 are used as vectors. 8 . 0 2 . 0 0 . 0 ● ●● ● ●● ●● ● ● ● ●● KKPeeeannrddsaaoll n''ss'sττρ:::aoavneveerapragegerecluster
Pearson's ρ: one per cluster 2 . 0 and
for one-for-cluster and average set 4
In summary, we did not nd any evidence supporting our research question clustering does not seem usable to nd a good-topic subset. Furthermore, we believe we provided convincing evidence that our results are independent from the parameters used to form the cluster-set or from the topic features used. We started from the hypothesis that clustering can be used to chose a subset of a few informative topics in the evaluation of information retrieval e ectiveness. We believe that we have provided convincing evidence that performing clustering is not e ective in the choice of a good-topic subset, at least when straightforward clustering strategies are used. It might be that more complex approaches, that we intend to try in the future, provide better results: for example, we did not yet try dimensionality reduction.