IR evaluation measures are oen compared in terms of rank correlation between two system rankings, agreement with the users' preferences, the swap method, and discriminative power. While we view the agreement with real users as the most important, this paper proposes to use the Worst-case Condence interval Width (WCW) curves to supplement it in test-collection environments. WCW is the worst-case width of a condence interval (CI) for the dierence between any two systems, given a topic set size. We argue that WCW curves are more useful than the swap method and discriminative power, since they provide a statistically well-founded overview of the comparison of measures over various topic set sizes, and visualise what levels of dierences across measures might be of practical importance. First, we prove that Sakai's ANOVA-based topic set size design tool can be used for discussing WCW instead of his CI-based tool that cannot handle large topic set sizes. We then provide some case studies of evaluating evaluation measures using WCW curves based on the ANOVA-based tool, using data from TREC and NTCIR.
•Information systems ! Retrieval eectiveness; ANOVA; condence intervals; eect sizes; evaluation measures; p-values; sample sizes; statistical signicance
IR systems are built to satisfy users’ information needs, but it is not practical to make the users evaluate the systems all the time for the purpose of improving them—that would annoy the users, not satisfy them! Hence, we oen turn to IR evaluation measures in laboratory experiments. But which IR measures are good?
In laboratory studies, evaluation measures are oen compared
in terms of rank correlation between two system rankings (e.g. [
To supplement user-based studies of IR evaluation measures,
we propose to use Worst-case Condence interval Width (WCW)
curves in test-collection environments. WCW is the worst-case
width of a condence interval (CI) for the dierence between any
two systems, given a topic set size. We argue that WCW curves
are more useful than the swap method and discriminative power,
since they provide a statistically well-founded overview of the
comparison of measures over various topic set sizes, and visualise
what levels of dierences across measures might be of practical
importance. To this end, we leverage one of the publicly available
topic set size design Excel tools of Sakai [
When a new IR evaluation measure is invented, a system ranking
according to this measure (averaged over a set of topics) is oen
compared with another according to a well-established measure;
rank correlation measures such as Kendall’s τ or the top-heavy
τap [
For a given query, a user sees two Search Engine Result Pages (SERPs) side by side, and says that SERP1 is beer than SERP2 (“SERP1 > SERP2”). If an evaluation measure also says “SERP1 > SERP2,” this is a preference agreement; if it says “SERP1 < SERP2,” this is a preference disagreement. We can count the number of agreements over dierent queries and SERP pairs, and use it for comparing the “goodness” of evaluation measures. In practice, this approach also has a few limitations: (a) the judges employed in the preference assessments are oen not real search engine users with an information need; (b) human assessments can be unreliable and/or inconsistent; and (c) hiring judges comes at a cost, no maer how small.
e swap method [
Given a set of runs and an evaluation measure, a p-value can be
obtained for every system pair using an appropriate signicance
test, and the sorted p-values can be ploed against the system
pairs [
Sakai [
While the relationship between minD for ANOVA and n can be
ploed for dierent evaluation measures, this seems problematic as
a way to compare evaluation measures, since, for example, a minD
of 0.1 in term of one measure is not equivalent to a minD of 0.1 in
terms of another. In contrast, if we plot δ against n, this is probably
a more valid comparison since, at least for any normalised measures
that lie in the »0; 1¼ score range, we usually want the CI width to be
as small as possible. is is why we propose to plot δ against topic
set sizes to compare dierent measures. However, Sakai’s CI-based
tool cannot handle large topic set sizes: the limitation of his
CIbased tool is due to that of Excel’s GAMMA function: GAMMA(172) is
greater than 10307 and cannot be computed [
According to Sakai’s CI-based topic set size design, the initial topic
set size estimate for ensuring that the CI width for the dierence in
means for any two systems is no larger than δ ¹> 0º is given by [
δ 2 δ 2 where zinv ¹P º is the upper z-value3 for probability P . Subsequently, this estimate is incremented until it actually satises the requirement ¹α ; δ º. us, while the actual CI relies on a t -distribution, the method starts o with a standard normal distribution by assuming that the variance estimate σˆt2 is perfectly accurate4. is is why Eq. 1 involves a z-value rather than a t -value.
Whereas, according to Sakai’s ANOVA-based topic set size
design, the initial topic set size estimate for ensuring 100¹1 β º%
statistical power whenever the true dierence between the best
and the worst systems is minD or larger is given by [
Sakai [
λδ 2
nANOVA =
nCI
λ δ 2
4fzinv ¹α 2ºg2minD2 = 4fzinv ¹α 2ºg2 ¹ minD º : (3)
Here, note that 4fzinv ¹α 2ºg2 is a constant for a given α ; also, λ is
a constant given α ; β and m. Figure 1 visualises the relationship
between the two constants for α = 0:01; 0:05 and β = 0:10; 0:20,
while varying the number of systems m. e linear formulae for
approximating λ based on ϕ = m 1 [
4fzinv ¹α 2ºg2 holds when:
Condition (a) α = 0:05; β = 0:20; m = 10; or Condition (b) α = 0:05; β = 0:10; m = 5; or Condition (c) α = 0:01; β = 0:20; m = 18; or
Condition (d) α = 0:01; β = 0:10; m = 10.
Hence, whenever one of the above four conditions holds true, then from Eqs. 3 and 4 we obtain: nANOVA
δ 2 ¹ minD º : nCI us, when one of the above four conditions holds, by leing δ = minD in Eq. 5 we obtain nANOVAnCI 1, that is, nANOVA nCI , regardless of the variance estimate σˆ 2. Q.E.D.
Henceforth, we only consider the popular Cohen’s ve-eighty
convention [
Having proven that the ANOVA-based tool can be used instead of the less robust CI-based tool, we now demonstrate how dierent evaluation measures can be compared using WCW curves obtained with the ANOVA-based tool.
Table 1 shows the variance estimates σˆ 2 of various evaluation
measures reported in the literature [
Figure 3 shows the WCW curves that correspond to the variances shown in Table 1, for α = 0:05, i.e., 95% CIs. For each evaluation measure, the δ is ploed against the required topic set size n; the curve was obtained by entering dierent values of minD (i.e., δ ) into the ANOVA-based tool (with α = 0:05; β = 0:20; m = 10) and recording the resultant n. e advantages of the proposed WCW-based comparison of evaluation measures are as follows: Unlike discriminative power and the swap method, we can easily consider a wide range of topic set sizes; For a particular topic set size, we can easily compare across dierent evaluation measures, since an evaluation measure with a small WCW is usually more desirable than one with a large WCW under the same condition; e WCW curves can visualise the dierences among measures that practically maer.
For example, from Figure 3(b), when the topic set size is n = 50, it is clear that the WCW of nDCG and that of Q are about the same (around 0.16), while those of AP and nERR are substantially larger (around 0.23). Similarly, from Figure 3(d), while it is clear that the standardised (“std-AB”) measures have substantially lower WCW values than the unstandardised ones, the dierences within the set of standardised measures are probably not of practical importance, as indicated by the near-perfect overlaps of the curves.
We proposed to evaluate evaluation measures by comparing the WCW for various topic set sizes, using an existing ANOVA-based tool instead of the less robust CI-based tool. We proved the relationship between these two topic set size design methods, and demonstrated the advantages of WCW curves over well-known methods such as the swap test and discriminative power. It is hoped that this method will supplement user-based studies of evaluation measures.