Despite the retrieval effectiveness of queries being mutually independent of one another, the evaluation of query performance prediction (QPP) systems has been carried out by measuring rank correlation over an entire set of queries. Such a listwise approach has a number of disadvantages, notably that it does not support the common requirement of assessing QPP for individual queries. In this paper, we propose a pointwise QPP framework that allows us to evaluate the quality of a QPP system for individual queries by measuring the deviations between each prediction versus the corresponding true value, and then aggregating the results over a set of queries. Our experiments demonstrate that this new approach leads to smaller variances in QPP evaluations across a range of different target metrics and retrieval models.
A major disadvantage of listwise QPP approaches is that evaluation is conducted in a relative
manner, so the performance of one query is measured relative to the others. However, a
downstream performance estimate of an individual query also needs to be evaluated independently
of the other queries. In contrast, a pointwise approach measures the effectiveness on individual
queries, and then, if required, aggregates the results over a complete set. This is analogous to
measuring the retrieval effectiveness metric MAP by computing the average precision values
for individual queries and then aggregating them. Pointwise evaluation also allows us to carry
out a per-query analysis of a method often leading to useful insights. For instance, Buckley [
Another drawback of listwise methods is that they can be overly sensitive to the configuration
setup used for evaluation. The two most important such configurations are: i) the target retrieval
evaluation metric that induces a ground-truth ordering over the set of queries; ii) the retrieval
model used to obtain the top- set of documents for QPP estimation. Indeed, variations in
these configurations can lead to both large standard deviations in the reported rank correlation
measures and significant differences in the relative ranks of various QPP systems [
Correlation with listwise ground-truth Before describing our new QPP evaluation framework APAE, we begin by introducing the required notation. Formally, a QPP estimate is a function of the form (, ()) ↦→ R, where () is the set of top- ranked documents retrieved by an IR model for a query ∈ , a benchmark set of queries.
For the purpose of listwise evaluation, for each ∈ , we first compute the value of a target
IR evaluation metric, () that reflects the quality of the retrieved list (). The next step
uses these () scores to induce a ground-truth ranking of the set , or in other words, arrange
the queries by their decreasing (or increasing) () values, i.e.,
= { ∈ : () > (+1), ∀ = 1, . . . , || − 1}}
(1)
Similarly, the evaluation framework also yields a predicted ranking of the queries, where this
time the queries are sorted by the QPP estimated scores, i.e.,
= { ∈ : () > (+1), ∀ = 1, . . . , || − 1}
(2)
A listwise evaluation framework then computes the rank correlation between these two ordered
sets ( , ), where : R|| × R|| ↦→ [
∑︁ ( (), ())
Unlike the rank correlation , here is a pointwise correlation function of the form : R× R ↦→ R.
It is often convenient to think of as the inverse of a distance function that measures the extent
to which a predicted value deviates from the corresponding true value. In contrast to ground-truth
evaluation metrics, most QPP estimates (e.g., NQC, WIG etc.) are not bounded within [
estimator will be agnostic with respect to the target IR metric . For instance, NQC scores can be
seen as being approximations of AP@100 values, but can also be interpreted as approximating
any other metric, such as nDCG@20 or P@10. Therefore, a question arises around which metric
should be used to compute the individual correlations in Equation 3. Of course, the results can
differ substantially for different choices of , e.g., AP or nDCG. This is also the case for listwise
QPP evaluation, as reported in [
Metric-agnostic pointwise QPP evaluation For a set of evaluation functions ∈ ℳ (e.g., ℳ = {AP@100, nDCG@20, . . .}), we employ an aggregation function to compute the overall pointwise correlation (Equation 3) of a QPP estimate with respect to each metric. Formally, (, ℳ, ) = Σ ∈ℳ(1 − | () − ()/ℵ|), (4) where Σ denotes an aggregation function (it does not indicate summation). In particular, we use the most commonly-used such functions as choices for Σ: ‘minimum’, ‘maximum’, and ‘average’ – i.e., Σ ∈ {avg, min, max}. Next, we find the average over these values computed for a given set of queries , i.e., we substitute (, ℳ, ) from Equation 4 into the summation of Equation 3.
A QPP experiment context [
RQ1 would indicate that our proposed metric APAE is consistent with existing metrics used for QPP evaluation, while an affirmative answer to
that APAE is preferable to existing methods due to its higher stability with respect to different experimental settings.
We conduct our QPP experiments on the TREC Robust dataset, which consists of 249 topics.
Following the standard practice for QPP experiments [
consistency of APAE with respect to three standard listwise QPP evaluation metrics: Pearson’s ,
Spearman’s and Kendall’s ; and a pointwise approach, scaled Absolute Rank Error (sARE)
[
The results presented in Table 2 answer RQ1 in the afrfimative. Each reported value here corresponds to the rank correlation (Kendall’s ) between the relative ranks of the QPP systems ordered by their effectiveness as computed via one of the standard metrics (one of , , or sARE) and APAE, i.e., one of avg(ℳ), min(ℳ) and max(ℳ)). The high correlation values between the standard listwise and the proposed pointwise metrics show that APAE can be used as a substitute for the standard listwise evaluation. Notably, we see that the average aggregate function yields the best results, and hence for the subsequent experiments we use avg(ℳ) as the
The correlation of our proposed pointwise evaluation metric APAE with the standard listwise metrics Pearson’s , Spearman’s , Kendall’s and sARE. The rank correlations between each pair of QPP system ranks (evaluated with a listwise measure and a pointwise measure) were computed with Kendall’s . The high values indicate that the pointwise measurement can effectively substitute a standard list-based measure, since they lead to a fairly similar relative ordering between the effectiveness of different QPP avg(ℳ) min(ℳ)
sARE sARE max(ℳ) BM25 LMDir LMJM 0.810 0.810 0.905 0.887
the IR metrics.
Metric
Model
BM25 (0.7, 0.3) BM25 (0.3, 0.7) LMDir (500) LMJM BM25 BM25 LMDir LMDir (0.6) (0.7, 0.3) (0.3, 0.7) (500) (1000) pairs of IR metrics and IR models. Red cells indicate the lowest value in each group, while the lowest values along each column are bold-faced. (a) Correlations between the relative ranks of 7 different
QPP systems across different pairs of IR target metrics. QPP systems were evaluated with the baseline listwise metric - Kendall’s .
(b) Similar to Table 3a, except QPP performance was evaluated with the pointwise approach APAE. A comparison with Table 3a indicates a better consistency in the relative ranks of QPP systems for variations in Metric
Model
BM25 (0.7, 0.3) BM25 (0.3, 0.7) (c) Here rank correlations between the relative ranks of
QPP systems are measured across IR model pairs.
As in Table 3a, QPP systems were evaluated with .
The numbers alongside the IR models denote their respective parameters.
(d) Unlike Table 3c, here the QPP outcomes were evaluated by APAE (instead of ).
the relative stability of QPP system ranks for variations in QPP contexts (i.e., different IR models and target metrics), comparing both listwise and pointwise approaches (see Table 3). To clarify with an example, if working with three QPP methods, say AvgIDF, NQC, WIG, we observe that (NQC) > (WIG) > (AvgIDF) for LMDir as measured relative to AP@100. We expect to observe a similar ordering for a different choice of the IR model and target IR metric, say BM25 with nDCG@100. As in our previous experiments, here we measure the rank correlations between a total of seven QPP systems (see Table 1) via Kendall’s .
Unlike the standard listwise QPP evaluation mechanism of measuring an overall rank correlation with respect to a reference ranking of the queries (in terms of retrieval effectiveness), we have proposed a pointwise evaluation method that computes the relative difference between a normalized QPP score and a true IR evaluation measure (e.g., AP@100 or nDCG@20). Our experiments demonstrated that the proposed metric exhibits a high correlation with standard listwise approaches and is more robust to changes in QPP experimental setup than listwise evaluation measures. Using this metric, it should thus be possible to evaluate the effectiveness of different QPP methods on downstream tasks on a per-query basis.
Acknowledgement.. The first and the third authors were supported by the Science Foundation Ireland (SFI) grant number SFI/12/RC/2289_P2.