Performance prediction is an important aspect of Information Retrieval (IR), as determining the efectiveness of search results without human relevance judgments has many important applications. We propose a novel Query Performance Prediction (QPP) method to predict the efectiveness of neural reranking models. Our approach uses the retrieval score distribution for a query and a set of highest-scoring documents to estimate the likelihood of efectiveness. This method is both eficient and unsupervised, making it possible to use in production retrieval systems. The new method uses entropy, which is the key measure in information theory. The core idea is simple but novel - measure the entropy of the retrieval scores for a reranking model while using no training data or corpus related statistics. Our empirical experiments show the efectiveness of our proposed method, which is comparable with traditional state-of-the-art QPP methods in terms of both prediction quality and computational eficiency.
Query Performance Prediction (QPP) has been an important open problem in Information
Retrieval (IR) research for many years [
More recently, neural Language Models (LMs) and neural re-ranking models have become
increasingly popular in modern IR systems. However, Datta et al. [
Our work is based on the hypothesis that score distributions difer for highly and poorly ranked search results. In contrast to prior work on score dispersion, which focused on finding the number of top documents required to compute SD, we propose the use of entropy to capture the dispersion.
A common type of post-retrieval QPP methods is score-based predictors. Score-based predictors use the retrieval scores (RSVs) generated by the retrieval system to estimate the success of the search. The simplicity and low computational cost of these methods make them an appealing choice in real-world applications as they do not require access to collection statistics, which can be computationally expensive. Additionally, they can be easily integrated into existing retrieval systems, and their predictions can be made in real-time.
A number of prior QPP methods estimate retrieval result efectiveness by measuring the
dispersion of the RSVs with SD [
Zendel et al. [
Retrieval methods. We test our proposed QPP approach using two probabilistic retrieval
models: Query Likelihood (QL) [
Evaluation methods. We report experimental results on an ad hoc retrieval TREC collection,
the Robust04 [
To evaluate our approach, we employ both correlation measures, as was reported in prior
work, as well as the recently proposed sARE and sMARE measures [
−8 −6 −4
−2 Document Score 0 2 4 Entropy for QPP. The entropy of a query q is calculated based on the relevance probability distribution of a set of documents Dq:
H(q) = −
X P (d|q) log P (d|q) where P (d|q) is the normalized probability of document d being relevant to the query q. The raw scores of QL and neural rankers, represented by S(q, d), must be normalized to be interpreted as probabilities. The normalization is achieved using the softmax function:
Entropy is maximized when the probabilities are uniform and indicates that the retrieval model was unable to efectively discriminate between good and bad documents. Conversely, lower entropy implies a more successful retrieval result. For convenience, in our experiments we use negative entropy, − H(q), as our QPP method.
A comprehensive evaluation on a benchmark dataset compares our method to the baseline SD method in terms of prediction quality, robustness, and similarity. Table 2 presents the prediction quality for both retrieval methods. It can be seen that for the task of document retrieval with the Robust04 collection, entropy performed comparably to SD, and was superior for the NeuralRanker. In terms of prediction quality, both methods produced similar results with only minor performance diferences.
Hyperparameter analysis. Both the SD- and entropy-based QPP approaches make use of a
hyperparameter, k, which represents the number of top-ranked documents that are included in
the prediction calculation. Figure 2 shows the performance of the entropy and SD approaches,
applied to predict the NeuralRanker results, with prediction quality measured using sMARE. It
is clear that entropy is much more robust to variations in k than SD. The prediction quality of
entropy remains consistent once it reaches k ≈ 400, after which it shows diminishing returns.
SD, on the other hand, reaches the minimal error using a much narrower range of documents,
which varies between collections and retrieval methods [
Method
Entropy SD
Method
Entropy SD 0 200
Similarity of the predictions. In Figure 3, we evaluate both methods for the NeuralRanker results on the Robust04 collection by calculating the sARE (error) values per query, plotting them as a categorical scatter plot. The methods have a correlation of Pearson’s r=0.5. It can be seen that while some queries receive similar sARE values, the majority of query values vary, implying that the predictors succeed and fail based on the query.
ER0.5 A s 0.0 E R AM0.25 s
Discussion: Entropy in neural networks. Cross-entropy loss is one of the most common optimization objectives used in neural networks. Neural LMs (e.g., BERT) are trained to minimize cross-entropy loss during pre-training. Many learning-to-rank losses, such as the pointwise binary cross-entropy loss, pairwise RankNet loss [22], and listwise ListNet loss [23], are derived from cross-entropy loss. In short, neural networks are likely to be naturally entropy-aware as they are optimized to minimize entropy. During training, the smaller the entropy is, the better the model is believed to be trained. This should be reflected at inference time as well, and we conjecture that the entropy of the predictions correlates to the model prediction performance. The smaller the inference entropy is, the more robust the model is, and thus it is more likely to be more efective.
Method
Entropy
SD 800 1000 2 D S 1 01 0 200
Discussion: entropy vs SD. Figure 2 shows that the optimal value of the hyperparameter k for SD is relatively small, whereas the optimal k for entropy is large in comparison. This could be related to their mathematical properties. Given k scored documents, the entropy (of the normalized probabilities) reaches the minimum (zero) when one document has a probability of 1.0 and others have probabilities of 0.0, and reaches the maximum (log(k)) when all the documents have the same probability; SD reaches the minimum of zero when all of the documents have the same score, and has no theoretical upper limit. However, if we know the upper bound b and lower bound a of the scores, Popoviciu’s inequality on variances shows that the upper limit of SD is b− 2a . That is, if we collect a and b for a large number of query-document pairs, we can derive a fixed empirical upper bound of SD. As depicted in Figure 4 (left), the upper SD is ifxed and independent of k. Consequently, information is lost by SD as k grows. This can be empirically observed in Figure 4 (right), which shows how a less discriminative SD compares to entropy when k =1000. Conversely, the maximum entropy increases as k is increased, which suggests that it retains information better than SD for large k. Finally, we note that the notion of entropy capturing more information than SD has been observed in other fields [24, 25].
This study evaluates the efectiveness of a new QPP method, entropy, and compares it to a similar baseline, SD, using a commonly used document retrieval collection Robust04. For Robust04, entropy performs similarly to SD when using QL, and outperforms it when using a NeuralRanker retrieval approach. The sensitivity of entropy to changes in the hyperparameter k (the number of top-ranked documents that are used in the prediction calculation) is also analyzed, and found to be more robust for entropy than for SD. Additional experiments demonstrate that SD and entropy typically succeed (or fail) for diferent queries, suggesting that the methods capture diferent properties of the distribution of the retrieval scores. Using entropy as a feature or in an ensemble model may ofer further improvements and would be interesting to explore in future work. Overall, our study provides evidence that entropy is a simple and promising method for QPP, and is more robust than currently used methods.
Acknowledgements. We thank the reviewers for their comments. This work was supported by the Australian Research Council’s Discovery Projects Scheme (grant DP190101113).
Evaluate and Understand Query Performance Prediction Methods, Inf. Retr. (2022). [22] C. Burges, T. Shaked, E. Renshaw, A. Lazier, M. Deeds, N. Hamilton, G. Hullender, Learning to Rank Using Gradient Descent, in: Proc. ICML, 2005, pp. 89–96. [23] Z. Cao, T. Qin, T. Liu, M. Tsai, H. Li, Learning to Rank: From Pairwise Approach to Listwise
Approach, in: Proc. ICML, 2007, pp. 129–136. [24] G. C. Philippatos, C. J. Wilson, Entropy, Market Risk, and the Selection of Eficient
Portfolios, Applied Economics 4 (1972) 209–220. [25] S. R. Bentes, R. Menezes, Entropy: A New Measure of Stock Market Volatility?, Journal of Physics: Conference Series 394 (2012) 012033.