When responding to any question from the user or an API, a conversational search or question answering system should ideally be able to attach an appropriate confidence score to its output. While such systems are often overconfident, there are also situations where the system responds correctly yet lacks enough confidence. Underconfident responses cannot be relied upon, and therefore may not be utilised by the user or downstream tasks. Ideally, we want to know when systems are underconfident as well as when they are overconfident, and want to suppress both phenomena in a balanced manner. Furthermore, in this scenario, we want an evaluation measure that is guaranteed to (a) penalise a lowered confidence for a correct response; and also (b) penalise a raised confidence for an incorrect response. In light of this, we propose HMR (Harmonic Mean of Rewards) and demonstrate its advantages over existing calibration measures for our purpose by means of examples, axioms, and theorems.
by means of examples, axioms, and theorems. More specifically, we show that while HMR possesses Properties (a) and (b) mentioned above, none of the existing calibration measures do.
In calibration tasks, the Expected Calibration Error (ECE) is probably the most widely used evaluation
measure. ECE is defined in Pakdaman Naeini et al. [
=1 Note that instance binning is a necessity for the introduction of the notion of binwise accuracy.
Two simple binning methods are commonly used in the literature: equal width binning (where the
[
One of the weaknesses of ECE and MCE is that they rely on the parameter . Hence, we also discuss existing binning-free calibration measures.
Consider a classification task with (≥ 2) classes with instances to classify; let GOLD = 1 if Class is the true class for the -th instance, and 0 otherwise. For a classifier that returns probabilities (1 , . . . , ) s.t. ∑︀
=1 = 1 for each instance, the Brier score [
BR = =1 =1 1 ∑︁ ∑︁ ( − GOLD)2 .
Brier proposed this measure in 1950 for verifying weather forecasts. To ensure a [
In 2021, Gupta et al. [19] proposed a binning-free measure called KS, inspired by the KolmogorovSmirnov test for equality of two distributions [20]. Given confidence scores (e.g., top probabilities), let (1, . . . , ) be these scores after an ascending sort, and let GOLD = 1 if the instance that corresponds to the -th score in the sorted list is correct, and 0 otherwise. Then,
cp = 1 ∑=︁1 , cGOLD =
1 ∑︁ GOLD ,
=1
KS = max |cp − cGOLD | .
Recall that, in classification tasks with classes, a system response may be associated with
probabilities rather than one confidence score; in principle, measures like ECE/MCE and KS may be
applied to non-top probabilities as well. Some studies have in fact incorporated non-top probabilities in
calibration evaluation [
Also in 2021, Minderer et al. [22, Section 5] empirically compared ECE with BR (along with negative log-likelihood) in the context of image classification. However, as their interest also lay in traditional calibration, the distinction between overconfidence and underconfidence was not discussed.
We propose a very simple and interpretable binning-free evaluation approach that first quantifies overconfidence and underconfidence separately. For a given system, let − and + denote the sets of instances for which the system’s choices are considered incorrect and correct, respectively (|− |+|+| = ). Let () denote the system’s confidence for Instance . Then, for each ∈ − (the system is incorrect), () should be as close to 0 as possible; whereas for ∈ + (the system is correct), () should be as close to 1 as possible. Hence, we first define the Rewards for suppressing overconfidence and underconfidence separately as follows.
= ∑︁ () , = ∑︁ (1 − ()) ,
∈+ ∈− = = {︃1 {︃1
if − = ∅ , 1 − /|− | otherwise . 1 − /|+| otherwise .
if + = ∅ , (5) (6) (7) (8) Note its advantage over the arithmetic mean. For example, consider two situations, R = R = 0.5 and R = 0.9, R = 0.1: the arithmetic means of R and R are the same, but HMR = 0.500 for the former and HMR = 0.180 for the latter.1
Despite its simplicity, our measure is clearly advantageous over existing calibration measures for the purpose of penalising overconfidence and underconfidence separately, as we shall demonstrate below.
In this section, we demonstrate how the proposed and existing measures can actually be computed, to clarify how (or whether) they work. The examples will also help us prove our theorems presented in Section 5 that generalise our observations. 4.1. Example 1 1Following the approach of the F-measure [24, 25], HMR can easily be generalised as ( 2+1) , where (≥ 0) is a 2+ parameter that means “undergeneralisation” is times as important as overgeneralisation.” 2Note that, with the exception of (N)BR, the measures discussed in this paper can be applied to situations where (≥ 2) varies across instances, for example, when the number of answer candidates within a system varies depending on the question: we can still take one probability per instance (e.g., top probability) for the evaluation.
Note, for example, that when − = ∅ (i.e., all system responses are correct), there is no way for the system to be overconfident for any of the instances and therefore = 1 (i.e., perfection).
As we want systems to balance the above two rather than to sacrifice one for the sake of the other, let us consider the Harmonic Mean [23]:
HMR = {︃0
if = = 0 , 2 R R /(R + R ) otherwise .
HMR↑ ECE↓ MCE↓ NBR↓ KS↓
X Y Pick one top probability that represents a correct case, and lower it while keeping it the top probability, thereby injecting underconfidence ; Z Pick one top probability that represents an incorrect case, and raise it, thereby injecting overconfidence ; W Apply both of the above perturbations.
Note that the above perturbations do not afect the top-probability-based accuracy which is 7/9 = 0.778 for this example. The perturbed probabilities are underlined in Table 1.
For our task where we are concerned with underconfidence and overconfidence of system responses, we would like to be able to say that Y, Z, and W all underperform X. However, for this example, only HMR and NBR satisfy this requirement, as shown in Table 2. Here, the results that we want (intuitive results) are shown in bold, and the counterintuitive ones are underlined. Note that HMR is a reward measure (i.e., higher means better), while the others quantify errors (i.e., lower means better), as indicated by the arrows. Below, we demonstrate how some of the numbers in Table 2 are obtained in order to clarify how the measures work (or not). We shall leave the discussion of NBR to the Appendix, in which we provide a diferent example where NBR gives counterintuitive scores for Y, Z, and W. Recall that, unlike the other measures, NBR relies on the probability for every class for each instance.
For System X in Table 1, = 0.4 + 0.6 = 1.0, = 2 * (1 − 0.4) + 2 * (1 − 0.6) + 3 * (1 − 0.8) = 2.6 (Eq. 5). Since | − | = 2, | +| = 7, = 1 − 1.0/2 = 0.500 (Eq. 6) and = 1 − 2.6/7 = 0.629 (Eq. 7). Hence X is more overconfident than underconfident ; note that this observation is not possible with the other measures. Finally, HMR() = 0.557 (Eq. 8).
Similarly, for System W, = 1.2 (same as Z), and = 2.7 (same as Y); = 0.400 (worse than X in terms of overconfidence), and = 0.614 (worse than X in terms of underconfidence). Hence HMR( ) = 0.485 (worse than X overall).
The instances in Table 1 are already sorted by top probability and binned for computing ECE (and MCE): we have = 3 bins, each containing three instances. The binwise accuracies () are (2/3, 2/3, 3/3) for all systems. For X, the average confidences ( ¯) are clearly (0.400, 0.600, 0.800); on the other hand, for Z which has an overconfidence injected in Bin 1, the average confidences are (0.467, 0.600, 0.800). Hence, the binwise absolute diferences ( |¯ − | in Eq. 1) are (0.267, 0.067, 0.200) for X, and (0.200, 0.067, 0.200) for Z. Thus, even though Z is more confident than X about the third instance (and they are both incorrect), ECE() = 0.178, ECE() = 0.156, MCE() = 0.267, MCE() = 0.200. That is, both ECE and MCE say that Z is better.
The above flaw arises as follows. For X, note that 1 = 2/3 > ¯1 = 0.400: that is, for Bin 1, X is underconfident on average . Hence the absolute diference |¯1 − 1| = 0.267 actually quantifies how underconfident X is for Bin 1. Now, the perturbation introduced in Z raises ¯1 (as Z is more confident than X about the third instance), and therefore Z is considered to be “less underconfident” than X for Bin 1. From this discussion, it is clear that binwise averaging of confidences is not a good idea for the purpose of evaluating both overconfidence and underconfidence while trying to separate them, as averaging confounds both phenomena.
Note also that in Table 2, MCE fails to detect the perturbation introduced in Y for Bin 3. This is because, although the average confidence ¯3 is lowered from 0.800 to (0.7 + 2 * 0.8)/3 = 0.767 and hence the absolute diference |¯3 − 3| = |¯3 − 1| is raised from 0.200 to 0.233, this new value is still smaller than the unchanged absolute diference for Bin 1: |¯1 − 1| = 0.267. In other words, when Y is obtained from X by perturbing Bin 3, MCE keeps looking at Bin 1 and ignores the change. Thus, although MCE was proposed to consider extreme cases, binwise averaging of confidences prior to applying the max operator (Eq. 1) can hide what is happening to individual instances.
Table 3 shows how KS scores are computed for Systems X and Z shown in Table 1 according to Eq. 4. Note that KS also requires instance sorting, and recall that Table 1 already provides the instances sorted
HMR↑ ECE↓ MCE↓ NBR↓ KS↓
X by top probabilities. It can be verified that, even though Z is overconfident about the third instance ( = 3) compared to X (where both systems are incorrect), KS says that Z is better.
The above flaw arises as follows. In Table 3, note that cp2 = 0.089 < cGOLD2 = 0.222 for both systems: the former is much smaller, even though KS requires the cp distribution to align with the cGOLD distribution. In other words, at = 2, the systems are on the side of underestimation so far. Therefore, if we raise 3 (from 0.4 to 0.6), this brings the cp distribution “closer” to the cGOLD distribution: it can be verified that, while cGOLD3 = 0.222, cp3 = 0.133 for X and cp3 = 0.156. Hence the counterintuitive result. 4.2. Example 2 In our first example (Tables 1-2), ECE and KS managed to say that Y (perturbed by injecting underconfidence for a correct case) is worse than X. Our second example, presented in Tables 4-5 ( = 3, = 9, with Y, Z, W perturbed as described earlier), shows that ECE and KS fail to do so; The same goes for MCE. From Table 4, it can be observed that the top probability of X for the first instance ( = 1) has been lowered from 0.5 to 0.4 in order to obtain Y, even though both X and Y are correct for this instance. Below, we examine why ECE, MCE, and KS say that Y is better.
From Table 4, the binwise accuracies () are (1/3, 2/3, 2/3); the average confidences ( ¯) are (0.500, 0.600, 0.700) for X, and (0.467, 0.600, 0.700) for Y due to the injection of underconfidence. Hence the binwise absolute diferences ( |¯ − |) are (0.167, 0.067, 0.033) for X, and (0.133, 0.067, 0.700) for Y. Therefore, from Eq. 1, MCE (which reflects only Bin 1) and ECE are smaller (i.e., “better”) for Y.
The above flaw arises as follows. Note that 1 = 1/3 < ¯1 = 0.500 for X; hence the absolute diference for Bin 1 actually quantifies overconfidence . Therefore, Y, which is less confident in Bin 1 due to the perturbation, is considered to be “less overconfident” than X. Again, it is clear that binwise averaging is not a good idea in our context.
The examples discussed in Section 4 demonstrated how the measures are actually computed, and how ECE, MCE, and KS can be counterintuitive for our purpose. (As mentioned earlier, counterintuitive cases for NBR are provided in the Appendix.) However, examples are examples: this section clarifies the advantages of HMR in terms of axioms that it satisfies, to generalise our previous observations.
All three axioms presented below start with the following common prerequisite. Consider a sequence of binary correctness labels for instances; the label for Instance is denoted by GOLD(). Under this setting, consider System that returns a sequence ⟨1, . . . , ⟩ of confidence scores (i.e., probabilities) for the same instances, where the scores have been sorted in ascending order (just for computing ECE, MCE, and KS). Let denote the -th instance in the sorted list; then the corresponding sequence of the correctness labels can be denoted as ⟨GOLD(1), . . . , GOLD( )⟩.
Theorem U-HMR HMR satisfies Axiom-U.
Proof: The perturbation described in Axiom-U does not afect (Eq. 5) and hence does not afect either (Eq. 6): for brevity, let = denote the unafected reward. On the other hand, the perturbation increases (Eq. 5) and hence decreases (Eq. 7): that is, if we let and denote the for X and the for Y, respectively, then > (≥ 0). From Eq. 8, HMR() = 2/( + ) since > 0. We need to show that Δ = HMR() − HMR( ) ≥ 0.
Suppose that = 0, i.e., = |− | (Eq. 6), that is, both X and Y are 100% confident for every incorrect case. Then HMR() = 0/ = 0. If > 0, HMR( ) = 2/( + ) = 0/ = 0; if = 0, then = = 0 so HMR( ) = 0 (Eq. 8). Either way, Δ = 0 − 0 = 0.
Otherwise (i.e., if > 0), Δ = 2/( + ) − 2/( + ) = 22( − )/( + )( + ) > 0. Theorem O-HMR HMR satisfies Axiom-O.
Proof: The perturbation described in Axiom-O does not afect (Eq. 5) and hence does not afect either (Eq. 7): for brevity, let = denote the unafected reward. On the other hand, the perturbation increases (Eq. 5) and hence decreases (Eq. 6): that is, if we let and denote the for X and the for Z, respectively, then > (≥ 0). Since > 0, HMR() = 2/( + ). We need to show that Δ′ = HMR() − HMR() ≥ 0.
Suppose that = 0, i.e., = |+| (Eq. 7), that is, both X and Z are 0% confident for every correct case. Then HMR() = 0/ = 0. If > 0, HMR() = 2/( + ) = 0/ = 0; if = 0, then = = 0 so HMR() = 0 (Eq. 8). Either way, Δ′ = 0 − 0 = 0. Theorem UO-HMR HMR satisfies Axiom-UO.
Proof: Based on the proofs of Theorems U-HMR and O-HMR, it is clear that the two perturbations described in Axiom-UO decrease both (due to the -th instance) and (due to the ′-th instance). Hence the harmonic mean (Eq. 8) also decreases; that is, HMR() − HMR( ) ≥ 0. Moreover, from the proofs of U-HMR and O-HMR, it follows that the equality can hold only when both X and W are 100% confident for every incorrect case and 0% confident for every correct case. However, we know that this is not possible: if X is 100% confident for every incorrect case, it is not possible to further inject overconfidence; if X is 0% confident for every correct case, it is not possible to further inject underconfidence. Hence, HMR() > HMR( ) holds: W is strictly inferior to X.
To prove that none of ECE, MCE, NBR, and KS satisfy any of the axioms, providing one actual counterexample for each situation sufices. Table 8 provides the counterexamples necessary: we discussed Examples 1 and 2 in Section 4; Example 3 is discussed in the Appendix.
For the purpose of penalising both overconfidence and underconfidence in system responses while balancing the two, we proposed a simple and intuitive evaluation measure called HMR. We proved that HMR satisfies our axioms (i.e., penalising a lowered confidence for a correct response, penalising a raised confidence for an incorrect response, and penalising a system that reflects both perturbations), and that existing calibration measures do not. Hence, while we do not claim that HMR should replace existing calibration measures in all calibration tasks, we do recommend its use in tasks where our axioms make sense.
We designed HMR primarily for conversational search systems where each response is either correct or not and has a confidence score; the score could represent a top probability (or more generally, the -th highest probability) among the probabilities for diferent response candidates; the candidates may be generated by the system itself or given to the system from outside, as in multiple choice questions. However, HMR can be used in any task where the system response has a binary gold label and one confidence score. As the present study is limited to axiomatic discussions with toy data, we plan to utilise HMR with real data in a shared task in our future work. [14] P. Liang, R. Bommasani, T. Lee, D. Tsipras, D. Soylu, M. Yasunaga, Y. Zhang, D. Narayanan, Y. Wu, A. Kumar, B. Newman, B. Yuan, B. Yan, C. Zhang, C. Cosgrove, C. D. Manning, C. Ré, D. AcostaNavas, D. A. Hudson, E. Zelikman, E. Durmus, F. Ladhak, F. Rong, H. Ren, H. Yao, J. Wang, K. Santhanam, L. Orr, L. Zheng, M. Yuksekgonul, M. Suzgun, N. Kim, N. Guha, N. Chatterji, O. Khattab, P. Henderson, Q. Huang, R. Chi, S. M. Xie, S. Santurkar, S. Ganguli, T. Hashimoto, T. Icard, T. Zhang, V. Chaudhary, W. Wang, X. Li, Y. Mai, Y. Zhang, Y. Koreeda, Holistic evaluation of language models, 2023. arXiv:2211.09110. [15] A. Kumar, P. Liang, T. Ma, Verified uncertainty calibration, 2020. arXiv:1909.10155. [16] G. W. Brier, Verification of forecasts expressed in terms of probability, Monthly Weather Review 78 (1950) 1 – 3. URL: https://journals.ametsoc.org/view/journals/mwre/78/1/1520-0493_1950_078_ 0001_vofeit_2_0_co_2.xml. doi:https://doi.org/10.1175/1520-0493(1950)078<0001: VOFEIT>2.0.CO;2. [17] Y. Ovadia, E. Fertig, J. Ren, Z. Nado, D. Sculley, S. Nowozin, J. V. Dillon, B. Lakshminarayanan, J. Snoek, Can you trust your model’s uncertainty? evaluating predictive uncertainty under dataset shift, 2019. arXiv:1906.02530. [18] K. Tian, E. Mitchell, A. Zhou, A. Sharma, R. Rafailov, H. Yao, C. Finn, C. Manning, Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models finetuned with human feedback, in: H. Bouamor, J. Pino, K. Bali (Eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics, Singapore, 2023, pp. 5433–5442. URL: https://aclanthology.org/2023.emnlp-main.330. doi:10.18653/v1/2023.emnlp-main.330. [19] K. Gupta, A. Rahimi, T. Ajanthan, T. Mensink, C. Sminchisescu, R. Hartley, Calibration of neural networks using splines, 2021. arXiv:2006.12800. [20] W. L. Hays, Statistics (Fifth Edition), Harcourt Brace College Publishers, 1994. [21] J. Vaicenavicius, D. Widmann, C. Andersson, F. Lindsten, J. Roll, T. Schön, Evaluating model calibration in classification, in: K. Chaudhuri, M. Sugiyama (Eds.), Proceedings of the TwentySecond International Conference on Artificial Intelligence and Statistics, volume 89 of Proceedings of Machine Learning Research, PMLR, 2019, pp. 3459–3467. URL: https://proceedings.mlr.press/v89/ vaicenavicius19a.html. [22] M. Minderer, J. Djolonga, R. Romijnders, F. Hubis, X. Zhai, N. Houlsby, D. Tran, M. Lucic, Revisiting the calibration of modern neural networks, 2021. URL: https://arxiv.org/abs/2106.07998. arXiv:2106.07998. [23] T. Sakai, Evaluating evaluation measures for ordinal classification and ordinal quantification, in: C. Zong, F. Xia, W. Li, R. Navigli (Eds.), Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), Association for Computational Linguistics, Online, 2021, pp. 2759–2769. URL: https://aclanthology.org/2021.acl-long.214. doi:10.18653/v1/2021. acl-long.214. [24] C. J. van Rijsbergen, Information Retrieval, Butterworths, 1979. [25] T. Sakai, Metrics, Statistics, Tests, Springer Berlin Heidelberg, Berlin, Heidelberg, 2014, pp. 116–163.
URL: https://doi.org/10.1007/978-3-642-54798-0_6. doi:10.1007/978-3-642-54798-0_6.
This section discusses our third example, which demonstrates that NBR can be counterintuitive when the perturbations described in Section 4.1 are applied to System X in order to obtain Y, Z, and W.
Table 9 presents our third example with = 6, = 3; Table 10 shows the HMR, NBR, and KS scores computed from Table 9. ECE and MCE are omitted here, as these measures require instance binning and binwise averaging of confidences but we only have three instances.
For X, the sum of squared errors (Eq. 2) for the third instance ( = 3) is (0.6 − 1)2 + 0.42 = 0.32. In contrast, for Y, the corresponding value is (0.5 − 1)2 + 5 * 0.12 = 0.30; this is why NBR rates Y higher than X. Meanwhile, for X, the sum of squared errors for the second instance ( = 2) is 0.42 + 0.32 + 3 * 0.12 = 0.68. In contrast, for Z, the corresponding value is 0.52 + 0.42 + 0.12 = 0.62; this is why NBR rates Z higher than X. Finally, NBR also rates W higher than X, as W reflects both of the above changes in sum of squared errors.
As a final remark, note that KS completely fails to detect the perturbations in Table 10.