Understanding the confidence with which a machine learning (ML) model classifies an input datum is an important, and often over-looked, concept. We propose a new probability calibration metric, the Entropic Calibration Diference (ECD). Inspired by existing research in the field of state estimation, specifically target tracking, we show how ECD may be applied to binary and multi-class classification ML models. We describe the relative importance of under- and over-confidence and how they are not conflated in the tracking literature. Indeed, our metric naturally distinguishes under- from over-confidence. We consider this important given that algorithms that are under-confident are likely to be “safer” or more trustworthy than algorithms that are over-confident, albeit at the expense of also being over-cautious and so statistically ineficient. We demonstrate how this new metric performs on real data and present a theoretical analysis to prove its properties. We also compare with other metrics for ML model probability calibration, including the Expected Calibration Error (ECE).
Calibration of probabilities is an important and often overlooked concept when developing machine learning (ML) models. Usually, accuracy is the main metric used to calculate how well an ML model performs in terms of predicting a class for unseen data. Generally speaking, the closer the accuracy is to 100%, the better the model is deemed to be. However, this does not take into account the probability of predictions that the model outputs, which can be just as important, if not more, than the accuracy.
In binary classification, a probability greater than a threshold, typically 0.5, is enough to decide whether an input belongs to one of two classes. While accuracy informs whether a classification is correct, a probability calibration metric informs how well the confidence probabilities match the true proportions of correct decisions. For example, a model that always outputs a probability of 0.6 for class label 1, but always classifies correctly, should produce a poor calibration score, as even though the model classifies the input correctly, it has low confidence in that decision.
Calibration has become even more important as of late, as the research of Guo et al. [
A well-calibrated model is defined as one that outputs probabilities that are representative of the real-life occurrences from the unseen data. For example, if, on average, 70% of people are correctly predicted to contract a certain disease, then one would expect the average probability outputted by a diagnosis model to be 0.7. A mathematical representation of calibration can be seen in (1), where is the true label, represents a class from the total number of classes , ˆ is the predicted probability distribution, p is the vector of class confidences with elements , and P is the true probability.
P( = |ˆ = p) = ∀ ∈ {1, ..., } (1)
In this paper, we present a novel calibration metric that addresses some weaknesses of some of the most commonly-used existing metrics. In section 2, we discuss existing calibration metrics that are widely discussed in the literature. In section 3, we detail our motivations for ‘safe’ or trustworthy calibration and why we feel our metric is necessary. Section 4 defines our new metric and details how the results can be interpreted. In section 5, we use hypothesis tests for our metric on pre-trained models using calibrated and uncalibrated probabilities and make a comparison with other metrics. Results for binary classifiers are included in the main paper, while multiclass results are found in Appendix A. Finally, section 6 concludes the paper.
In this section, we explore the literature and highlight some important calibration metrics, from early pioneers of the field to widely-used modern standards.
Various methods exist that attempt to calibrate the probabilities outputted by badly calibrated models.
Some attempt to do this by altering their training process, such as using a loss function that addresses
neural network calibration as it is being trained [
Methods have been developed to allow one visually to assess the calibration of a model. First brought
to the limelight by DeGroot and Fienberg [
Once every probability has been assigned to the appropriate bin, the average predicted probability
and fraction of class 1 labels is calculated for each bin. For multiclass reliability diagrams, a common
method is to take the top-label, also known as most confident, prediction [
Reliability diagrams can be a good tool for visualising a model’s calibration. However, they do not, in themselves, return a calibration score, and the diagram can be misleading when some bins are sparsely populated.
Initially proposed by Naeini et al. [
ECE works by calculating the accuracy and confidence of each bin in a reliability diagram. The
accuracy is the traditional definition, where the number of correct predictions is divided by the total
number of predictions in the bin. The confidence refers to the average probability in a bin. It is not
explicitly stated in the original ECE paper, but Guo et al. [
1 |∑︁| ˆ || =1 () =
1 |∑︁| 1( = 1) || =1 = ∑︁ || | () − ( )| (4)
=1
While ECE is widely-used in the literature, it has some shortcomings. One of the most noticeable is
the use of bins, and the fact that the user needs to decide on an optimal binning strategy, or rely on
an adaptive algorithm [
Additionally, the ECE equation deals with averages within bins rather than individual sample probabilities and their respective true labels. Due to this, outliers (such as a wildly incorrect prediction) may not have a large impact on the final calibration score. While it may be the case that this is a good thing, since a model should not be heavily penalised for a single mistake, it could be considered much more important in models for sensitive applications, such as predicting the probability of a medical patient having a certain disease.
The Maximum Calibration Error (MCE) [
max ∈{1,..,}
|() − ( )|
Theoretically, it is a miscalibration metric that checks the worst-case scenario, which is especially useful in applications where the largest error should be minimised. This makes it useful in safety-critical applications. The closer the metric is to 0, the less deviation there is between accuracy and confidence. A value closer to 1 shows that there is a large discrepancy present.
This metric sufers from the same drawback as ECE due to its utilisation of bins. To ensure that a
valid MCE value is received, it is good practice to use an adaptive binning practice, such as the one
used by Lee et al. [
There exist numerous other calibration metrics outside of the ones we mentioned in this section. A
comprehensive review of classifier probability calibration metrics is given in [
Luo et al. [
Verhaeghe et al. [
The above metrics attempt to find how well-calibrated a model is according to the definition in (1). In the following section, we detail our proposal – the concept of ‘safe’ calibration. This not only looks for perfect calibration, but also strongly penalises over-confidence, which is considered unsafe in some models.
In this section, we present background knowledge that frames the metric we propose in this paper, and what we call ‘safe’ calibration, a property that makes models trustworthy. In the following subsections, we talk about the target tracking (TT) literature which is the inspiration for safe calibration. We also propose our metric, the Entropic Calibration Diference, show how this fits into the TT field, and demonstrate how it can be adapted for general use in ML model calibration.
A fundamental goal of target tracking is to derive the state (e.g. position and velocity) of an object over time through noisy measurements. This is accomplished by using algorithms called target trackers.
Tracking systems generally consist of multiple components; however, one of the core algorithms
is track filtering. Commonly used methods include Kalman Filters and Particle Filters [
One of the most popular consistency metrics is the Normalised Estimation Error Squared (NEES), shown in (6), where ˆ represents the estimated value and 2 is the variance associated with the -th estimation error. Note that this equation assumes that the true value, , is 1D. This metric works by calculating the ratio between the actual estimation error (being the diference between the predicted and true states), and the predicted error.
=1
NEES is a good consistency metric for single-target tracking. In this paper, we propose the Entropic
Calibration Diference metric, which is inspired by NEES. One of the main aspects that the metric
borrows from NEES is the higher penalisation of over-confidence compared to under-confidence [
2 1D, is expected on average and shows perfect consistency, as it reflects a balance between the squared prediction error and the predicted uncertainty. Values greater than 1, or , show over-confidence, as the prediction error is large relative to the predicted uncertainty. This suggests that the model underestimated its uncertainty, which led it to be too confident in its predictions. A value smaller than 1 shows under-confidence as the prediction error is small relative to the predicted uncertainty. This suggests that the model overestimated its uncertainty, making it overly cautious about its predictions. The ability to find over- and under-confident predictions using only a single equation is desirable for calibration, as it gives good context in understanding the model’s predictions. Treating over- and under-confidence diferently would also give us an idea of whether a model is safely calibrated.
The aim of safe calibration is to determine whether an ML model can be deemed safe to use and is thus trustworthy. A non-calibrated model runs the risk of being over-confident in incorrect answers, or under-confident in the correct answers. We bring the TT way of thinking, where over-confidence is considered much worse than uncertainty or under-confidence, into ML. The theory behind this is that we would prefer a model be uncertain in the correct class rather than confidently choose an incorrect class. To this end, we penalise overconfidence more than under-confidence and uncertainty.
Entropic Calibration Diference can determine whether a model is well-calibrated, which can be
interpreted as whether the model is safe to use. For example, if a binary classification model was
determining whether it is safe for an aircraft to land, over-confidence in the incorrect class could
potentially be fatal. Therefore, we would prefer to have under-confidence in the correct class or
uncertainty, rather than random confident guessing [
The Entropic Calibration Diference (ECD) is applicable to both TT and ML calibration, or any other probabilistic model, and does not require any parameters other than the true label and prediction. Equation (7) shows the general definition of the ECD metric.
1 ∑︁ [︂ ∫︁
=1 log (|)(|) − log ( |) where is the total number of data points, are the measured data for data point , (|) is the algorithm’s estimate of the probability density or probability mass of true state given the measurement, and are the known true states for a test set.
The first term in the summand in (7), containing the integral, is the negative entropy of the predicted probability distribution, or expected log likelihood, for a particular data point. This is used to represent under-confidence. The second term of the summand is the log likelihood, which is used to represent over-confidence. It should be noted that if the entropy term were zero, the overall expression would be negative log-likelihood (NLL), which is a commonly used metric to measure the calibration of classifiers. In general, ECD measures the diference between expected and actual log likelihoods. In this case, we have a metric that can produce negative scores for under-confident values and positive scores for over-confident values. However, unlike other calibration metrics such as ECE, under- and over-confidence are not treated the same, as the metric follows safe calibration scoring. Some metrics require the use of optimal binning methods or other computationally expensive calculations. However, ECE, MCE, and ECD all run in linear time complexity, in terms of the number of data points.
NEES can be proven to be a special case of ECD by substituting the Gaussian distribution into both equations. Equation (8) is the Gaussian formula, (9) is the Gaussian ECD, and (10) is the Gaussian NEES. (8) (9) (10) (11) (12) (13) (14) (|) = √︀(2| |) In the above, is the true state vector, is the observation, and and are the mean vector and covariance matrix of the Gaussian uncertainty given the observation. NEES should be equal to the dimensionality in (9) for a system to be consistent and the ECD score would be zero.
To apply the ECD formula to classification problems, it is necessary to allow it to be used with discrete variables. This is a simple change, as noted in (11). ∑︁ ( = |) log ( = |) − log ( |) =1 ]︃ ]︃
In this modified equation, we predict the probability that is of class given a piece of observed data , where and are the number of data points and number of classes, respectively.
Equation (11) allows for an easy transition to an ECD formula for binary calibration. By substituting (12) and (13) into (11), we create the binary classification ECD formula in (14).
ˆ = ( = 1|) log (|) = log ˆ + (1 − ) log(1 − ˆ ) based on its calibration score, as described in the following section.
When attempting to transition the binary logic of ECD to a multiclass scenario, to preserve the original logic behind the equation, we suggest using a ‘True Class vs. Rest’ approach, which converts the problem into a binary one. set to 1 in this binary representation.
This, as outlined in (15), uses the probability of the true class, , with the true class label permanently model, while ˆ is the probability assigned to class k.
To test whether our metric works correctly, we present a theoretical analysis of its properties. The definition of calibration is in (1), while the definition of ECD is in (7). The expected population metric for ECD can be re-written as in (16). Here ˆ refers to the probability assigned by the true class by the = E [︃(︃ ∑︁ ˆ log ˆ − log
ˆ =1 )︃
By using the law of total expectation, conditioning on the predicted probability ˆ = , and the linearity of expectation, we obtain (17).
= E^ ∑︁ log − E[log |ˆ = ] ]︃ When evaluating the second term of the equation, since is the true class, we obtain (18). Under the perfect calibration assumption in (1), this results in (19).
E[log |ˆ = ] = ∑︁ ( = |ˆ = ) log
E[log |ˆ = ] = ∑︁ log [︃(︃ =1 =1 )︃ =1 (15) (16) (17) (18) (19)
The logic remains the same as the binary case, where the ECD calculates whether the model is under-confident in the correct class or over-confident in all other classes.
Due to the fact that the ECD score is theoretically unbounded, it is not as easily interpretable as ECE
which is bound between [
)︂ 1 −
When (19) is substituted back into (17), this results in the cancellation of values. Therefore, under the assumption of perfect calibration, the ECD metric will return a value of zero. Similarly, if the probability of the most likely class is 0.5, the ECD will compute to zero, indicating safely calibrated probabilities that exhibit neither under- nor over-confidence.
To measure and compare ECD values with other metrics, we utilise a hypothesis testing model. Widmann,
Lindsten, and Zachariah [
Consistency resampling, a type of bootstrapping, is used for all metrics. This method helps combat
the unfairness of judging a model’s calibration score based on a single set of predictions. Rather, the
model’s confidence scores are re-sampled with replacement to create multiple new datasets, so that
calibration metrics provide a range of scores over the multiple datasets. A threshold is set based on the
distribution of metric values. If the original score is statistically larger than the re-sampled scores, the
null hypothesis is rejected [
The Null Hypothesis, 0, is the case of perfect calibration for ECE and MCE. However, since ECD can reach a value of zero due to uncertainty, 0 for ECD is the case of safe calibration.
The alternative Hypothesis, , is the case of statistically significant miscalibration for ECE and
MCE. For ECD, refers to statistically significant unsafe calibration. As Lee et al. [
Binary tests were conducted on two datasets: the Adult Income dataset [
Table 1 shows the metrics scores and hypothesis test results for each dataset and calibration technique. Interestingly, the ECD score is considered safe in the Adult Income’s Platt Scaling results, whereas the ECE does not consider it to be within range of perfect calibration. Looking at Figure 1a, it shows that there is some under-confidence present, and a well-calibrated uncertainty. The MCE metric hypothesis accepts a higher score than the other metrics; this is because it is below the expected value from the consistency resampling technique.
The Telco Customer Churn values show the most interesting results. The ECE metric accepts the null hypothesis for all calibration techniques; however, the ECD rejects them.
This is evidence that, while a model may be well-calibrated according to ECE, it does not necessarily mean that the model does not sufer from over-confident predictions.
Figure 1b shows that although most bins are well calibrated, there is some over-confidence in the higher bins, reinforcing the reason for the ECD rejection.
Results for multiclass classification experiments are included in the Appendix A, with tables 2 and 3. s l e b a L 1 s s a l C f o n o i t c a r F (a) Adult Income Dataset (b) Telco Customer Churn Dataset
We have introduced a novel calibration metric, named the Entropic Calibration Diference (ECD) due to its consideration of the entropy of probabilities. The metric is influenced by the Normalised Estimation Error Squared (NEES) metric, which is used to determine the consistency of a state estimator within the target-tracking field of research. The ECD metric is equivalent to applying a generalised version of NEES to the ML problem domain. This new metric also brings a new perspective to the probability calibration literature, namely the concept of safe calibration, which is commonly found in the target tracking literature. We define safe calibration as a metric that prefers under-confidence to over-confidence, due to the belief that an under-confident score in the correct class is safer than an overconfident score in the incorrect class. Therefore, over-confidence is penalised more than under- confidence, rather than equal penalties, which are present in other metrics. In terms of future directions, ECD could be implemented as an objective function as well as integration into re-calibration techniques, with the hopes of making probabilities safer as well as more well-calibrated.
The author(s) have not employed any Generative AI tools.
Multiclass classification hypothesis tests were carried out on the CIFAR-10, CIFAR-100 [
Unlike the previous binary tests, one calibration method was used on the data. This is due to the
consistency resampling technique in the ECD tests requiring the full probability vector, whereas the
ECE requires the maximum probability only. To make the tests fair, a calibration method was used that
takes the whole vector into account, Temperature Scaling [
The results tables are split into two: Table 2 results use uncalibrated model probabilities, and Table 3 results are based on post-hoc temperature scaling. In both tables, the ‘Dataset’ row details which dataset the tests are conducted with. The ‘Calibration’ row either specifies ‘None’ or ‘Temperature Scaling’. The ‘Model’ row specifies which model’s probabilities are being used with the results in its respective column. Finally, each other row specifies the calibration metric and whether the null hypothesis is accepted or rejected based on the calibration score and consistency resampling simulations. Figures 2, 3, and 4 show the reliability diagrams for each model, both calibrated and uncalibrated. These reliability diagrams use a top-label prediction approach, where the most confident prediction is chosen.
Table 2 shows the results for the uncalibrated probabilities for the CIFAR-10, CIFAR-100 and ImageNet datasets. The ECE metric rejected the null hypothesis on every test, even though some of the scores were quite low. On the other hand, the MCE and ECD metrics accepted some null hypotheses, with the most surprising being the ECD ImageNet ResNet50 result, which has a high threshold value. However, Figure 4a shows that the majority of probabilities are actually under-confident, and therefore the acceptance of the null hypothesis makes sense. Similarly, VGG-19 ImageNet was rejected because of its over-confidence as seen in Figure 4b.
Table 3 shows the results for the datasets that have undergone a temperature scaling post-hoc recalibration. The results remain relatively similar, except for the fact that ECD has accepted the null hypothesis for both ImageNet scores, while ECE has continued to reject them. While this shows that there is statistically significant miscalibration in the model’s probabilities, it also does not seem to show that there is enough evidence for the ECD to deem that the probabilities are over-confident. In fact, Figure 4b shows under-confidence, explaining why the null hypothesis was accepted. This furthers the theory that safe calibration and perfect calibration should not be assumed to be one and the same. y c a r u c c A y c a r u c c A (a) ResNet32 CIFAR-10
(b) VGG-19 CIFAR-10
(a) ResNet32 CIFAR-100 (b) VGG-19 CIFAR-100