For the binary prevalence quantification problem under prior probability shift, we determine the asymptotic variance of the maximum likelihood estimator. We find that it is a function of the Brier score for the regression of the class label on the features under the test data set distribution. This observation suggests that optimising the accuracy of a base classifier, as measured by the Brier score, on the training data set helps to reduce the variance of the related quantifier on the test data set. Therefore, we also point out training criteria for the base classifier that imply optimisation of both of the Brier scores on the training and the test data sets.
or should we rather strive to develop an optimal classifier,
possibly based on an comprehensive feature selection
The survey paper [
A lot of diferent approaches to quantification of prior significantly, with no guarantee of an accurate
clasclass probabilities has been proposed and analysed (see, sifier. Thus, an imperfect classifier is often all that
e.g. [
Is it worth the efort to try to train a good (accurate) hard have a perfect classifier in order to estimate the (or soft or probabilistic) classifier as the ‘base classifier’ for class distribution well. If the number of false posthe task of quantification if the class labels of individual itives balances against false negatives, then the instances are unimportant and only the aggregate prior overall count of predicted positives is nonetheless class probabilities are of interest? correct. Intuitively, the estimation task is easier
In principle, there is a clear answer to this question. for not having to deliver accurate predictions on
The accuracy of the classifier matters at least in the ex- individual cases.”
treme cases:
• If a classifier is least accurate because its
predictions and the true class labels are stochastically
independent, then quantification is not feasible.
• If a classifier is most accurate in the sense of making
perfect predictions then perfect quantification is
easy by applying Classify & Count [
But if no perfect classifier is around, can we be happy to deploy a moderately accurate classifier for quantification
The point on the mutual cancellation of false positives
and false negatives is mentioned also by a number of
other researchers like for instance [
There are also researchers that hold the contrary
position, at least as quantification under an assumption
of prior probability shift (see (1) below for the formal
definition) is concerned:
• [
• [
Prior probability shift is a special type of data set shift,
see [
The problem of estimating the test set prior class
probIn the following, we revisit the question of the usefulness abilities can also be interpreted as a problem to estimate
of accurate classifiers for quantification: the parameters of a ‘mixture model’ [
After giving an overview of related research in Sec- [
In Section 5, we describe, based on its representation have this property. [
In Section 6, we show for the test distribution of the the test set. Complementing the work of [
In Section 7, we illustrate the observations of
Sections 5 and 6 with a numerical example. Table 1 below
demonstrates variance reduction through more powerful
features both for the ML quantifier and a non-ML
quantiifer. Figure 1 below suggests that diferences in eficiency
between the ML quantifier and non-ML quantifiers may
depend both on the information content (power) of the
feature vector and on the true value of the positive class
prevalence.
• There is a training (or source) data set
(1, 1), . . . , (, ) ∈ X × { 0, 1}. It is
assumed to be an i.i.d. sample of a random vector
(, ) with values in X × { 0, 1}. The vector
(, ) is defined on a probability space (Ω , ),
the training (or source) domain. The elements
of Ω are the instances (or objects). Each instance
belongs to one of the classes 0 and 1, and its
class label is () ∈ {0, 1}. In addition, each
instance has features () ∈ X. Often, X
is the -dimensional Euclidian space such that light on the importance of both maximum likelihood
estiaccordingly is a real-valued random vector. See mation and accurate classifiers for the eficient estimation
Appendix B.1 of [
It is assumed to be an i.i.d. sample of the random Hence the likelihood function vector with values in X, under a probability mea- () = (; 1, . . . , ) sure on Ω that may be diferent to the training distribution . for the sample 1, . . . , is given by • Under the test distribution , only the features () of the instances are observed in a series of () = ∏︁(︀ (1() − 0()) + 0()︀) . (4) independent experiments resulting in the sample =1 1, . . . , . The sample can be used to infer the distribution of under . This implies for the first two derivatives of the log• The goal of quantification is to infer the dis- likelihood with respect to : torfibfuetaitounreosf 1, .u.n. d,er ge,nbearsaetdedounntdheer samapnlde log () = (5a) (on1,th1e)j,o.i.n.t, (samp,leo)f gfeenateurraetesdanudndcelrass .laIbteilss ∑︁ 1() − 0() , not possible to design a method for this inference =1 (1() − 0()) + 0() without any assumption on the relation of and . 2 log2 () = (5b) • Icbnoyntphdriiistoirpopanrpaoelbrfa,ebwaitleuitaryesssdhuiimsftte,riibtnhutathtieonsesanansrdeetthhaeatrsteharmeeclealatuesnsd-- − ∑=︁1 (︂ (1(1)(− )0−(0)()+)0() ︂) 2 ≤ 0. der and , i.e. it holds that
We assume that there is at least one ∈ {1, . . . , } such [ ∈ | = ] = [ ∈ | = ] (1) that for ∈ {0, 1} and all measurable subsets of X.
Denoting [ = 1] = and [ = 1] = , (1) implies that the distribution of the features under and respectively can be represented as [ ∈ ] =
(2a) [ ∈ | = 1] + (1 − ) [ ∈ | = 0],
(2b) [ ∈ ] = [ ∈ | = 1] + (1 − ) [ ∈ | = 0],
Under (6), ↦→ log () is strictly concave in [
Hence (see Example 4.3.1 of [
(8a)
for ⊂ X. In the following, we assume that the com- This solution is then the unique point in [
=1 1()
applies or
and
[
In summary, under the natural assumption (6), the
likelihood function of (4) has an absolute maximum
in [
function : X In the setting of Section 3, let ̃︀ be any unbiased estimator of the positive class prevalence under the test distribution , i.e. ̃︀ = (1, . . . , ) for some
→ R such that [̃︀] = [ = 1] = , in the sense that √ (ˆ −
Actually, since ˆ from Section 4 is the ML estimator of , (2b) presents not only a lower bound for the
variances of unbiased estimators of but also the large sample variance of ˆ ︃(
) converges in dis︂[ (︁ 1()− 0() )︁ 2]︂ − 1)︃
()() tribution toward
0, ︂[ (︁ 1()− 0() )︁ 2]︂ − 1
()() the normal distribution with mean 0 and variance
, the asymptotic variance of ˆ
conditional distributions of the feature vector , such
that the density () of under is given by (3).
tion of the feature vector under the test distribution ,
then the variance of ̃︀ is bounded from below by the
inverse of the product of the Fisher information of the test
distribution with respect to and the size of the sample
(Cramér-Rao bound, see, e.g., Corollary 7.3.10 of [
︁) 2]︂ ︂[ (︁ 1()− 0() )︁ 2]︂
()() that there are positive densities 0 and 1 of the class- in terms of the variance of . ture probability measure of (2b). Assume additionally where denotes the expected value under the mix- result on the representation of the Fisher information mentioned above in the context of the Cramér-Rao bound If (1, . . . , ) is an i.i.d. sample from the distribu- a density as specified by (3) then the Fisher information .
(9) ˆ: ,
): bound of (9) may not hold for ˆ because it need not be
In summary, if the densities of the class-conditional
feature distributions are known, the ML estimator of
has asymptotically the smallest variance of all unbiased
estimators of on i.i.d. samples from the test distribution
of the features. This is demonstrated in the two upper
panels of Table 4 of [
Denote by () the posterior positive class probability given = under . Assume that the feature vector under has a density that is given by (3). Then it holds that () = 1() ()() .
(10) This representation immediately implies the following Proposition 6.1. If the feature vector under has ︂[ (︁ log ()
︁) 2]︂ with respect to can be represented as follows: ()
of the distribution of under [︃ ︂( log ()
︂) 2]︃ () = var[ ()] 2 (1 − )2 .
From Proposition 6.1 we obtain the following
representation of the asymptotic variance of the ML estimator
︂[ (︁ 1() − 0() )︁ 2]︂ − 1
In (12), the optimal Brier Score () is also called
refinement loss , while var[ ] and var[ ()] are
known as uncertainty and resolution respectively [
By (11), the asymptotic variance of the ML estimator ˆ is reduced if a feature vector with greater variance of () is found (or if the Brier Score with respect to decreases). Consider, for instance, a feature vector ′ that is a function of the feature vector , i.e. it holds that ′ = () for some function : X → X′. Since potentially maps diferent values ∈ X onto the same value ′ ∈ X′ the amount of information carried by ′ is reduced compared to the amount of information carried by . Therefore, the approximation of the class label by regression on ′ is less close than the approximation of by regression on .
From this observation, it follows that (′) ≥ (). This in turn implies by (12) and (11) for the asymptotic variances of the ML estimators ˆ(′) and ˆ() that var[ˆ(′)] ≤ var[ˆ()].
(13) Observe that ′ = () also implies (′) ≥ (), i.e. also under the training distribution , the posterior positive class probability () based on is a better predictor of than (′) which is based on ′. By the assumption underlying this paper, (′) and () are observable while (′) and () are not, because the class label is not observed in the test data set. (′) ≥
But does (′) ≥
() always imply () and therefore also (13), thus gen (). eralising the implication “′ = () ⇒ (13)”?
This paper has no fully general answer to this
question. Instead we can only point to alternative conditions
on ′ and that imply both (′) ≥
and (′) ≥ (), but are weaker than ′ =
Brier curves: Recall the notion of Brier curve from [
ROC analysis: Recall the notion of Receiver Operating (′) ≥ () hold.
Characteristic (ROC) as defined, for instance, in [
If the ROC for the density ratio associated with
dominates the ROC for the density ratio associated
with ′, then by Remark 5.4 and Proposition 4.1
of [
By Eq. (2) of [
In this section, we numerically compare the variance of
the Sample Mean Matching (SMM) estimator ˆ, of
class prevalences [
(14a) for conditional means 0 < 1 and some >
0. For the sake of simplicity, we choose 0 = 0, 1 > 0, = 1.
(14b) Greater values of 1 imply less overlap of the classconditional feature distributions, corresponding to more powerful (or accurate) models. Or in other words, for greater values of 1, the feature variable carries more information on the class label .
As stated in Section 3, we assume we are dealing with
an infinitely large training sample and a test sample of
size . By Section 6, then for large the variance of
ˆ, is approximately
1
[︃
︂( 1() − 0() )︂ 2]︃− 1
refer to Theorem 3 of [
( 1 − 0)2 1 (︂
1 21 + (1 − ) .
︂)
In the model specified by (14a) and (14b), the
classification power (or accuracy) is driven by the diference of
the conditional means, i.e. by the mean conditional on
the positive class 1. If we measure the power by the
Area under the Curve (AUC, see for instance Section 6.1
of [
(18) ︂( 1 ︂) √2
, and √︁ √︁ =
var[ˆ, ] =
var[ˆ,]. with Φ denoting the standard normal distribution function. 8. Conclusions
From (18) und (17), it is clear that for fixed test sample size the variance of ˆ, decreases when the power In this paper, we have revisited the binary quantification of the model increases. This is less obvious from (15) for problem, i.e. the problem of estimating a binary prior the asymptotic variance of ˆ, but if follows from (11) class distribution on the test data set when training and in that case. test distributions are diferent.
Table 1 above illustrates these observations. In the table, we use the notation • Specifically, under the assumption of prior probability shift we have looked at the asymptotic variance of the maximum likelihood estimator (MLE). • We have found that this asymptotic variance is closely related to the Brier score for the regression of the class label variable against the features vector under the test set distribution. In particular, the asymptotic variance can be reduced by selection of a more powerful feature vector. • At the end of Section 6, we have pointed out sufifcient conditions and associated training criteria (Brier curves and ROC analysis) for minimising both the Brier score on the training data set and the Brier score on the test data set. • These findings suggest methods to reduce the variance of the ML estimator of the prior class probabilities (or prevalences) on the test data set. Due to the statistical consistency of ML estimators, by reducing the variance of the estimator also its mean squared error is minimised. • The large sample variance of the MLE associated with its asymptotic variance is identical to the Cramér-Rao lower bound for the variances of unbiased estimators of the prior positive class probability. Therefore, it seems likely that also other estimators benefit from improved performance of the underlying classifiers or feature vectors.
Indeed, results of a simulation study in [
However, these findings must be qualified in so far
as the observations made in this paper apply only to the
case where both training data set and test data set are
large. This is a severe restriction indeed as [
Further research on developing eficient quantifiers
for small or moderate training and test data set sizes
therefore is highly desirable. A promising step in this
direction has already been done by [
The author thanks four reviewers for their useful and supportive comments. 1–8