The increasing adoption of machine learning and deep learning models in critical applications raises the issue of ensuring their trustworthiness, which can be addressed by quantifying the uncertainty of their predictions. However, the black-box nature of many such models allows only to quantify uncertainty through ad hoc superstructures, which require to develop and train a model in an uncertainty-aware fashion. However, for applications where previously trained models are already in operation, it would be interesting to develop uncertainty quantification approaches acting as lightweight “plug-ins” that can be applied on top of such models without modifying and re-training them. In this contribution we present a research activity of the Pattern Recognition and Applications Lab of the University of Cagliari related to a recently proposed post hoc uncertainty quantification method, we named dropout injection, which is a variant of the well-known Monte Carlo dropout, and does not require any re-training nor any further gradient descent-based optimization; this makes it a promising, lightweight solution for integrating uncertainty quantification on any theoretically grounded solution to make dropout injection as efective as Monte Carlo dropout through a suitable rescaling of its uncertainty measure; we are also evaluating its efectiveness in the computer vision tasks of crowd counting and density estimation for intelligent video surveillance, thanks to our participation in a project funded by the European Space Agency.
1. Introduction
have acquired a prominent role in the Artificial
Intelligence (AI) field. One of the main problems arising
from their widespread adoption is the black-box nature of
many ML and DL models. A related issue we are
exploring in our ongoing work is the trustworthiness of their
predictions. One possible approach to tackle this problem
is the introduction of methods for quantifying the
uncertainty of the model’s predictions, which is fundamental
in many critical applications (e.g., healthcare and public
security) [
We are currently investigating practical applications of the post hoc injected dropout method to computer vision. In particular, we are focusing on the crowd counting and density estimation tasks related to intelligent video surveillance, which we addressed in our participation in the project IMMAGINA (IMaging MAnagement Guidelines and Informatics Network for law enforcement Agencies, 2020-2022), funded by the European Space Agency.
In this application context, uncertainty quantification can enable end users (e.g., law enforcement operators) to better interpret the outputs of DL-based tools, making them aware of unreliable estimates due to, e.g., dif
ferent operating conditions than the ones represented category, which means they cannot capture this depenin training data. We are also studying how to exploit dence because of their deterministic set of parameters. uncertainty estimates provided by injected dropout to One possible solution to make them able to estimate the automatically improve the accuracy of an already trained uncertainty of their prediction is using their Bayesian crowd counting and density estimation model online (i.e., extension, namely Bayesian Neural Networks [9, 10] during operation). tools devoted to capturing such uncertainty is essential interesting theoretical properties, obtaining the proba
Traditional neural networks are usually deterministic
models that associate a point estimate with an input
sample. However, such estimators are far from perfect, and
their mistakes are often noteworthy: hence, developing
in many applications [
Uncertainty sources, in neural networks, can be of two • Epistemic Uncertainty, caused by lack of knowledge about the correct model’s parameters; • Aleatoric Uncertainty, caused by the intrinsic randomness of the prediction.
Epistemic uncertainty can be, theoretically, reduced to zero by collecting enough data, while aleatoric uncertainty cannot [6]. For example, if one trains a neural network to predict the outcome of a dice roll no matter the number of training instances, the inherent randomness of the stochastic event makes correct outcomes with zero uncertainty impossible for any prediction model.
For aleatoric uncertainty quantification, a common choice is to define metrics upon the probability distribution ( |, )
of the outcomes given the input and the model parameters , such as the entropy [7]. However, although obtaining such measures for classification problems is straightforward, further superstructures are necessary for regression [7].
For epistemic uncertainty quantification, on the other hand, the problem becomes more complicated: indeed, for obtaining such a measure in neural networks, a quantification of the model’s parameters uncertainty is needed2.
A well-known approach for dealing with prediction models’ uncertainty is using Bayesian methods [8]; indeed, such models output a full probability distribution that is not only conditioned on the input but also dependent upon the model parameters distribution. Nevertheless, traditional neural networks fall into the non-Bayesian 2Neural networks are usually assumed to be capable of learning any function, so the lack of knowledge of the correct model parameters is usually considered a proxy for epistemic uncertainty. (BNN). The output of a BNN can be considered as a predictive distribution conditioned on the input and the training data : ( |, ) = ∫ ( |, ) ⋅ (, ) d, (1) where is the input, is the output, and are the parameters of the model. Although this formulation has bility distribution of the model’s parameters (, ) infeasible due to the many connections in modern neural networks. This problem is approached in the literature is by substituting the distribution of the parameters (, ) using an approximated distribution () [11].
Among the existing approximation strategies, Monte
Carlo Dropout [
∑ ( |, ( |, ) = ).
(2) After computing the predictive distribution, one should choose a metric for modeling the predictive uncertainty. For classification problems, a categorical distribution is usually employed: for quantifying the model’s uncertainty, typical choices are predictive entropy, predictive variance, or mutual information [13, 7]. For regression problems, instead, one possible option is to assume that the predictive distribution ( |) follows a Normal distribution: in this context, the variance and standard deviations of the model output are the typical choices for modeling uncertainty [7].
It is essential to note that in the original formulation
of dropout as a Bayesian approximation [
With this adaptation, we have obtained well-calibrated uncertainty measures while keeping relatively low neural networks, regardless of their use of dropout at dropout rates (which, when using injected dropout, is plug-in), and its lightness (no ad hoc training is needed). In this section we show how our implementation of crucial to avoid damaging the prediction quality).
dropout injection can be practically applied to quantify the uncertainty of state-of-the-art DL architectures for crowd counting and density estimation, and how uncertainty measures can also be exploited to improve their accuracy.
training time [14].
Our ongoing work started from the observation that such modification can be advantageous compared to the original version of Monte Carlo dropout because of its lfexibility (searching from a suitable dropout rate does not require multiple training), its non-invasivity (acting as a However, we noticed that this variant of Monte Carlo dropout has never been comprehensively analyzed or compared with its original version, neither by the author proposing this modification [ 14] nor by any following work using this technique. Taking these considerations as a starting point, we began our research by analyzing this technique, which we call dropout injection, and its main diference from the original version, which we call embedded dropout. ing dropout injection [15]. Indeed, it turned out that such a problem, which is also present to a less extent in embedded dropout, is amplified by injecting dropout because a network trained without dropout is less robust to random neural deactivation (since it has not been optimized for handling such a circumstance). Without a suitable rescaling, if the injected dropout rate is too high, it may result in low-quality prediction; on the other hand, if it is too low, it will result in almost no variations in the network output, which would be inefective for uncertainty estimation. We found that a possible solution consists of suitably scaling the uncertainty measure. To this aim, we exploited a technique originally proposed as a post hoc calibration method for embedded dropout, named “ -scaling” [16]. Such a scaling technique can be applied to jointly seek a suitable scaling factor and dropout rate in a unified optimization problem with no additional processing cost with respect to the original one. To this aim, instead of simply minimizing the negative log-likelihood (NLL) – that, in variational inference, is used as a proxy for the divergence between the actual and the approximated distribution [11] – we consider a proper scaling factor, which can be computed analytically [16] by using a validation set. This results in finding the desired dropout rate Φ among all possible values ,
1 ( − ̂ ()) 2 =1 2 () ⋅ ̂ 2()
2 +
1 log(() ⋅ ̂ 2()) , where is the size of the validation set, () timal scale factor, and, finally, , ̂ ()
and ̂ 2() respectively the ground truth, the model’s prediction and the uncertainty of the ℎ sample of the validation set.
is the opThe first characteristic we observed was the necessity
Crowd counting and density estimation are computer viof suitably rescaling the uncertainty measure when us- sion (CV) tasks aimed at estimating the number of people which minimizes: Φ = arg min 1 ∑ (3) are present in images or video frames and their density map.
By their nature, the technologies employed for solving these tasks evolved together with image analysis and processing, and nowadays, DL models represent the predominant choice. DL-based crowd counting models can be categorized into two main approaches [17]: • detection-based models, which rely on object
detectors for counting the number of people; • regression-based models, which first perform a multi-variate regression to estimate the crowd density map, whose sum of pixel values corresponds to the people count.
In our research, we focus on regression-based approaches. In this context, the problem can be viewed as an image-to-image translation problem [20], where input and output consist of a crowd image and a crowd density map. The ground truth density map of a training image is typically obtained by first manually annotating the head position of each pedestrian. A binary map is then constructed, containing zero values for each pixel except for the ones corresponding to the head’s locations, which are set to one. The density map is finally obtained by applying to the binary map a convolution with a kernel with unit area, e.g., a Gaussian (see the example in equals the sum of all the pixels of the resulting density map. During inference, a trained model estimates the density map of input images, and the corresponding estimate of the crowd count is obtained by summing up its pixel values. State-of-the-art DL architectures for regression-based crowd counting and density estimation do not incorporate any measure of uncertainty. To this aim, dropout injection can be implemented during inference to obtain a full probability distribution for each pixel of the estimated density map, which is modeled with a Normal distribution: its mean represents the point prediction, and we employ its variance as a measure of uncertainty.
As shown in the example of Fig. 1, this approach provides an estimated crowd density map and a corresponding uncertainty map. We obtain the total variance on the predicted crowd count by summing up all the pixel values of the uncertainty map. This allows us to compute a confidence interval on the predicted crowd count. Our ifrst experiments provided evidence that the accuracy of the confidence intervals obtained through our implementation of dropout injection relies on a suitable scaling of the underlying uncertainty measure. It is worth pointing out that a pixel-wise uncertainty measure makes it possible to compute confidence intervals not only for the whole image but also for sub-regions of it. This feature is helpful when a video surveillance operator is interested in monitoring a crowd in a specific region of a video.
In our ongoing work, we found that dropout injection could also be helpful for automatically detecting image regions with no pedestrians but with non-zero estimated density, which we call “false-positives;” in particular, from our experiments, it seems that such regions are characterized by relatively low density values and relatively high uncertainty. An example is shown in Fig. 2. Detecting false positive regions would allow us to automatically reject their contribution to the predicted crowd count and to the corresponding confidence interval, thus improving the accuracy of the underlying model. We are currently investigating a suitable policy for false positive detection and correction.
The crowd counting and density estimation tasks, that we used to investigate the injected dropout technique, were chosen as they were the subjects of previous research activities by the PRA Lab, involving also the participation in research projects. The most recent project is IMMAGINA3 (IMaging MAnagement Guidelines and Informatics Network for law enforcement Agencies), funded by the European Space Agency under the ARTES Integrated Applications Promotion Programme (Nov. 2020 – Oct. 2022), aimed at exploring applications integrating space assets and 5G networks, including law enforcement. Our task was to develop a high-TRL prototype of a realtime crowd counting and density estimation system, in the form of a Web application service accessible through a Web browser both from the control room of a law enforcement agency (LEA) and from mobile devices (tablets) of oficers in the field. Our participation in IMMAGINA helped us to better understand the needs of a relevant category of potential end users (LEA operators) of AI- and machine learning-enabled computer vision tools, whose trustworthiness is crucial for their acceptance in critical, security-related operational scenarios.
In this contribution, we summarized our ongoing research on uncertainty quantification related to injected dropout, a post hoc version of the well-known Monte Carlo dropout technique. Injected dropout allows incorporating uncertainty quantification on already-trained neural networks. We are currently investigating its application to security-related computer vision tasks, focusing on regression-based crowd counting and density estimation methods using deep learning models, thanks to our participation in the recent IMMAGINA project. In this
kind of task, an easy-to-interpret representation of the and engineering Of Security of Artificial Intelligence” uncertainty on the outputs of a machine learning model, (S.O.S. AI) included in the Spoke 3 – Attacks and Defences besides its bare point prediction, could be beneficial for of the Research and Innovation Program PE00000014, end users (e.g., LEA operators monitoring crowds in pub- “SEcurity and RIghts in the CyberSpace (SERICS)”, Nalic spaces), both to better support their decisions and to tional Recovery and Resilience Plan, Mission 4 ”Educaimprove their trust in machine learning-based systems. tion and Research” - Component 2 ”From Research to In particular, we showed how injected dropout allows to Enterprise” - Investment 1.3, funded by the EU – NextGencompute a pixel-level uncertainty map associated to the erationEU; ELSA – European Lighthouse on Secure and predicted density map, and a confidence interval on the Safe AI, grant agreement No. 101070617 funded by the corresponding predicted crowd count. EU. E. Ledda is afiliated with the Italian National PhD
Looking ahead, based on preliminary empirical evi- in Artificial Intelligence, Sapienza University of Rome. dence, we are investigating the possibility of automat- He acknowledges the cooperation with and the support ically improving the accuracy of the predicted density from the PRA Lab of the University of Cagliari. map and crowd count by exploiting pixel-based uncertainty quantification, which seems capable of highlighting “false positive” pedestrian detections in the density References map. Other interesting directions for future research include the extension of the scope of our investigation to further post hoc uncertainty quantification methods, their application to classification with a reject option, and the detection of out-of-distribution as well as adversarial examples.
This work was partially supported by the projects: “Law Enforcement agencies human factor methods and Toolkit for the Security and protection of CROWDs in mass gatherings” (LETSCROWD), Horizon 2020 programme, grant agreement No. 740466, funded by the EU; “IMaging MAnagement Guidelines and Informatics Network for law enforcement Agencies” (IMMAGINA), ARTES Integrated Applications Promotion Programme, contract No. 4000133110/20/NL/AF, funded by the ESA; “Science