The black-box behavior of Convolutional Neural Networks is one of the biggest obstacles to the development of a standardized validation process. Methods for analyzing and validating neural networks currently rely on approaches and metrics provided by the scientific community without considering functional safety requirements. However, automotive norms, such as ISO26262 and ISO/PAS21448, do require a comprehensive knowledge of the system and of the working environment in which the network will be deployed. In order to gain such a knowledge and mitigate the natural uncertainty of probabilistic models, we focused on investigating the influence of filter weights on the classification confidence in Single Point Of Failure fashion. We laid the theoretical foundation of a method called the Neurons' Criticality Analysis. This method, as described in this article, helps evaluate the criticality of the tested network and choose related plausibility mechanism.
The need to understand and rely on the inference processes
of Convolution Neural Networks (CNNs) grows in
importance since probabilistic models are being integrated in
autonomous vehicles
The transparency, evaluation criteria and types of
explanations of the achieved results face low interpretability
Furthermore, as recently shown by various adversary
attack examples
In this work we introduced a new metric called criticality, which can be either assigned to neurons (in CNN’s terminology, those are referred to as ”filters’ weight”) or to layers. We verified the importance and plausibility of the proposed criticality as a possible safety metric by conducting two experiments. We designed and implemented a method called Neurons’ Criticality Analysis (NCA) and tested it on four image classification models.
The results of our experiment are extensively discussed at the end of this work, where we also summarized the pros and cons of the presented metric and methodology and highlighted the use-cases we intend to investigate in the future. 2
As mentioned in
As a first step, we looked at the straightforward VGG16
architecture
Since building a model in ResNet fashion comes at
computational cost, a family of mobileNets emerged. The
standard convolution operation which was used up to this point
was, in case of MobileNet v1
A research done by Google discovered a new method of
scaling the CNN. The goal of this optimization search was
to find scaling coefficients (network width, depth and
resolution) with respect to the accuracy and amount of operations.
EfficientNet
The need to understand how the decision process is made
lead us to several papers
The aleatoric and epistemic uncertainty
Many papers additionally address the problem of
datadriven ML algorithms and how to incorporate safety
mechanism in order to monitor the prediction
uncertainty
Our approach is driven by ISO/PAS21448
SOTIF (
The original idea of adversary attack is to introduce a small perturbation to an input image so that the original class doesn’t have the highest confidence and the adversary noise stays unrecognized to the human perception system.
One of the first attacks used the Fast Sign Gradient
Descent (FSGD) method
The defense mechanism started to be deeply investigated
in the work of Lie et al.
In this section we defined the metric and methodology related to manipulating and analyzing the CNN’s decision process. We took the inspiration for the Neurons’ Criticality Analysis (NCA) method from the Software Criticality Analysis (SWCA). The SWCA is a method which divides modules of any action chain between critical (the SPOF of which could have fatal consequences) and non-critical. In case of an automotive SW component, the SWCA is carried out by analyzing the signal flow from the actuator to the sensor, while heuristically justifying the signal’s non-criticality. In order to investigate if the decision of any CNN can be significantly influenced by the SPOF of the filter connections, the idea of an approach similar to SWCA was explored. From now on we will refer to filter connection as a neuron, since the principle can be generally applied to FC, CONV and Depth-wise CONV layer, Firstly, we denote the analyzed convolutional neural network as N , which consists of a set of layers L and contains weights W and biases b. Secondly, we introduce the criticality metric according to Equation 1 and the evaluation algorithm 1 which calculates the criticality for a given input image xi, belonging to class i, drawn from a test set X .
8>y^i fcr = <>> 1 1y^mj ; >2; > >:0; y^mi; if fm(xi) : (y^i
y^mi) if fm(xi) : y^mi if fm(xi) : y^mi otherwise; y^mj and y^mj < 0:5 y^mj and y^mj 0:5 (1) where fcr returns a criticality with domain [0; 2] for a given CNN which is masked, fm(xi) N . The masking of a CNN is carried out by setting neurons’ weights to zero. In case of convolution, all the values of a filter are set to zero. A different kind of error modeling would lead to extensive permutation and was therefore not further investigated.
The term y^mi denotes the masked network’s prediction confidence of ground-truth class i, whereas the predicted confidence value y^mj belongs to another class j. In the first case of fcr, the difference between the non-masked predicted confidence y^i and the masked one y^mi is taken as metric, by considering the parametrizable ”criticality” with domain [0; 1].
In the second case of fcr, the network missed the groundtruth class and predicted a different one. Since this misclassification can have severe consequences, we define the criticality measure as the proportion of 1 and difference between maximum likelihood and predicted confidence y^mj . The denominator will always result in a number greater than 1, consequently ensuring the distinguishability of neurons which have class-changing ability. Experiments in the early phase showed that criticality can reach multi-digit number and therefore we decided in the third case of fcr to clip its maximum to 2, so that the results remain tractable. For all other cases we set the criticality to 0. This covers the cases where the network predicted the right class with negligible deterioration of the confidence (< ).
It can occur that by masking a neuron the decision likelihood of the correct class will increase, which will result in a negative criticality. In this case, we refer to this neuron as anti-critical to the related class i and calculate its criticality according to Equation 2. However, it should be noted that the anti-criticality will be computed only in case = 0 and it doesn’t exclude the criticality of the neuron for a different class j.
fanti cr = y^i y^mi; if fm(xi) : (y^i y^mi) < 0 (2) 3.2
We define the task of NCA as the analysis of the neurons’ contribution to the classification hypothesis which can be seen as equivalent to the Single Point Of Failure analysis. If all neurons are active, the resulting hypothesis is strong hstr, whereas in case a certain amount of neurons have been excluded from the decision, the hypothesis is considered weakened hweak. The neuron’s criticality observation of the weakened hypothesis has to be done for every image and class within a test set. The algorithm is described in Algorithm 1.
Output: Neural criticality statistics for Xi Data: Let X be a testing set, i a tested class, N the analyzed CNN, k the number of filters in a layer L and fcr is the criticality function for image xi 2 X do y^i = calculate conf (N; xi) clsi = predict(N; xi) for every L in N do for every k in layer L do mask neuron(k) y^mi = calculate conf (N; xi) clsmi = predict(N; xi) criticality = fcr(y^i; y^mi; clsi; clsmi) 4
The motivation behind testing different network
architectures was to see the influence of models’
chronological improvements on the decision stability, such as
residual connections, depthwise convolution and scaling. We
therefore evaluated VGG16, Resnet50V2, MobileNetV2
and EfficientNetB0, all pre-trained Keras models on
ImageNet
As a first step, we gathered statistics of every neuron as
described in Algorithm 1 with = 0:0. As a second step,
we normalized the list of hypotheses over the number of
layers’ neurons and highlighted in red the layers for which
masking at least one neuron caused a drop of confidence by
0:5 and more or lead to misclassification of the predicted
class. It is noticeable that in Figures 3 and 4, especially the
first projection layers have very high criticality. This
confirms the sparsity theory of projection layers
VGG16 (Figure 1) and ResNet50v2 (Figure 2) have on the other hand an average criticality spread over all layers.
The experiments’ results shown in this work are only related to the ”mountain bike” class. The results for the fairly simple ”traffic sign” class backed up the intuition about simple features being predominantly filtered in early layers, since their criticality raised significantly. We advise the reader to visit our GitHub, where additional figures and stored statistics for both classes can be found. To further evaluate the beneficial effects of neural criticality, we conducted a stability experiment with n most critical neurons (derived from NCA results) on original and adversary datasets. We gradually masked the n most critical neurons and calculated the mean and standard deviation of the model’s accuracy on the aforementioned test set. The intuition behind this test was that the mean accuracy increases with respect to the criticality of lower neurons, and hence proves our analysis’ reliability. In our text we refer to this approach as Network Stability Analysis.
As can be seen in Figure 5, gradually masking the 20 most critical neurons has a major influence on the accuracy only in case of MobileNetv2 and EfficientNetB0. VGG16 and ResNet50V2, on the other hand, show a high accuracy stability. Figure 6 shows the raising tendency of MobileNets’ accuracy with respect to lower neurons criticality, reaching a mean accuracy of 0:8 approximately at the 50th most critical neuron.
Results of the accuracy on adversary dataset didn’t confirm the hypothesis that critical neurons are the only neurons allowing malicious adversary attacks. As can be seen in Figure 9, masking the critical neuron generally doesn’t improve the accuracy. On the other hand, several neurons lifted the ground-truth class accuracy. The awareness of such neurons could lead to on-the-fly diagnoses, where masking a combination of specific neurons (e.g. only for xth frame, which would be excluded from the classification or detection task) would uncover irregularities in inference process, e.g. adversary attack. It has to be mentioned that only critical neurons from projection layers (MobileNetV2 and EfficientNetB0) have such an ability, but they have to be chosen with respect to the mean and standard deviation of the calculated accuracy. Other models sensitivity to adversary noise are plotted in Figure 8.
5
At the beginning of this work in Chapter 2 we pointed out the current functional safety issues and open research area related to convolutional neural networks. In Section 1 we described our motivation related to autonomous driving and (b) ResNet50V2
(a) VGG16 y c ra0.6 u c c A 0.4 (b) ResNet50V2
(a) VGG16 (c) MobileNetv2 (d) EfficientNetB0 Figure 8: Results of accuracy stability related to the 20 most critical neurons for all models, evaluated on adversary dataset. It is obvious that different results of generating adversary attacks was achieved since initial models accuracy differs. The VGG16 model shows minimal accuracy fluctuation, whereas more modern models contains neurons with higher sensitivity to adversary noise.
0.8 missing validation process. Further, as outlined in Section 3, we introduced a new metric and auxiliary analysis method which we implemented and verified on four classification CNNs in Chapter 4.
We dedicated a great part of our work to introducing and testing an innovative method, the Neurons’ Criticality Analysis. The outcome of this analysis was a comprehensive report diagram depicting the criticality of each layer and each neuron of evaluated model. The domain of criticality is [ 1; 2]. We discussed that masking neurons with negative criticality can also have a positive influence on the model’s decision confidence. We called this behavior ”anti-critical”.
The inter-class anti-critical neurons could hypothetically be removed from the decision process. This idea led us to the conclusion that the correlation between the neurons removed during the pruning process and the anti-critical neurons discovered via NCA should be further investigated.
We claimed that using spatial aggregation via projection layers may on the one hand improve the high dimensional feature representation(Szegedy et al. 2016), but on the other hand creates very critical dense connections, especially in the shallow layers, as we pointed out in Section 4.1. From functional safety point of view this isn’t necessarily negative, since the plausibility function could be applied to only a concentrated area of neurons. In addition, some critical neurons showed the ability to increase mean accuracy on adversary dataset, which could be used in order to discover adversary attacks and irregularities during inference. We hypothesize that an equilibrium between the position of the first projection layer, number of critical neurons and models’ accuracy should be further investigated.
As aforementioned, the purpose of NCA is to identify all critical neurons. With further measures, the mean and standard deviation of the criticality should be decreased and the flawless calculation of the neuron should be ensured. Concretely this can be achieved by several approaches, such as: • fine-tuning of the model with deterministic dropout and
loss which will incorporate the layers criticality • plausibility check of the critical neurons or layers or re
dundant computational branch results • storage of the neurons’ weights and biases in two places
in RAM and comparing them • introduction of deconvolutional layers in order to compute and evaluate the original inputs over critical connections
Our method can also be used for Out-of-Distribution detection, where instead of randomly sampling sub-networks predictions, as it is done by MC dropout, deterministic dropout would be based on several highly critical neurons for every class. Such an approach would decrease the computational demand and arguably increase the reliability and transparency of such a network. In order to encourage additional experiments and deeper explorations, we published our code and supplement results on GitLab 1.
6
The work has been supported by the grant of the University of West Bohemia, project No. SGS-2019-027.
wo gui branch