Neural Criticality: Validation of Convolutional Neural Networks
Václav Diviš1
Marek Hrúz
1
Supported by ARRK Engineering and the University of West Bohemia.
Faculty of Applied Sciences, University of West Bohemia
Pilsen, CZE, 306 14
divisvaclav@gmail.com
Abstract this motivated us to define a methodology and understand-
able metrics according to which State-Of-The-Art (SOTA)
The black-box behavior of Convolutional Neural Networks is
CNNs can be analyzed and the achieved results validated.
one of the biggest obstacles to the development of a standard-
ized validation process. Methods for analyzing and validat- In this work we introduced a new metric called criticality,
ing neural networks currently rely on approaches and met- which can be either assigned to neurons (in CNN’s terminol-
rics provided by the scientific community without consid- ogy, those are referred to as ”filters’ weight”) or to layers.
ering functional safety requirements. However, automotive We verified the importance and plausibility of the proposed
norms, such as ISO26262 and ISO/PAS21448, do require a criticality as a possible safety metric by conducting two ex-
comprehensive knowledge of the system and of the working periments. We designed and implemented a method called
environment in which the network will be deployed. In order Neurons’ Criticality Analysis (NCA) and tested it on four
to gain such a knowledge and mitigate the natural uncertainty image classification models.
of probabilistic models, we focused on investigating the in- The results of our experiment are extensively discussed
fluence of filter weights on the classification confidence in
at the end of this work, where we also summarized the pros
Single Point Of Failure fashion. We laid the theoretical foun-
dation of a method called the Neurons’ Criticality Analysis. and cons of the presented metric and methodology and high-
This method, as described in this article, helps evaluate the lighted the use-cases we intend to investigate in the future.
criticality of the tested network and choose related plausibil-
ity mechanism. 2 Previous work
As mentioned in (Belle and Papantonis 2020), data-driven
1 Introduction and motivation techniques struggle to be robust against domain shift, data
corruption and input space perturbation. The robustness can
The need to understand and rely on the inference processes be influenced by three aspects: architecture, training dataset
of Convolution Neural Networks (CNNs) grows in impor- and optimization algorithm. It is not among the goals of this
tance since probabilistic models are being integrated in au- paper to give a comprehensive overview of all three top-
tonomous vehicles (Tesla 2019),(Mobileye 2020), where the ics, but to highlight the most influential milestones which
SW development follows functional safety standards and on inspired our experiment.
which lives may depend.
The transparency, evaluation criteria and types of explana- 2.1 Classification via CNN
tions of the achieved results face low interpretability (Belle As a first step, we looked at the straightforward VGG16 ar-
and Papantonis 2020) due to the increasing complexity of chitecture (Simonyan and Zisserman 2014), where Batch-
the models used. Norm layer, 3 × 3 CONV with stride and padding equal to
Furthermore, as recently shown by various adversary at- 1 and 2 × 2 MAX POOL with stride 2 are several times
tack examples (Liu et al. 2016),(Brown et al. 2017),(Eykholt repeated. As a second step, we reviewed the ResNet (He
et al. 2018), even a small perturbation in the input im- et al. 2016a) architecture, which differentiates from the pre-
age can cause a major change to the algorithm’s decision. vious models mainly by having residual blocks (He et al.
In addition, the current leading norms (ISO26262:2018, 2015) and using 1x1 convolutional operation. We tested
ISO/PAS21448:2019) do not define validation process nor ResNet50V2 (He et al. 2016b), which is the successor of the
metrics related to the probabilistic model (BMW 2019). Fi- original ResNet with enhanced residual connections for a
nally, to our knowledge. there isn’t any method similar to smoother information propagation. The aim of residual con-
the Software Criticality Analysis (SWCA) or MC/DC mod- nections is to create additional information flow and to learn
ule test in the CNN field (Salay and Czarnecki 2018) which additive residual function. As explained in (Lin, Chen, and
can analyze potential Single Point Of Failure (SPOF). All of Yan 2014) and further investigated in (Szegedy et al. 2014),
Copyright © 2021, for this paper by its authors. Use permitted un- the concept known as ”Network In Network” (NIN) allows
der Creative Commons License Attribution 4.0 International (CC reducing the filter’s dimensionality and increasing the mod-
BY 4.0). els’ non-linearity. With the combination of computationally
more expensive 3 × 3 and 5 × 5 convolution and parallel Our approach is driven by ISO/PAS21448 SO-
branches the extraction of features from different scales can TIF (ISO2019 2019), ISO2626 (ISO 2011) norms, which
be achieved simultaneously. This kind of blocks are often lack validation-unambiguity, and by the conclusion that
called ”projection layer” (Li et al. 2018). only 40% of the current automotive verification/validation
Since building a model in ResNet fashion comes at com- methods can be transferred to ML application (Salay and
putational cost, a family of mobileNets emerged. The stan- Czarnecki 2018). We mostly considered the SOTIF norm,
dard convolution operation which was used up to this point which is an extension of the well-known ISO2626 norm
was, in case of MobileNet v1 (Howard et al. 2017), replaced and provides a guidance (recommended activities) on
by depthwise separable convolution. It was shown that stan- applicable design, verification and validation measures in
dard convolution can be split into depthwise and pointwise order for the product to be norm-compliant. The goal of the
convolutions, which decreases the number of operations by recommended activities is to maximize the area of known
a square of the spatial dimension of the used filter kernel. safe scenarios and minimize the unknown or unsafe areas
Further improvements were carried out by Sandle et al. in by applying technical measures.
MobileNet v2 (Sandler et al. 2018), where residual con-
nections between the cells and the expand/projection layers 2.3 Adversary attacks
were added. The original idea of adversary attack is to introduce a small
A research done by Google discovered a new method of perturbation to an input image so that the original class
scaling the CNN. The goal of this optimization search was doesn’t have the highest confidence and the adversary noise
to find scaling coefficients (network width, depth and resolu- stays unrecognized to the human perception system.
tion) with respect to the accuracy and amount of operations. One of the first attacks used the Fast Sign Gradient De-
EfficientNet (Tan and Le 2019) was designed to demonstrate scent (FSGD) method (Goodfellow, Shlens, and Szegedy
the effectiveness of this scaling method and achieved SOTA 2015), which calculates the gradient of a model’s loss func-
accuracy on the ImageNet dataset in 2019. tion with respect to the input image and ground-truth label
The need to understand how the decision process is made and either adds or subtracts a small portion of it, depend-
lead us to several papers (Simonyan, Vedaldi, and Zisserman ing whether the gradient was positive or negative. Addi-
2013),(Zeiler and Fergus 2014),(Bach et al. 2015),(Shriku- tional papers proposed a general and large perturbation at-
mar, Greenside, and Kundaje 2017),(Sundararajan, Taly, and tack algorithm of physical objects considering spatial con-
Yan 2017), where different visualization methods are de- straints and physical limits on imperceptibility, (Brown et al.
scribed. A comprehensive overview of the methods and their 2017), (Eykholt et al. 2018). Ian Goodfellow summarized
goals can be found in (Rafegas et al. 2019), where Rafegas additional weaknesses of the classification task in (Goodfel-
et al. also presented a novel method of quantifying the neu- low 2018).
rons’ selectivity to color and class. The defense mechanism started to be deeply investigated
in the work of Lie et al. (Liu et al. 2016), which shows a
2.2 Safety related issues comprehensive experiment of different ResNets Architec-
tures trying to resist non-target adversarial images and states
The aleatoric and epistemic uncertainty (Kendall and Gal that ResNet-152 has a 0% resistivity. The explanation to that
2017) of probabilistic models are currently under in-depth phenomenon is still in the open research area (Brown et al.
study. The source of the aleatoric uncertainty is brought by 2017), but one of the latest works (Song et al. 2018) shows
the randomness contained within the training-set, whereas promising results and mentions defending methods, showing
the epistemic uncertainty is caused by the lack of the that the robustness against different attacks can improve.
model’s knowledge. Bayesian machine learning (Neal 2012)
approach allows propagating the intermediate covariances
to the final layer and quantifying the hypothesis uncer- 3 Analysis methodology
tainty (Graves 2011), (Shridhar, Laumann, and Liwicki In this section we defined the metric and methodology re-
2019). Such an approach requires time-consuming training, lated to manipulating and analyzing the CNN’s decision pro-
and, for the moment, models do not achieve the expected ac- cess. We took the inspiration for the Neurons’ Criticality
curacy. An additional one-shot approach uses Monte Carlo Analysis (NCA) method from the Software Criticality Anal-
dropout during inference in order to sample a subset of net- ysis (SWCA). The SWCA is a method which divides mod-
works, build statistics and calculate the thereof resulting un- ules of any action chain between critical (the SPOF of which
certainty (Gal and Ghahramani 2016). This improves the de- could have fatal consequences) and non-critical. In case of
mands on the inference time to reasonable limits and can an automotive SW component, the SWCA is carried out by
therefore be applicable in automotive. analyzing the signal flow from the actuator to the sensor,
Many papers additionally address the problem of data- while heuristically justifying the signal’s non-criticality. In
driven ML algorithms and how to incorporate safety order to investigate if the decision of any CNN can be signif-
mechanism in order to monitor the prediction uncer- icantly influenced by the SPOF of the filter connections, the
tainty (Cheng 2020), (Lakshminarayanan, Pritzel, and Blun- idea of an approach similar to SWCA was explored. From
dell 2017), (DeVries and Taylor 2018). Several uncer- now on we will refer to filter connection as a neuron, since
tainty estimation methods were lately evaluated by Henne the principle can be generally applied to FC, CONV and
et al. (Henne et al. 2020) with respect to functional safety. Depth-wise CONV layer,
3.1 Criticality metric excluded from the decision, the hypothesis is considered
Firstly, we denote the analyzed convolutional neural net- weakened hweak . The neuron’s criticality observation of the
work as N , which consists of a set of layers L and con- weakened hypothesis has to be done for every image and
tains weights W and biases b. Secondly, we introduce the class within a test set. The algorithm is described in Algo-
criticality metric according to Equation 1 and the evaluation rithm 1.
algorithm 1 which calculates the criticality for a given input
image xi , belonging to class i, drawn from a test set X . Algorithm 1: NCA algorithm
Input: Criticality threshold τ
ŷi − ŷmi , if fm (xi ) : (ŷi − ŷmi ) ≥ τ Output: Neural criticality statistics for Xi
1 ,
if fm (xi ) : ŷmi ≤ ŷmj and ŷmj < 0.5 Data: Let X be a testing set, i a tested class, N the
fcr = 1−ŷmj analyzed CNN, k the number of filters in a
2, if fm (xi ) : ŷmi ≤ ŷmj and ŷmj ≥ 0.5 layer L and fcr is the criticality function
0, otherwise,
for image xi ∈ X do
(1) ŷi = calculate conf (N, xi )
where fcr returns a criticality with domain [0, 2] for a given clsi = predict(N, xi )
CNN which is masked, fm (xi ) ⊂ N . The masking of a for every L in N do
CNN is carried out by setting neurons’ weights to zero. In for every k in layer L do
case of convolution, all the values of a filter are set to zero. mask neuron(k)
A different kind of error modeling would lead to extensive ŷmi = calculate conf (N, xi )
permutation and was therefore not further investigated. clsmi = predict(N, xi )
The term ŷmi denotes the masked network’s prediction criticality = fcr (ŷi , ŷmi , clsi , clsmi )
confidence of ground-truth class i, whereas the predicted
confidence value ŷmj belongs to another class j. In the first
case of fcr , the difference between the non-masked pre-
dicted confidence ŷi and the masked one ŷmi is taken as met-
ric, by considering the parametrizable ”criticality” τ with
domain [0, 1].
4 Experiments
In the second case of fcr , the network missed the ground- The motivation behind testing different network architec-
truth class and predicted a different one. Since this mis- tures was to see the influence of models’ chronologi-
classification can have severe consequences, we define the cal improvements on the decision stability, such as resid-
criticality measure as the proportion of 1 and difference be- ual connections, depthwise convolution and scaling. We
tween maximum likelihood and predicted confidence ŷmj . therefore evaluated VGG16, Resnet50V2, MobileNetV2
The denominator will always result in a number greater than and EfficientNetB0, all pre-trained Keras models on Ima-
1, consequently ensuring the distinguishability of neurons geNet (Deng et al. 2009). We chose two classes, ”street sign”
which have class-changing ability. Experiments in the early and ”mountain bike”, in order to evaluate the criticality. For
phase showed that criticality can reach multi-digit number each class, 150 samples were taken. All samples had ground-
and therefore we decided in the third case of fcr to clip its truth confidence higher than 0.8 so that we ensured that ker-
maximum to 2, so that the results remain tractable. For all nels’ responses would be highly excitated. Adversary sam-
other cases we set the criticality to 0. This covers the cases ples were generated by non-target FSGD method until either
where the network predicted the right class with negligible achieving a confidence greater than 0.5 or ending after 20 it-
deterioration of the confidence (< τ ). erations. For all tests we set the criticality threshold τ to 0.0,
It can occur that by masking a neuron the decision likeli- which allows, as described in Section 3.1, the algorithm to
hood of the correct class will increase, which will result in measure and visualize the criticality of all neurons and dis-
a negative criticality. In this case, we refer to this neuron as tinguish between critical and anti-critical ones. In practice,
anti-critical to the related class i and calculate its criticality the threshold should be justifiable via hazard and risk assess-
according to Equation 2. However, it should be noted that ment and will be presumably higher than 0.0.
the anti-criticality will be computed only in case τ = 0 and
it doesn’t exclude the criticality of the neuron for a different 4.1 Neural criticality
class j.
As a first step, we gathered statistics of every neuron as de-
scribed in Algorithm 1 with τ = 0.0. As a second step,
fanti cr = ŷi − ŷmi , if fm (xi ) : (ŷi − ŷmi ) < 0 (2) we normalized the list of hypotheses over the number of
layers’ neurons and highlighted in red the layers for which
3.2 Analysis algorithm masking at least one neuron caused a drop of confidence by
We define the task of NCA as the analysis of the neurons’ 0.5 and more or lead to misclassification of the predicted
contribution to the classification hypothesis which can be class. It is noticeable that in Figures 3 and 4, especially the
seen as equivalent to the Single Point Of Failure analysis. first projection layers have very high criticality. This con-
If all neurons are active, the resulting hypothesis is strong firms the sparsity theory of projection layers (Szegedy et al.
hstr , whereas in case a certain amount of neurons have been 2014, 2016), which states that projection layers are helpful
in terms of higher space feature extraction, whereas our ex- 1.2
periment shows that they cause an increase of criticality. 1.0
VGG16 (Figure 1) and ResNet50v2 (Figure 2) have on
Mean of normalized criticality
0.8
the other hand an average criticality spread over all layers.
0.6
0.4
0.2
Mean of normalized criticality
0.05
0.0
0.00
0.05
Figure 4: This figure shows that EfficientNetB0 contains
many neurons which criticality is exceeds 1.0. Because the
EfficientNetB0 architecture contains, in its early stage, many
ck1 1
ck2 2
ck2 1
ck3 2
ck3 1
ck3 2
ck4 3
ck4 1
ck4 2
ck5 3
ck5 1
2
blo conv
blo conv
blo conv
blo conv
blo conv
blo conv
blo conv
blo conv
blo conv
blo conv
blo conv
nv
_co
projection layers with only few neurons, we removed the
_
_
_
_
_
_
_
_
_
_
_
c k1
blo
overlapping x-axis labels.
Figure 1: Results of NCA on VGG16 showed that straight
architecture, without any projection and residual layers, has 4.2 Network stability
the lowest criticality.
To further evaluate the beneficial effects of neural critical-
ity, we conducted a stability experiment with n most critical
neurons (derived from NCA results) on original and adver-
0.10 sary datasets. We gradually masked the n most critical neu-
0.08 rons and calculated the mean and standard deviation of the
0.06 model’s accuracy on the aforementioned test set. The intu-
Mean of normalized criticality
0.04 ition behind this test was that the mean accuracy increases
0.02 with respect to the criticality of lower neurons, and hence
0.00 proves our analysis’ reliability. In our text we refer to this
0.02
approach as Network Stability Analysis.
0.04
As can be seen in Figure 5, gradually masking the 20 most
critical neurons has a major influence on the accuracy only
in case of MobileNetv2 and EfficientNetB0. VGG16 and
Figure 2: The neural criticality of ResNet50v2 peaked in the ResNet50V2, on the other hand, show a high accuracy sta-
early projection layers, but remained overall very small. bility. Figure 6 shows the raising tendency of MobileNets’
accuracy with respect to lower neurons criticality, reaching
a mean accuracy of 0.8 approximately at the 50th most crit-
ical neuron.
1.0 Results of the accuracy on adversary dataset didn’t con-
firm the hypothesis that critical neurons are the only neurons
0.8
allowing malicious adversary attacks. As can be seen in Fig-
Mean of normalized criticality
0.6 ure 9, masking the critical neuron generally doesn’t improve
0.4
the accuracy. On the other hand, several neurons lifted the
ground-truth class accuracy. The awareness of such neurons
0.2
could lead to on-the-fly diagnoses, where masking a com-
0.0 bination of specific neurons (e.g. only for xth frame, which
would be excluded from the classification or detection task)
would uncover irregularities in inference process, e.g. adver-
Figure 3: Test on MobileNetv2 architecture showed a higher sary attack. It has to be mentioned that only critical neurons
instability caused by projection layers. The criticality of sev- from projection layers (MobileNetV2 and EfficientNetB0)
eral neurons exceeds 1.0, which means that masking just one have such an ability, but they have to be chosen with respect
neuron can cause misclassification. to the mean and standard deviation of the calculated accu-
racy. Other models sensitivity to adversary noise are plotted
in Figure 8.
The experiments’ results shown in this work are only re-
lated to the ”mountain bike” class. The results for the fairly
simple ”traffic sign” class backed up the intuition about 5 Conclusion
simple features being predominantly filtered in early lay- At the beginning of this work in Chapter 2 we pointed out
ers, since their criticality raised significantly. We advise the the current functional safety issues and open research area
reader to visit our GitHub, where additional figures and related to convolutional neural networks. In Section 1 we
stored statistics for both classes can be found. described our motivation related to autonomous driving and
Accuracy Accuracy
0.0
0.5
1.0
Accuracy
0.00
0.25
0.50
0.75
1.00
blo No
ck m blo No
ex b _2_p askin ck5 ma
0.0
0.2
0.4
0.6
0.8
1.0
1.2
pa loc ro g blo _con sking
ex nded k_2_ ject 1 c v
pa _c pr 5 blo k1_c 2 48
nd on oje ck2 on 7
ed v_ ct
_co pro 9 v
blo _con 2 59
blo nv_p ject 4 ck v2
ck ro blo 1_co 100
blo _2_p ject 3 c n
ck ro blo k2_c v2 1
ck5 on 9
blo _2_p ject 7 _ v1
ex b ck_3 rojec blo conv 72
pa loc _p t 0 blo ck2_ 1 26
nd k_ ro ck co 6
ed 3_p jec
_co ro t 7 blo 3_co nv1
ck2 nv 6
ex bloc nv_p ject 2
pa k_ ro 1 3
blo _con 128
nd 13 jec
ed _pr t 1 c v
_c o 1 blo k1_c 2 12
ck on 5
blo onv_ ject 1 blo 3_co v2 5
ck_ pro 1 ck5 nv 7
blo 3_p ject 2
ck ro 8 blo _con 168
blo _2_p ject 1 (a) VGG16 ck v1
ck ro 3 blo 1_co 456
(c) MobileNetv2
ck2 nv
blo _2_p ject 2
ck ro 1 2
blo _con 35
blo _1_p ject 1 blo ck3_ v2 2
ck_ roj 8
e ck4 co 7
blo 6_pr ct 14 blo _co nv1
c o ck nv 2
ex b k_1_ ject 2 blo 5_co 3 49
pa loc pr 0 ck5 nv 1
ex nded k_2_ oject 2
pa _c pr 3 blo _con 303
nd on oje c v
ed v_ ct blo k1_c 2 29
_co pro 8 ck2 on 4
nv jec _co v2 4
_pr t 7 nv 7
oje 21
ct
2 27
neurons decreases and hence the mean accuracy increases.
to decreasing neurons’ criticality in case of MobileNetv2 model.
blo Accuracy co Accuracy
ck nv
0.0
0.5
1.0
0.0
0.5
1.0
blo 2a_p No co 2_blo No
nv ck m
ck ro ma
blo 2a_p ject_ skin co 2_blo 1_0_ ask
ck ro co g nv ck co ing
blo 1a_p ject_ nv 1 co 2_blo 1_0_ nv 1
ck ro co 8 nv ck co 08
2_b 1_ nv
blo 2a_p ject_ nv 1 co lock 0_co 180
ck ro co 1 n 1 n
blo 3a_p ject_ nv 1 co v2_b _0_c v 15
ck3 roj con 4 n lo o 4
e co v2_b ck1 nv 1
blo a_pr ct_c v 21 nv loc _0 55
c o o co 2_blo k1_0 _con
blo k1a_ ject_ nv 3 nv ck _c v 2
ck pr co 6 5 1 o
blo 3a_p ojec nv 2 co _bloc _0_c nv 6
ck2 roj t_co 8 n k o 8
e co v2_b 3_2_ nv 1
blo a_pr ct_c nv 7 nv loc co 60
ck oje on 2 k n
blo 1a_p ct_c v 19 co _bloc 1_1 v 34
ck ro on co nv2_ k1_0 _con 2
blo 3a_p ject_ v 12 nv blo _c v 2
ck ro co 5 c o 3
blo 1a_p ject_ nv 2 co _bloc k1_1 nv 2
nv k2 _c 19
c r c 5 _ o
blo k1a_ ojec onv co _bloc 3_co nv 1
ck pr t_c 8 n k n 0
blo 1a_p ojec onv co v3_b 3_1_ v 17
ck ro t_c 1 nv loc co 61
(b) ResNet50V2
5 k n
(d) EfficientNetB0
blo 2a_p ject_ onv co _bloc 4_1_ v 23
ck ro co 6 nv k2 co 5
blo 2a_p ject_ nv 1 co 2_blo _2_c nv 1
ck ro co 1 nv ck on 3
blo 4a_p ject_ nv 1 co 2_blo 1_0_ v 38
ck ro co 3 nv ck co 3
blo 3a_p ject_ nv 1 co 2_blo 1_0_ nv 7
ck ro co 0 nv ck co 1
blo 3a_p ject_ nv 1 co 2_blo 1_0_ nv 9
nv ck co 2
ck3 roj con 2 2_b 1_ nv
a_p ect_ v 2 loc 0_c 60
roj con 9 k1 on
ec v _1_ v 5
t_c 17 co 8
on nv
v2
6 51
Figure 6: Accuracy stability on normal dataset (for class ”mountain bike”), showing a gradual increase of accuracy with respect
models’ accuracy without masking can be found on the first position, marked in green. Going to the right, the criticality of the
Figure 5: Results of all models’ accuracy stability related to the 20 most critical neurons, evaluated on a normal dataset. The
Accuracy Accuracy
Accuracy
0.1
0.0
0.1
0.2
0.3
noise.
0.1
0.0
0.1
0.2
0.3
0.4
blo No
ck m blo No
0.0
0.2
0.4
0.6
0.8
1.0
ex b _2_p aski
pa loc ro ng
n
ck5 ma
blo _con sking 1.2
ex ded k_2_ ject
pa _c pr 15 c v
nd on oje blo k1_c 2 48
ck2 on 7
ed v_ ct
_co pro 9 v
blo _con 2 59
blo nv_p ject ck v2
ck ro 4 blo 1_co 100
blo _2_p ject c n
ck ro 3 blo k2_c v2 1
ck5 on 9
blo _2_p ject _ v1
ex b ck_3 roje 7 blo conv 72
pa loc _p ct blo ck2 1 26
nd k_ ro 0 ck _c 6
ed 3_ jec
_co pro t 7 blo 3_co onv1
ck2 nv 6
ex bloc nv_p ject
pa k_ ro 21 3
blo _con 128
nd 13 jec
ed _p t 1 c v
_c ro 1 blo k1_c 2 12
blo onv_ ject ck on 5
ck_ pro 11 blo 3_co v2 5
ck5 nv 7
blo 3_p jec 2
ck ro t 8 blo _con 168
(a) VGG16
blo _2_p ject ck v1
ck ro 13 blo 1_co 456
(c) MobileNetv2
ck2 nv
blo _2_p ject
ck ro 21 2
blo _con 35
blo _1_p ject blo ck3 v2 2
ck_ roj 18 ck _c 7
e
blo 6_pr ct 1 blo 4_co onv1
c o 4 ck nv 2
ex b k_1_ ject blo 5_co 3 49
pa loc pr 20 ck5 nv 1
ex nded k_2_ oject 2
pa _c pr 3 blo _con 303
nd on oje
ed v_ ct c v
blo k1_c 2 29
_co pro 8 ck2 on 4
nv jec _co v2 4
_pr t 7 nv 7
oje 21
ct
2 27
Accuracy Accuracy
of the few critical neurons can be found in layer block1 0 conv c0.
blo co
nv
0.00
0.25
0.50
0.75
ck2
0.00
0.25
0.50
0.75
blo a_p No co 2_blo No
nv ck m
ck ro ma
blo 2a_p ject_ skin co 2_blo 1_0_ ask
ck ro co g nv ck co ing
blo 1a_p ject_ nv 1 co 2_blo 1_0_ nv 1
ck ro co 8 nv ck co 08
2_b 1_ nv
blo 2a_p ject_ nv 1 co lock 0_co 180
ck ro co 1 n 1 n
blo 3a_p ject_ nv 1 co v2_b _0_c v 15
ck3 roj con 4 n lo o 4
e co v2_b ck1 nv 1
blo a_pr ct_c v 21 nv loc _0 55
c o o co 2_blo k1_0 _con
blo k1a_ ject_ nv 3 nv ck _c v 2
ck pr co 6 5 1 o
blo 3a_p ojec nv 2 co _bloc _0_c nv 6
ck2 roj t_co 8 n k o 8
e co v2_b 3_2_ nv 1
blo a_pr ct_c nv 7 nv loc co 60
ck oje on 2 k n
blo 1a_p ct_c v 19 co _bloc 1_1 v 34
ck ro on co nv2_ k1_0 _con 2
blo 3a_p ject_ v 12 nv blo _c v 2
ck ro co 5 c o 3
blo 1a_p ject_ nv 2 co _bloc k1_1 nv 2
nv k2 _c 19
c r c 5 _ o
blo k1a_ ojec onv co _bloc 3_co nv 1
ck pr t_c 8 n k n 0
blo 1a_p ojec onv co v3_b 3_1_ v 17
ck ro t_c 1 nv loc co 61
(b) ResNet50V2
5 k n
(d) EfficientNetB0
blo 2a_p ject_ onv co _bloc 4_1_ v 23
ck ro co 6 nv k2 co 5
blo 2a_p ject_ nv 1 co 2_blo _2_c nv 1
ck ro co 1 nv ck on 3
blo 4a_p ject_ nv 1 co 2_blo 1_0_ v 38
ck ro co 3 nv ck co 3
blo 3a_p ject_ nv 1 co 2_blo 1_0_ nv 7
ck ro co 0 nv ck co 1
blo 3a_p ject_ nv 1 co 2_blo 1_0_ nv 9
nv ck co 2
ck3 roj con 2 2_b 1_ nv
a_p ect_ v 2 loc 0_c 60
roj con 9 k1 on
ec v _1_ v 5
t_c 17 co 8
on nv
v2
6 51
model shows minimal accuracy fluctuation, whereas more modern models contains neurons with higher sensitivity to adversary
is obvious that different results of generating adversary attacks was achieved since initial models accuracy differs. The VGG16
Figure 8: Results of accuracy stability related to the 20 most critical neurons for all models, evaluated on adversary dataset. It
was explained by missing projection layers in Chapter 4.1. Deeper investigation showed that for the ”moutain bike” class, all
Figure 7: Compared to MobileNetv2 or EfficientNetB0, the ResNet50V2 architecture has a higher accuracy stability, which
0.8
0.6
0.4
Accuracy
0.2
0.0
Figure 9: MobileNetV2’s accuracy stability of the 100 most critical neurons, taken from the analysis of a normal dataset and
evaluated on an adversary dataset. Neurons with increased accuracy could be further used for diagnosis purposes, but have to be
chosen with respect to both mean and standard deviation of the resulting accuracy. In such a diagnostic case, masking multiple
neurons could be desirable and would lead to higher diagnoses accuracy.
missing validation process. Further, as outlined in Section 3, critical neurons. With further measures, the mean and stan-
we introduced a new metric and auxiliary analysis method dard deviation of the criticality should be decreased and the
which we implemented and verified on four classification flawless calculation of the neuron should be ensured. Con-
CNNs in Chapter 4. cretely this can be achieved by several approaches, such as:
We dedicated a great part of our work to introducing and • fine-tuning of the model with deterministic dropout and
testing an innovative method, the Neurons’ Criticality Anal- loss which will incorporate the layers criticality
ysis. The outcome of this analysis was a comprehensive
report diagram depicting the criticality of each layer and • plausibility check of the critical neurons or layers or re-
each neuron of evaluated model. The domain of criticality dundant computational branch results
is [−1, 2]. We discussed that masking neurons with negative
criticality can also have a positive influence on the model’s • storage of the neurons’ weights and biases in two places
decision confidence. We called this behavior ”anti-critical”. in RAM and comparing them
The inter-class anti-critical neurons could hypothetically be • introduction of deconvolutional layers in order to compute
removed from the decision process. This idea led us to the and evaluate the original inputs over critical connections
conclusion that the correlation between the neurons removed
during the pruning process and the anti-critical neurons dis- Our method can also be used for Out-of-Distribution de-
covered via NCA should be further investigated. tection, where instead of randomly sampling sub-networks
We claimed that using spatial aggregation via projection predictions, as it is done by MC dropout, deterministic
layers may on the one hand improve the high dimensional dropout would be based on several highly critical neurons
feature representation(Szegedy et al. 2016), but on the other for every class. Such an approach would decrease the com-
hand creates very critical dense connections, especially in putational demand and arguably increase the reliability and
the shallow layers, as we pointed out in Section 4.1. From transparency of such a network. In order to encourage ad-
functional safety point of view this isn’t necessarily nega- ditional experiments and deeper explorations, we published
tive, since the plausibility function could be applied to only a our code and supplement results on GitLab 1 .
concentrated area of neurons. In addition, some critical neu-
rons showed the ability to increase mean accuracy on adver- 6 Acknowledgment
sary dataset, which could be used in order to discover adver-
sary attacks and irregularities during inference. We hypoth- The work has been supported by the grant of the University
esize that an equilibrium between the position of the first of West Bohemia, project No. SGS-2019-027.
projection layer, number of critical neurons and models’ ac-
curacy should be further investigated. 1
https://gitlab.com/divisvaclav/cnn eval tool/-/tree/
As aforementioned, the purpose of NCA is to identify all wo gui branch
References Kendall, A.; and Gal, Y. 2017. What uncertainties do we need in
Bach, S.; Binder, A.; Montavon, G.; Klauschen, F.; Müller, K.-R.; bayesian deep learning for computer vision? In Advances in neural
and Samek, W. 2015. On pixel-wise explanations for non-linear information processing systems, 5574–5584.
classifier decisions by layer-wise relevance propagation. PloS one Lakshminarayanan, B.; Pritzel, A.; and Blundell, C. 2017. Sim-
10(7): e0130140. ple and scalable predictive uncertainty estimation using deep en-
Belle, V.; and Papantonis, I. 2020. Principles and Practice of Ex- sembles. In Advances in neural information processing systems,
plainable Machine Learning. 6402–6413.
BMW, D. e. a. 2019. Safety First for Automated Driv- Li, Z.; Peng, C.; Yu, G.; Zhang, X.; Deng, Y.; and Sun, J. 2018.
ing (SaFAD) Safety First for Automated Driving (SaFAD). DetNet: A Backbone network for Object Detection.
https://www.daimler.com/innovation/case/autonomous/safety- Lin, M.; Chen, Q.; and Yan, S. 2014. Network In Network.
first-for-automated-driving-2.html. Accessed: 2020-06-08. Liu, Y.; Chen, X.; Liu, C.; and Song, D. 2016. Delving into trans-
Brown, T. B.; Mané, D.; Roy, A.; Abadi, M.; and Gilmer, J. 2017. ferable adversarial examples and black-box attacks. arXiv preprint
Adversarial patch. arXiv preprint arXiv:1712.09665 . arXiv:1611.02770 .
Cheng, C.-H. 2020. Safety-Aware Hardening of 3D Object Detec- Mobileye. 2020. Advanced Technologies - Mobileye Future of Mo-
tion Neural Network Systems. arXiv preprint arXiv:2003.11242 bility - Advanced Technologies. https://www.mobileye.com/. Ac-
. cessed: 2020-11-13.
Deng, J.; Dong, W.; Socher, R.; Li, L.-J.; Li, K.; and Fei-Fei, Neal, R. M. 2012. Bayesian learning for neural networks, volume
L. 2009. Imagenet: A large-scale hierarchical image database. 118. Springer Science & Business Media.
In Computer Vision and Pattern Recognition, 2009. CVPR 2009. Rafegas, I.; Vanrell, M.; Alexandre, L. A.; and Arias, G. 2019. Un-
IEEE Conference on, 248–255. IEEE. derstanding trained CNNs by indexing neuron selectivity. Pattern
DeVries, T.; and Taylor, G. W. 2018. Learning confidence for Recognition Letters .
out-of-distribution detection in neural networks. arXiv preprint Salay, R.; and Czarnecki, K. 2018. Using machine learning
arXiv:1802.04865 . safely in automotive software: An assessment and adaption of
Eykholt, K.; Evtimov, I.; Fernandes, E.; Li, B.; Rahmati, A.; Xiao, software process requirements in ISO 26262. arXiv preprint
C.; Prakash, A.; Kohno, T.; and Song, D. 2018. Robust physical- arXiv:1808.01614 .
world attacks on deep learning visual classification. In Proceedings Sandler, M.; Howard, A.; Zhu, M.; Zhmoginov, A.; and Chen, L.-
of the IEEE Conference on Computer Vision and Pattern Recogni- C. 2018. Mobilenetv2: Inverted residuals and linear bottlenecks.
tion, 1625–1634. In Proceedings of the IEEE Conference on Computer Vision and
Gal, Y.; and Ghahramani, Z. 2016. Dropout as a bayesian approxi- Pattern Recognition, 4510–4520.
mation: Representing model uncertainty in deep learning. In inter- Shridhar, K.; Laumann, F.; and Liwicki, M. 2019. A comprehen-
national conference on machine learning, 1050–1059. sive guide to bayesian convolutional neural network with varia-
Goodfellow, I. 2018. Defense Against the Dark Arts: An overview tional inference. arXiv preprint arXiv:1901.02731 .
of adversarial example security research and future research direc- Shrikumar, A.; Greenside, P.; and Kundaje, A. 2017. Learn-
tions. arXiv preprint arXiv:1806.04169 . ing important features through propagating activation differences.
Goodfellow, I. J.; Shlens, J.; and Szegedy, C. 2015. Explaining and In Proceedings of the 34th International Conference on Machine
Harnessing Adversarial Examples. Learning-Volume 70, 3145–3153. JMLR. org.
Graves, A. 2011. Practical variational inference for neural net- Simonyan, K.; Vedaldi, A.; and Zisserman, A. 2013. Deep inside
works. In Advances in neural information processing systems, convolutional networks: Visualising image classification models
2348–2356. and saliency maps. arXiv preprint arXiv:1312.6034 .
He, K.; Zhang, X.; Ren, S.; and Sun, J. 2015. Deep Residual Learn- Simonyan, K.; and Zisserman, A. 2014. Very deep convolu-
ing for Image Recognition. tional networks for large-scale image recognition. arXiv preprint
arXiv:1409.1556 .
He, K.; Zhang, X.; Ren, S.; and Sun, J. 2016a. Deep residual learn-
Song, C.; He, K.; Wang, L.; and Hopcroft, J. E. 2018. Improving
ing for image recognition. In Proceedings of the IEEE conference
the Generalization of Adversarial Training with Domain Adapta-
on computer vision and pattern recognition, 770–778.
tion. arXiv preprint arXiv:1810.00740 .
He, K.; Zhang, X.; Ren, S.; and Sun, J. 2016b. Identity mappings
Sundararajan, M.; Taly, A.; and Yan, Q. 2017. Axiomatic attribu-
in deep residual networks. In European conference on computer
tion for deep networks. In Proceedings of the 34th International
vision, 630–645. Springer.
Conference on Machine Learning-Volume 70, 3319–3328. JMLR.
Henne, M.; Schwaiger, A.; Roscher, K.; and Weiss, G. 2020. org.
Benchmarking Uncertainty Estimation Methods for Deep Learn-
Szegedy, C.; Liu, W.; Jia, Y.; Sermanet, P.; Reed, S.; Anguelov, D.;
ing With Safety-Related Metrics. In SafeAI@ AAAI, 83–90.
Erhan, D.; Vanhoucke, V.; and Rabinovich, A. 2014. Going Deeper
Howard, A. G.; Zhu, M.; Chen, B.; Kalenichenko, D.; Wang, W.; with Convolutions.
Weyand, T.; Andreetto, M.; and Adam, H. 2017. Mobilenets: Effi- Szegedy, C.; Vanhoucke, V.; Ioffe, S.; Shlens, J.; and Wojna, Z.
cient convolutional neural networks for mobile vision applications. 2016. Rethinking the inception architecture for computer vision. In
arXiv preprint arXiv:1704.04861 . Proceedings of the IEEE conference on computer vision and pat-
ISO. 2011. Road vehicles – Functional safety. tern recognition, 2818–2826.
ISO2019. 2019. Road vehicles — Safety of the intended function- Tan, M.; and Le, Q. V. 2019. EfficientNet: Rethinking Model
ality. RFC 1654, RFC Editor. URL http://www.rfc-editor.org/rfc/ Scaling for Convolutional Neural Networks. arXiv preprint
rfc1654.txt. arXiv:1905.11946 .
Tesla. 2019. Autopilot and Full Self-Driving Capability Autopilot
and Full Self-Driving Capability. https://www.tesla.com/support/
autopilot. Accessed: 2020-06-08.
Zeiler, M. D.; and Fergus, R. 2014. Visualizing and understand-
ing convolutional networks. In European conference on computer
vision, 818–833. Springer.