=Paper=
{{Paper
|id=Vol-1649/187
|storemode=property
|title=Vulnerability of Machine Learning Models to Adversarial Examples
|pdfUrl=https://ceur-ws.org/Vol-1649/187.pdf
|volume=Vol-1649
|authors=Petra Vidnerová, Roman Neruda
|dblpUrl=https://dblp.org/rec/conf/itat/VidnerovaN16
}}
==Vulnerability of Machine Learning Models to Adversarial Examples==
https://ceur-ws.org/Vol-1649/187.pdf
ITAT 2016 Proceedings, CEUR Workshop Proceedings Vol. 1649, pp. 187–194
http://ceur-ws.org/Vol-1649, Series ISSN 1613-0073, c 2016 P. Vidnerová, R. Neruda
Vulnerability of machine learning models to adversarial examples
Petra Vidnerová, Roman Neruda
Institute of Computer Science, Academy of Sciences of the Czech Republic
petra s. as. z
Abstract: We propose a genetic algorithm for generating We propose a genetic algorithm for generating adversarial
adversarial examples for machine learning models. Such examples. Though the evolution is slower than techniques
approach is able to find adversarial examples without the described in [12, 4], it enables us to obtain adversarial ex-
access to model’s parameters. Different models are tested, amples even without the access to model’s weights. The
including both deep and shallow neural networks archi- only thing we need is to be able to query the network to
tectures. We show that RBF networks and SVMs with classify a given example. From this point of view, the mis-
Gaussian kernels tend to be rather robust and not prone classification of adversarial examples represent a security
to misclassification of adversarial examples. flaw.
This paper is organized as follows. Section 2 brings a
brief overview of machine learning models considered in
1 Introduction this paper. Section 3 describes the proposed genetic algo-
rithm. Section 4 describes the results of our experiments.
Deep networks and convolutional neural networks enjoy Finally, Section 5 concludes our paper.
high interest nowadays. They have become the state-of-
art methods in many fields of machine learning, and have
been applied to various problems, including image recog- 2 Deep and Shallow Architectures
nition, speech recognition, and natural language process-
ing [5]. 2.1 Deep and Convolutional Networks
In [12] a counter-intuitive property of deep networks is
described. It relates to the stability of a neural network Deep neural networks are feedforward neural networks
with respect to small perturbation of their inputs. The with multiple hidden layers between the input and output
paper shows that applying an imperceptible non-random layer. The layers typically have different units depend-
perturbation to an input image, it is possible to arbitrar- ing on the task at hand. Among the units, there are tra-
ily change the network prediction. These perturbations are ditional perceptrons, where each unit (neuron) realizes a
found by optimizing the input to maximize the prediction nonlinear function, such as the sigmoid function: y(z) =
error. Such perturbed examples are known as adversarial tanh(z) or y(z) = 1+e1 −z . Another alternative to the percep-
examples. On some datasets, such as ImageNet, the adver- tron is the rectified linear unit (ReLU): y(z) = max(0, z).
sarial examples were so close to the original examples that Like the sigmoid neurons, rectified linear units can be used
the differences were indistinguishable to the human eye. to compute any function, and they can be trained using al-
Paper [4] suggests that it is the linear behaviour in high- gorithms such as back-propagation and stochastic gradient
dimensional spaces what is sufficient to cause adversarial descent.
examples (for example, a linear classifier exhibits this be- Convolutional layers contain the so called convolutional
haviour too). They designed a fast method of generating units that take advantage of the grid-like structure of the in-
adversarial examples (adding small vector in the direction puts, such as in the case of 2-D bitmap images, time series,
of the sign of the derivation) and showed that adding these etc. Convolutional units perform a simple discrete convo-
examples to the training set further improves the general- lution operation, which – for 2-D data – can be represented
ization of the model. In [4], in addition, the authors state by a matrix multiplication. Usually, to deal with large data
that adversarial examples are relatively robust, and they (such as large images), the convolution is applied multiple
generalize between neural networks with varied number times by sliding a small window over the data. The con-
of layers, activations, or trained on different subsets of the volutional units are typically used to extract some features
training data. In other words, if we use one neural net- from the data, and they are often used together with the so-
work to generate a set of adversarial examples, these ex- called max pooling layers that perform an input reduction
amples are also misclassified by another neural network by selecting one of many inputs, typically the one with
even when it was trained with different hyperparameters, maximal value. Thus, the overall architecture of a deep
or when it was trained on a different set of examples. An- network for image classification tasks resembles a pyra-
other results of fooling deep and convolutional networks mid with smaller number of units in higher layers of the
can be found in [10]. networks.
This paper examines a vulnerability to adversarial ex- For the output layer, mainly for classification tasks, the
zj
amples throughout variety of machine learning methods. softmax function: y(z) j = Ke ezk is often used. It has
∑k=1
188 P. Vidnerová, R. Neruda
the advantage that the output values can be interpreted as some pixels. With the probability pmutate_pixel each pixel
probabilities of individual classes. is changed:
Networks with at least one convolutional layer are ii = ii + r,
called convolutional neural networks (CNN), while net-
where r is drawn from Gaussian distribution. As a selec-
works with all hidden layers consisting of perceptrons are
tion, the tournament selection with tournament size 3 is
called multi-layer perceptrons (MLP).
used.
The fitness function should reflect the following two cri-
2.2 RBF networks and Kernel Methods teria:
The history of radial basis function (RBF) networks can be 1. the individual should resemble the target image
traced back to the 1980s, particularly to the study of inter-
polation problems in numerical analysis [8]. The RBF net- 2. if we evaluate the individual by our machine learning
work [3] is a feedforward network with one hidden layer model, we aim to obtain a prescribed target output
realizing the basis functions and linear output layer. It rep- (i.e., misclassify it).
resents an alternative to classical models, such as multi-
layer perceptrons. There is variety of learning methods for Thus, in our case, a fitness function is defined
RBF networks [9]. as: f (I) = −(0.5 ∗ cdist(I,target_image) + 0.5 ∗
In 1990s, a family of machine learning algorithms, cdist(model(I),target_answer)), where cdist is a
known as kernel methods, became very popular. They Euclidean distance.
have been applied to a number of real-world problems, and
they are still considered to be state-of-the-art methods in 4 Experimental Results
various domains [14].
Based on theoretical results on kernel approximation,
The goal of our experiments is to test various machine
the popular support vector machine (SVM) [2, 13] algo-
learning models and their vulnerability to adversarial ex-
rithm was developed. Its architecture is similar to RBF –
amples.
one hidden layer of kernel units and a linear output layer.
The learning algorithm is different, based on search for a
separating hyperplane with the highest margin. Common 4.1 Overview of models
kernel functions used for SVM learning are linear hx, x′ i,
polynomial (γ hx, x′ i + r)d , Gaussian exp(−γ |x − x′ |2 ), and As a representative of deep models we use two deep ar-
sigmoid tanh(γ hx, x′ i + r). chitectures – an MLP network with rectified linear units
Recently, due to popularity of deep architectures, such (ReLU), and a CNN. The MLP used in our experiments
models with only one hidden layer are often referred to as consist of three fully connected layers. Two hidden layers
shallow models. have 512 ReLUs each, using dropout; the output layer has
10 softmax units. The CNN has two convolutional lay-
ers with 32 filters and ReLUs each, a max pooling layer,
3 Genetic Algorithms a fully connected layer of 128 ReLUs, and a fully con-
nected output softmax layer. In addition to these two mod-
To obtain an adversarial example for the trained machine els, we also used an ensemble of 10 MLPs. All models
learning model (such as a neural network), we need to were trained using the KERAS library [1].
optimize the input image with respect to network out- Shallow networks in our experiments are represented by
put. For this task we employ genetic algorithms (GA). an RBF network with 1000 Gaussian units, and SVM mod-
GA represent a robust optimization method working with els with Gaussian kernel (SVM-gauss), polynomial kernel
the whole population of feasible solutions [7]. The popu- of grade 2 and 4 (SVM-poly2 and SVM-poly4), sigmoidal
lation evolves using operators of selection, mutation, and kernel (SVM-sigmoid), and linear kernel (SVM-linear).
crossover. Both the machine learning model and the target SVMs were trained using the SCIKIT library [11], Grid
output are fixed during the optimization. search and crossvalidation techniques were used to tune
Each individual represents one possible input vector, i.e. hyper-parameters. For RBF networks, we used our own
one image encoded as a vector of pixel values: implementation. Overview of train and test accuracies can
be found in Tab. 1.
I = {i1 , i2 , . . . , iN },
where ii ∈< 0, 1 > are levels of grey, and N is the size of 4.2 Experimental setup
a flatten image. (For the sake of simplicity, we consider
only greyscale images in this paper, but it can be seen that The well known MNIST data set [6] was used. It contains
the same principle can be used for RGB images as well.) 70 000 images of hand written digits, 28 × 28 pixel each.
The crossover operator performs a classical two-point 60 000 are used for training, 10 000 for testing. The ge-
crossover. The mutation introduces a small change to netic algorithm was run with 50 individuals, for 10 000
Vulnerability of Machine Learning Models to Adversarial Examples 189
In general, it often happens that adversarial example
-li id
o
M 2
M 4
ar
s
evolved for one model is misclassified by some of the other
SV igm
SV aus
SV oly
SV oly
ne
-g
-p
-p
-s
models (see Tab. 6 and 7). There are some general trends:
M
M
M
N
LP
F
CN
RB
SV
M
• adversarial example evolved for CNN was never mis-
Train 0.96 1.00 1.00 0.99 1.00 0.99 0.87 0.95 classified by other models, and CNN never misclas-
Test 0.96 0.98 0.99 0.98 0.98 0.98 0.88 0.94 sified other adversarial examples than those evolved
Table 1: Overview of accuracies on train and test sets. for the CNN;
• adversarial examples evolved for MLP are misclas-
sified also by ensemble of MLPs (all cases except
generations, with crossover probability set to 0.6, and mu-
two) and adversarial examples evolved for ensemble
tation probability set to 0.1. The GA was run 9 times for
of MLPs are misclassified by MLP (all cases);
each model to find adversarial examples that resemble 9
different images (training samples from the beginning of • adversarial examples evolved for the SVM-sigmoid
training set). All images were optimized to be classified model are misclassified by SVM-linear (all cases ex-
as zero. cept two);
• adversarial examples for the SVM-poly2 model are
4.3 Results often (6 cases) misclassified by other SVMs (SVM-
poly4, SVM-sigmoid, SVM-linear), and in 4 cases
Figures 1 and 2 show two selected cases from our set of also by the SVM-gauss. In three cases it was also
experiments. For example, the first set of images shows a misclassified by MLP and ensemble of MLPs, in one
particular image of digit five from the training set, and best case, the adversarial example for SVM-poly2 is mis-
evolved individuals from the corresponding runs of GA for classified by all models but CNN (however, this ex-
individual models. ample is quite noisy);
In Tables 2–5, the outputs of individual models are
listed. In Tab. 2 and 4, we show output vectors for train- • adversarial example for the SVM-poly4 model is in
ing sample of digit five and four, respectively. In Tab. 3 two cases misclassified by all models but CNN, in
and 5, we show output vectors for adversarial examples different case it is misclassified by all but the CNN
from Fig. 1 and 2, respectively. and RBF models, and in one case by all but CNN,
RBF, and SVM-gauss models;
For this case, the adversarial examples were found for
MLP, CNN, ensemble of MLPs, SVM-poly2, and SVM- • RBF network, SVM-gauss, and SVM-linear were re-
sigmoid. For RBF network, SVM-gauss, SVM-poly4, and sistant to adversarial examples by genetic algorithm,
SVM-linear, the GA was not able to find image that re- however they sometimes misclassify adversarial ex-
sembles the digit 5 and at the same time it is classified as amples of other models. These examples are already
zero. quite noisy, however by human they would still be
If we look on a vulnerability of individual models over classified correctly.
all 9 GA runs we can conclude the following:
• MLP, CNN, ensemble of MLPs, and SVM-sigmoid 5 Conclusion
were always misclassifying the best individuals;
We proposed a genetic algorithm for generating adversar-
• RBF network, SVM-gauss, and SVM-linear; never ial examples for machine learning models. Our experi-
misclassified, i.e. the genetic algorithm was not able ment show that many machine models suffer from vulnera-
to find adversarial example for these models; bility to adversarial examples, i.e. examples designed to be
misclassified. Some models are quite resistant to such be-
• SVM-poly2 and SVM-poly4 were resistant to finding haviour, namely models with local units – RBF networks
adversarial examples in 2 and 5 cases, respectively. and SVMs with Gaussian kernels. It seems that it is the lo-
cal behaviour of units that prevents the models from being
Fig. 3 and 4 deal with the generalization of adversar- fooled.
ial examples over different models. For each adversarial Adversarial examples evolved for one model are often
example the figure lists the output vectors of all models. misclassified also by some of other models, as was elabo-
In the case of a digit 3, the adversarial example evolved rated in the experiments.
for MLP is also misclassified by an ensemble of MLPs,
and vice versa. Both examples are misclassified by SVM-
sigmoid. However, adversarial example for the SVM- Acknowledgements
sigmoid is misclassified only by the SVM-linear model. This work was partially supported by the Czech Grant
Adversarial example for SVM-poly2 is misclassified also Agency grant 15-18108S, and institutional support of the
with other SVMs, except the SVM-gauss model. Institute of Computer Science RVO 67985807.
190 P. Vidnerová, R. Neruda
0 1 2 3 4 5 6 7 8 9
RBF 0.04 -0.07 0.08 0.24 -0.04 0.73 -0.04 0.21 0.03 -0.18
MLP 0.00 0.00 0.00 0.00 0.00 1.00 0.00 0.00 0.00 0.00
CNN 0.00 0.00 0.00 0.00 0.00 1.00 0.00 0.00 0.00 0.00
ENS 0.00 0.00 0.00 0.00 0.00 1.00 0.00 0.00 0.00 0.00
SVM-gauss 0.00 0.00 0.00 0.00 0.00 1.00 0.00 0.00 0.00 0.00
SVM-poly2 0.00 0.00 0.00 0.02 0.00 0.98 0.00 0.00 0.00 0.00
SVM-poly4 0.00 0.00 0.00 0.02 0.00 0.98 0.00 0.00 0.00 0.00
SVM-sigmoid 0.01 0.00 0.03 0.32 0.00 0.61 0.00 0.01 0.01 0.01
SVM-linear 0.00 0.00 0.01 0.10 0.00 0.89 0.00 0.00 0.00 0.00
Table 2: Evaluation of the target digit five (see Fig. 1) by individual models.
0 1 2 3 4 5 6 7 8 9
RBF 0.21 -0.05 0.09 0.23 -0.04 0.51 -0.05 0.17 0.07 -0.09
MLP 0.98 0.00 0.00 0.00 0.00 0.01 0.00 0.00 0.00 0.00
CNN 0.95 0.00 0.01 0.01 0.00 0.02 0.00 0.00 0.01 0.00
ENS 0.98 0.00 0.01 0.00 0.00 0.01 0.00 0.00 0.00 0.00
SVM-gauss 0.00 0.00 0.01 0.39 0.00 0.59 0.00 0.00 0.00 0.00
SVM-poly 0.88 0.00 0.02 0.02 0.00 0.07 0.00 0.00 0.00 0.00
SVM-poly4 0.01 0.01 0.14 0.29 0.01 0.50 0.01 0.01 0.02 0.01
SVM-sigmoid 0.82 0.00 0.03 0.05 0.00 0.08 0.00 0.01 0.01 0.01
SVM-linear 0.02 0.03 0.21 0.21 0.02 0.33 0.03 0.04 0.05 0.05
Table 3: Evaluation of adversarial digit five (from Fig. 1) by individual models.
Target RBF MLP CNN ENS
SVM-rbf SVM-poly SVM-poly4 SVM-sigmoid SVM-linear
Figure 1: Best individuals evolved for individual models and digit five. The first ’Target’ image is the digit from the
training set, than follows adversarial examples evolved for individual models.
Vulnerability of Machine Learning Models to Adversarial Examples 191
0 1 2 3 4 5 6 7 8 9
RBF -0.06 0.07 0.16 0.06 0.67 0.00 0.01 0.15 -0.01 -0.05
MLP 0.00 0.00 0.00 0.00 1.00 0.00 0.00 0.00 0.00 0.00
CNN 0.00 0.00 0.00 0.00 1.00 0.00 0.00 0.00 0.00 0.00
ENS 0.00 0.00 0.00 0.00 1.00 0.00 0.00 0.00 0.00 0.00
SVM-gauss 0.00 0.00 0.00 0.00 0.99 0.00 0.00 0.00 0.00 0.00
SVM-poly2 0.00 0.00 0.00 0.00 0.97 0.00 0.00 0.01 0.00 0.00
SVM-poly4 0.00 0.00 0.00 0.01 0.96 0.00 0.00 0.01 0.00 0.00
SVM-sigmoid 0.00 0.00 0.01 0.02 0.94 0.00 0.00 0.01 0.00 0.01
SVM-linear 0.00 0.01 0.05 0.04 0.84 0.00 0.01 0.04 0.00 0.02
Table 4: Evaluation of the target digit four (see Fig. 2) by individual models.
0 1 2 3 4 5 6 7 8 9
RBF 0.06 0.08 0.14 0.08 0.49 -0.01 0.03 0.14 0.02 0.02
MLP 0.96 0.00 0.01 0.00 0.02 0.00 0.00 0.01 0.00 0.01
CNN 0.89 0.00 0.02 0.00 0.05 0.00 0.00 0.01 0.02 0.00
ENS 0.96 0.00 0.01 0.00 0.02 0.00 0.00 0.00 0.00 0.01
SVM-gauss 0.01 0.01 0.07 0.12 0.50 0.03 0.01 0.06 0.09 0.08
SVM-poly 0.75 0.00 0.03 0.01 0.09 0.00 0.01 0.04 0.03 0.04
SVM-poly4 0.71 0.01 0.04 0.01 0.11 0.02 0.02 0.04 0.02 0.02
SVM-sigmoid 0.84 0.00 0.02 0.03 0.03 0.02 0.01 0.01 0.00 0.03
SVM-linear 0.02 0.02 0.18 0.18 0.26 0.03 0.05 0.11 0.03 0.11
Table 5: Evaluation of adversarial digit four (from Fig. 2) by individual models.
Target RBF MLP CNN ENS
SVM-rbf SVM-poly SVM-poly4 SVM-sigmoid SVM-linear
Figure 2: Best individuals evolved for individual models and digit four. The first ’Target’ image is the digit from the
training set, than follows adversarial examples evolved for individual models.
192 P. Vidnerová, R. Neruda
Evolved against RBF
0
0 1 2 3 4 5 6 7 8 9
5 RBF 0.16 0.06 0.12 0.79 0.01 -0.02 -0.06 -0.00 0.02 0.03
MLP 0.00 0.00 0.00 1.00 0.00 0.00 0.00 0.00 0.00 0.00
10 CNN 0.00 0.00 0.00 1.00 0.00 0.00 0.00 0.00 0.00 0.00
ENS 0.00 0.00 0.00 1.00 0.00 0.00 0.00 0.00 0.00 0.00
15 SVM-gauss 0.00 0.00 0.00 1.00 0.00 0.00 0.00 0.00 0.00 0.00
SVM-poly 0.00 0.00 0.00 1.00 0.00 0.00 0.00 0.00 0.00 0.00
20 SVM-poly4 0.00 0.00 0.00 1.00 0.00 0.00 0.00 0.00 0.00 0.00
SVM-sigmoid 0.04 0.00 0.02 0.79 0.00 0.06 0.00 0.01 0.05 0.02
25 SVM-linear 0.00 0.00 0.00 0.96 0.00 0.00 0.00 0.00 0.03 0.00
0 5 10 15 20 25
Evolved against MLP
0
0 1 2 3 4 5 6 7 8 9
5 RBF 0.30 0.04 0.17 0.75 0.02 -0.03 -0.04 -0.01 -0.07 -0.00
MLP 0.96 0.00 0.01 0.01 0.00 0.00 0.00 0.00 0.01 0.01
10 CNN 0.00 0.00 0.00 1.00 0.00 0.00 0.00 0.00 0.00 0.00
ENS 0.86 0.00 0.01 0.04 0.00 0.00 0.00 0.00 0.01 0.08
15 SVM-gauss 0.00 0.00 0.00 0.99 0.00 0.00 0.00 0.00 0.00 0.01
SVM-poly 0.04 0.00 0.01 0.91 0.00 0.00 0.00 0.00 0.02 0.02
20 SVM-poly4 0.03 0.00 0.01 0.93 0.00 0.00 0.00 0.00 0.01 0.01
SVM-sigmoid 0.49 0.00 0.03 0.30 0.00 0.04 0.00 0.01 0.10 0.02
25 SVM-linear 0.25 0.02 0.10 0.30 0.02 0.05 0.02 0.03 0.18 0.06
0 5 10 15 20 25
Evolved against CNN
0
0 1 2 3 4 5 6 7 8 9
5 RBF 0.12 0.05 0.15 0.89 0.01 -0.18 -0.02 0.02 0.10 -0.03
MLP 0.00 0.00 0.00 1.00 0.00 0.00 0.00 0.00 0.00 0.00
10 CNN 0.94 0.00 0.02 0.01 0.00 0.00 0.00 0.00 0.03 0.00
ENS 0.00 0.00 0.00 1.00 0.00 0.00 0.00 0.00 0.00 0.00
15 SVM-gauss 0.00 0.00 0.00 1.00 0.00 0.00 0.00 0.00 0.00 0.00
SVM-poly 0.00 0.00 0.00 1.00 0.00 0.00 0.00 0.00 0.00 0.00
20 SVM-poly4 0.00 0.00 0.00 1.00 0.00 0.00 0.00 0.00 0.00 0.00
SVM-sigmoid 0.04 0.00 0.02 0.86 0.00 0.03 0.00 0.00 0.04 0.01
25 SVM-linear 0.00 0.00 0.00 0.99 0.00 0.00 0.00 0.00 0.00 0.00
0 5 10 15 20 25
Evolved against ENS
0
0 1 2 3 4 5 6 7 8 9
5 RBF 0.30 0.05 0.18 0.76 -0.01 -0.06 -0.04 -0.03 -0.05 -0.00
MLP 0.83 0.00 0.05 0.06 0.00 0.00 0.00 0.00 0.05 0.00
10 CNN 0.00 0.00 0.00 1.00 0.00 0.00 0.00 0.00 0.00 0.00
ENS 0.96 0.00 0.01 0.02 0.00 0.00 0.00 0.00 0.01 0.00
15 SVM-gauss 0.00 0.00 0.00 0.99 0.00 0.00 0.00 0.00 0.00 0.01
SVM-poly 0.02 0.00 0.01 0.94 0.00 0.00 0.00 0.00 0.01 0.02
20 SVM-poly4 0.01 0.00 0.00 0.96 0.00 0.00 0.00 0.00 0.01 0.01
SVM-sigmoid 0.40 0.00 0.03 0.35 0.01 0.06 0.00 0.01 0.11 0.02
25 SVM-linear 0.19 0.01 0.06 0.50 0.01 0.05 0.01 0.02 0.11 0.04
0 5 10 15 20 25
Figure 3: Model outputs for individual adversarial examples.
Vulnerability of Machine Learning Models to Adversarial Examples 193
Evolved against SVM-RBF
0
0 1 2 3 4 5 6 7 8 9
5 RBF 0.06 0.01 0.15 0.74 -0.00 -0.03 -0.04 -0.01 0.26 0.05
MLP 0.00 0.00 0.00 1.00 0.00 0.00 0.00 0.00 0.00 0.00
10 CNN 0.00 0.00 0.00 1.00 0.00 0.00 0.00 0.00 0.00 0.00
ENS 0.00 0.00 0.00 0.99 0.00 0.00 0.00 0.00 0.01 0.00
15 SVM-gauss 0.00 0.00 0.00 0.90 0.00 0.00 0.00 0.00 0.10 0.00
SVM-poly 0.00 0.00 0.00 0.50 0.00 0.00 0.00 0.00 0.50 0.00
20 SVM-poly4 0.00 0.00 0.00 0.60 0.00 0.00 0.00 0.00 0.40 0.00
SVM-sigmoid 0.03 0.00 0.03 0.63 0.00 0.09 0.00 0.01 0.19 0.02
25 SVM-linear 0.00 0.00 0.00 0.36 0.00 0.00 0.00 0.00 0.63 0.00
0 5 10 15 20 25
Evolved against SVM-poly
0
0 1 2 3 4 5 6 7 8 9
5 RBF 0.32 0.02 0.17 0.86 -0.01 -0.09 -0.09 -0.03 -0.12 0.01
MLP 0.00 0.00 0.00 1.00 0.00 0.00 0.00 0.00 0.00 0.00
10 CNN 0.00 0.00 0.00 1.00 0.00 0.00 0.00 0.00 0.00 0.00
ENS 0.00 0.00 0.00 1.00 0.00 0.00 0.00 0.00 0.00 0.00
15 SVM-gauss 0.00 0.00 0.00 0.99 0.00 0.00 0.00 0.00 0.00 0.00
SVM-poly 0.87 0.00 0.02 0.04 0.00 0.00 0.00 0.00 0.04 0.02
20 SVM-poly4 0.38 0.01 0.11 0.23 0.01 0.02 0.01 0.02 0.15 0.04
SVM-sigmoid 0.55 0.01 0.04 0.19 0.01 0.05 0.01 0.01 0.13 0.02
25 SVM-linear 0.71 0.01 0.02 0.06 0.01 0.02 0.01 0.01 0.15 0.01
0 5 10 15 20 25
Evolved against SVM-poly4
0
0 1 2 3 4 5 6 7 8 9
5 RBF 0.07 0.02 0.12 0.84 0.04 -0.07 -0.07 -0.01 0.10 0.06
MLP 0.00 0.00 0.00 1.00 0.00 0.00 0.00 0.00 0.00 0.00
10 CNN 0.00 0.00 0.00 1.00 0.00 0.00 0.00 0.00 0.00 0.00
ENS 0.00 0.00 0.00 1.00 0.00 0.00 0.00 0.00 0.00 0.00
15 SVM-gauss 0.00 0.00 0.00 0.99 0.00 0.00 0.00 0.00 0.01 0.00
SVM-poly 0.00 0.00 0.00 0.57 0.00 0.00 0.00 0.00 0.41 0.03
20 SVM-poly4 0.00 0.00 0.00 0.67 0.00 0.00 0.00 0.00 0.31 0.03
SVM-sigmoid 0.04 0.00 0.02 0.73 0.01 0.07 0.00 0.01 0.08 0.02
25 SVM-linear 0.01 0.00 0.00 0.58 0.00 0.00 0.00 0.00 0.39 0.01
0 5 10 15 20 25
Evolved against SVM-sigmoid
0
0 1 2 3 4 5 6 7 8 9
5 RBF 0.30 0.04 0.22 0.94 -0.01 -0.14 -0.10 0.02 -0.21 -0.02
MLP 0.00 0.00 0.00 1.00 0.00 0.00 0.00 0.00 0.00 0.00
10 CNN 0.00 0.00 0.00 1.00 0.00 0.00 0.00 0.00 0.00 0.00
ENS 0.00 0.00 0.00 1.00 0.00 0.00 0.00 0.00 0.00 0.00
15 SVM-gauss 0.00 0.00 0.00 1.00 0.00 0.00 0.00 0.00 0.00 0.00
SVM-poly 0.06 0.00 0.03 0.84 0.00 0.01 0.00 0.00 0.03 0.02
20 SVM-poly4 0.04 0.00 0.01 0.93 0.00 0.00 0.00 0.00 0.01 0.01
SVM-sigmoid 0.74 0.00 0.02 0.12 0.00 0.02 0.00 0.00 0.08 0.01
25 SVM-linear 0.41 0.02 0.06 0.18 0.02 0.05 0.02 0.03 0.17 0.05
0 5 10 15 20 25
Figure 4: Model outputs for individual adversarial examples.
194 P. Vidnerová, R. Neruda
Evolved for Also misclassified by Evolved for Also misclassified by
Example 1: digit 5 Example 8: digit 3
MLP — MLP ensemble, SVM-sigmoid
ensemble MLP ensemble MLP, SVM-sigmoid
CNN — CNN —
SVM-poly2 SVM-poly4, SMV-sigmoid, SVM-poly2 SVM-poly4, SVM-sigmoid, SVM-linear
SVM-linear SVM-sigmoid SVM-linear
SVM-sigmoid — Example 9: digit 1
Example 3: digit 4 MLP ensemble
MLP ensemble ensemble MLP
ensemble MLP CNN —
CNN — SVM-poly2 all except CNN
SVM-poly2 ensemble, MLP, SVM-gauss, SVM-poly4, SVM-sigmoid MLP, ensemble, SVM-gauss,
SVM-sigmoid, SVM-linear SVM-poly2, SVM-poly4, SVM-linear
SVM-poly4 RBF, ensemble, MLP, SVM-gauss, Example 10: digit 4
SVM-poly4, SVM-sigmoid, SVM-linear MLP ensemble
SVM-sigmoid SVM-linear ensemble MLP
Example 4: digit 1 CNN —
MLP ensemble SVM-poly2 SVM-poly4, SVM-linear
ensemble MLP SVM-poly4 MLP, ensemble, SVM-gauss, SVM-poly2,
CNN — SVM-sigmoid, SVM-linear
SVM-poly2 SVM-gauss, SVM-poly4, SVM-sigmoid, SVM-sigmoid SVM-linear
SVM-linear
SVM-sigmoid SVM-linear Table 7: Generalization of adversarial examples.
Example 5: digit 9
MLP ensemble
[4] Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy.
ensemble MLP Explaining and harnessing adversarial examples, 2014.
CNN — arXiv:1412.6572.
SVM-poly2 ensemble, MLP, SVM-poly2 [5] Yoshua Bengio Ian Goodfellow and Aaron Courville. Deep
SVM-poly4, SVM-sigmoid, SVM-linear learning. Book in preparation for MIT Press, 2016.
SVM-poly4 all except CNN [6] Yann LeCun and Corinna Cortes. The mnist database of
SVM-sigmoid SVM-linear handwritten digits, 2012.
Example 6: digit 2 [7] M. Mitchell. An Introduction to Genetic Algorithms. MIT
MLP — Press, Cambridge, MA, 1996.
ensemble MLP [8] J. Moody and C. Darken. Fast learning in networks
CNN — of locally-tuned processing units. Neural Computation,
SVM-poly4 MLP, ensemble, SVM-poly2, 1:289–303, 1989.
SVM-sigmoid, SVM-linear [9] R. Neruda and P. Kudová. Learning methods for radial ba-
SVM-sigmoid SVM-linear sis functions networks. Future Generation Computer Sys-
Example 7: digit 1 tems, 21:1131–1142, 2005.
MLP ensemble [10] Anh Mai Nguyen, Jason Yosinski, and Jeff Clune. Deep
ensemble MLP neural networks are easily fooled: High confidence pre-
CNN — dictions for unrecognizable images. CoRR, abs/1412.1897,
SVM-sigmoid — 2014.
[11] F. Pedregosa et al. Scikit-learn: Machine learning in
Table 6: Generalization of adversarial examples. Python. Journal of Machine Learning Research, 12:2825–
2830, 2011.
[12] Christian Szegedy, Wojciech Zaremba, Ilya Sutskever,
References Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob
Fergus. Intriguing properties of neural networks, 2013.
arXiv:1312.6199.
[1] François Chollet. Keras.
https://github. om/f hollet/keras, 2015. [13] V. N. Vapnik. Statistical Learning Theory. Wiley, New-
York, 1998.
[2] C. Cortes and V. Vapnik. Support-vector networks. Ma-
chine Learning, 20(3):273–297, 1995. [14] J. P. Vert, K. Tsuda, and B. Scholkopf. A primer on kernel
methods. Kernel Methods in Computational Biology, pages
[3] F. Girosi, M. Jones, and T. Poggio. Regularization theory
35–70, 2004.
and Neural Networks architectures. Neural Computation,
2:219–269, 7 1995.