Universal adversarial perturbations (UAPs) are small perturbations imposed on images that are able to fool a single convolutional neural network image classi er. They have been shown to generalise well to other neural networks. Here, we report on our reproduction e ort of the results given in a work by Moosavi-Dezfooli et al. on UAPs and study two methods to construct UAPs for several neural networks. While the results are not strong enough to make general conclusions, they suggest that UAPs indeed pro t from being constructed on several neural networks. Also, we show that a linear interpolation between two UAPs does not produce a viable UAP on both networks.
The discovery of Szegedy et al. [12] that several machine learning models
including deep neural networks are vulnerable to adversarial attacks was seminal
for a new sub eld of studying deep learning. Probably the most intriguing, but
also unsettling result was that adversarial examples can be made quite
imperceptible to the human eye while still fooling a convolutional neural network to
misclassify the image [
These results suggest that neural networks and convolutional neural networks for image classi cation in particular partly share a common structure that can be exploited by universal adversarial perturbations (UAPs) while yet other aspects are di erent. Thus, understanding UAPs provides a window into the weaknesses of neural networks. By building UAPs that are viable on several networks, a common weakness of those networks is identi ed. This in turn helps in building more robust networks, for example again by adversarial training. Even more generally, therefore, we view work on UAPs as one more way of (partly) approaching the bigger question of why neural networks with a large number of weights work at all (or when they don't).
In this context, we ask whether combining two neural networks in
generating adversarial perturbations can improve the transferability of UAPs to new
convolutional neural networks. We observe that in the approach by
MoosaviDezfooli et al. [
Given the practical potential and relevance of Deep Learning and the
potential security threats of adversarial perturbations, nding a robust resolution
is important and urgent. However, research into the topic is hampered by the
reproducibility crisis in machine learning [10]. Research results are often di cult
to reproduce due to undocumented values for hyperparameters, software library
versions etc. [
This paper is organised as follows. In Sec. 2 we brie y review related work. In
Sec. 3 we present the basic methods used to generate adversarial and universal
adversarial perturbations as introduced by Moosavi-Dezfooli et al. [
Many authors have suggested adversarial perturbation generating methods in
various di erent settings. These attempts often generate per-instance
perturb3 Instead of combining UAPs from di erent networks, UAPs of networks trained on
di erent data domains can be combined, as done e.g. by Naseer et al. [
Many of these methods generate perturbations that transfer well between di erent model architectures. As part of our contribution we seek methods that optimise the transferability of perturbations between models. Our approach tries to achieve this by using multiple models to generate perturbations. 3 3.1
Given an image classi er k^(x) = sgn (f (x)) that is based on the sign of a classi cation function f : Rn ! R, adversarial attacks seek a perturbation v such that
k^(x + v) = sgn (f (x + v)) 6= sgn (f (x)) = k^(x): In the following we brie y describe the adversarial perturbation generating method DeepFool and its generalisations to UAPs in the multi-class classi cation setting. We then describe the two approaches analysed here to build UAPs from several networks. 3.2
In the generation procedure DeepFool [
kfw(xk)2p w. As f is nonlinear in general, the Taylor approximation of the function f around x, f (x + v) = f (x) + (rf (x))T v is used to iteratively reach the decision boundary. In the multi-class setting considered here, an additional step is required that identi es the closest decision boundary. 4 In practice and also in our code, the vectors (elements of a vector space) such as x and v have additional structure such as (for images) height, width and depth that is used to implement the classi er k^. In the eld of Deep Learning these data structures are therefore often called tensors. 3.3
Perturbations for each image in a dataset X (such as those generated using
DeepFool) can be combined to form universal adversarial perturbations for a
single network [
Intriguingly, although the directions to the class boundaries vary for di
erent training images, the resulting average over all image perturbations works
well according to [
In the following subsections, we detail two approaches to generating UAPs for several classi ers.
adding up the perturbations generated using DeepFool, there is a natural way to combine perturbations generated by the two networks: We add up the contributions from all networks. We note that adding up the perturbations is not a commutative operation, since projections take place once the size (norm) of the perturbation becomes too large. The precise procedure used here is given in Alg. 1.
Variants of Alg. 1 exist that sample the images di erently. One might for example generate perturbation by alternating the classi er for each image. We show this variant in Alg. 2. We evaluate both variants and compare their performance in Tab. 2.
Interpolation Between UAPs on Individual Networks A simple yet instructive alternative to the above rather involved perturbation construction is given by a simple weighted average of the perturbation vectors generated on the individual networks. If we restrict our attention to the combination of two neural networks for now, then the weighted averages of the perturbations lie on a line in the high-dimensional vector space of images. In the following, we speci cally investigate whether any perturbation lying on this line improves on the fooling capability of the two endpoints with respect to a third network. More formally, given UAPs vf and vg of two neural networks f and g, we consider vfg ( ) = vf + (1 ) vg (1) for values 2 [0; 1]. We seek the value of that maximises the average fooling rate over f , g and a third network, given the current training data set. Outlook to Other Approaches We have also investigated other, more involved approaches that led to unsatisfactory results. In particular, we were interested in constructing a multi-class DeepFool procedure that uses the gradients of two networks towards the next class boundary to compute a perturbation of a single image. One might try to nd perturbations towards class boundaries that are aligned as much as possible, but where the class boundaries correspond to di erent classes in di erent networks. As of now, these attempts have not provided e cient UAPs for multiple networks.
1 Input:Data set X, set of classi ers K, desired norm k:kp of the perturbation 2 Output: Universal perturbation vector v 3 Initialise v 0 4 while Average fooling rate is too low and max. number of iterations is not reached do foreach image x 2 X do foreach k^ 2 K do if k^(x) = k^(x + v) then
v DeepFool(x + v; k^)
v v + v
v p v= kvkp
In this section we rst discuss our attempt at the reproduction of the results
in [
1 Input:Data set X, ordered set of classi ers K, desired norm k:kp of the perturbation 2 Output: Universal perturbation vector v 3 Initialise v 0 4 while Average fooling rate is too low and maximum number of iterations is not reached do 5 foreach image x 2 X do 6 k^ rst element of K 7 K cyclic rotation of K if k^(x) = k^(x + v) then v DeepFool(x + v; k^) v + v p v= kvkp
Moosavi-Dezfooli et al. tested the DeepFool and Universal Adversarial
Perturbations algorithms on 5 di erent neural networks [
Fig. 1 shows the perturbations generated. In their general structure and
appearance, they are similar to the ones reported in [
Tab. 1 shows the achieved fooling rates for the tested models (left values,
in boldface) and the fooling rates reported by Moosavi-Dezfooli et al. [
To reproduce these numbers, several insu ciently documented design choices
had to be researched. For example, the original results are not stated for a given
epoch but using a stopping condition on the error rate. Values reported here
are for epoch 20, at which point the mean value over the last 5 epochs typically
varies by less than 0.005. Parameter values are p = 1 and = 10. We used
a maximum of 10 DeepFool update iterations. The overshoot parameter was
= 0:02. These parameter values were taken from the work of Moosavi-Dezfooli
et al. [
Another important parameter is the number of tested class boundaries5, called num classes. It gives the number of class boundaries in whose direction a perturbation is searched for. Computation time is highly sensitive to this para5 As given in the code at https://github.com/LTS4/universal/ meter, and checking all 1000 training boundaries of ImageNet was infeasible. Higher values generally improve the fooling performance, though exceptions are observed. Other minor parameters such as random initialisation values for the train-test split, etc. were taken from the provided code. We have tried to optimise these parameters using grid searches, but the available computing resources have limited these e orts.
As part of this contribution we provide our code for the above reproduction e ort and include the NeurIPS-2019 reproducibility checklist6. 4.2
In this section we present our results for constructing UAPs that make use of two neural networks. We discuss results for the two approaches presented in Sec. 3.4, the alternating generation of perturbations and the linear interpolation between UAP on individual networks.
The alternating generation of perturbations as given in Alg. 1 has been applied to the set of classi ers K = fInception; VGG-16g. In Tab. 2 the fooling rates are reported for these two networks and for ResNet-152. The table has two sections. In the upper section, the network listed in the rst column is used to generate the perturbation. The columns give the fooling rates on the network given in the column title as well as their average as our measure for the generalisability of the UAPs. We include the original network in this average. The second section reports results by applying the alternating generation of UAPs and the interpolation method given in Eq. (1) using the value = 0:05.
Alg. 1 achieved results similar to the ones of a perturbation generated on VGG-16 only. Alg. 2 achieved slightly higher fooling rates on VGG-16 and ResNet-152 than a perturbation generated directly on VGG-16 (with an absolute increase of 2 and 3 percentage points, respectively). For Inception, the perturbation achieved a fooling rate of 67%. This is 15% below the measured self-fooling rate of Inception (82%) but 7% higher than the fooling rate achieved with a VGG-16 model. 6 See the le checklist.md in https://github.com/mauruskuehne/lwda-paper
For a linear interpolation between the UAPs of VGG-16 and Inception V1, the best results were achieved for = 0:05 (see Eq. (1)). Using this con guration, the fooling rates remained essentially unchanged compared to the UAP generated on VGG-16 only. The small value = 0:05 results in a perturbation that is similar to the VGG-16 perturbation as the VGG-16 perturbation is weighted with 1 = 0:95 while the Inception perturbation contributes only with a weight of 5%. Choosing 2 [0:1; 0:15; : : : ; 0:4] resulted in perturbations with lower fooling rates for both models with respect to the rates achieved by separately training UAPs on the two networks. For 2 [0:4; 1:0], the fooling rates for Inception improved again but did not exceed the fooling rate of a UAP generated on Inception itself. The fooling rate on VGG-16 continued to deteriorate, stabilising at a low fooling rate of 15% after 0:65. This suggests that linear interpolation does not result in improved fooling rates.
For most models the fooling rates reported in the original paper could not be achieved, indicating that our reproduction results fell short of being satisfactory. For VGG-F, VGG-16, VGG-19 and ResNet-152 our self-fooling rates were between 4 and 24 absolute percentage points lower. For Inception we achieved a self-fooling rate 3 absolute percentage points higher than the one reported in the original paper. Further research is needed to state the precise conditions under which a reliable reproduction of the reported fooling rates is possible. As a step in this direction we have provided our code.
The non-diagonal values in Tab. 1 are large but typically signi cantly smaller than the diagonal values. They show a degree of transferability of UAPs generated with DeepFool to other models. Therefore, despite the reproducibility problems, these results broadly con rm that UAPs generated with DeepFool generalise to other network architectures. Nevertheless, it is clear that some aspects of UAPs are speci c to a given neural network architecture. We discuss our results on nding a way to improve the non-diagonal elements (potentially at the cost of the diagonal ones) in the next section. Interestingly, we achieved lower fooling rates than Moosavi-Dezfooli et al., except for the Inception network, for which we achieve 3 to 8 absolute percentage points higher fooling rates. This di erence may be due to di erent stopping criteria, resulting in MoosaviDezfooli et al. running less optimisation epochs. Another possibility is that the chosen hyperparameter values for DeepFool and UAP might be particularly well suited or optimised for the Inception model. This in turn would explain the lower fooling rates achieved on other models. 5.2
As the results in Sec. 4.2 clearly show a linear interpolation between two UAPs does not give good results. This suggests that using a weighted average to combine UAPs is not a suitable approach to produce good UAPs for several neural networks. A more sophisticated UAP-combination procedure is clearly necessary to generate perturbations that fool both networks to a high degree.
The results given by the alternating generation of perturbations (Alg. 2) are much better (see Tab. 2). The fooling rate of the perturbation generated jointly on Inception and VGG-16 is better than the ones generated on any one of the two networks. A perturbation generated on Inception achieves a fooling rate of 16% on VGG-16 while a perturbation generated on VGG-16 achieves a fooling rate of 60% on Inception. Both rates are lower than the ones of a perturbation generated jointly on Inception and VGG-16, achieving 67% on Inception and 61% on VGG-16. Furthermore, the fooling rate of a jointly trained UAP on Inception and VGG-16 on a third network (Resnet-152) is better than the fooling rate of both single-network UAPs. The UAP generated jointly on both networks even worked slightly better for VGG-16 than the one trained on VGG-16 alone. The reason for this e ect and its statistical signi cance are not yet established.
In judging these results, we note that estimates of their uncertainties are still lacking due to constraints on our computational resources. Ideally, one would like to provide mean and standard deviation values of all these numbers over multiple train-test splits as encouraged by the NeurIPS-2019 reproducibility checklist. Furthermore, the choice of the ImageNet training and testing data and the hyperparameter values (such as p, , num classes, etc.) should be investigated. 6
The results reported here on generalising UAPs across several networks clearly have to be interpreted cautiously given the fact that even the reproduction of previously reported results has not been satisfactory. Establishing reproducibility standards for machine learning publications remains a crucial challenge that is hampering progress.
With the above caution in mind, the results reported here suggest that nding universal adversarial perturbations that generalise across di erent convolutional neural networks is not a hopeless endeavour. As we found, such a UAP is likely not a linear combination of UAPs of di erent networks but must be constructed in a more subtle way. Our best approach, Alg. 2, most certainly is not optimal. Nevertheless, it already shows some promising results: The generalisability of the fooling rates to ResNet-152 is enhanced by combining the UAPs of two networks, with respect to the UAPs generated on either one of the Inception or VGG-16 network. This suggests that combining several or even many networks might produce UAPs that are e cient on a whole class of trained convolutional neural networks. Neural Information Processing Systems 32, pp. 12905{12915. Curran Associates, Inc. (2019), http://papers.nips.cc/paper/9450-cross-domain-transferabilityof-adversarial-perturbations.pdf 10. Ra , E.: A step toward quantifying independently reproducible machine learning research. In: Wallach, H., Larochelle, H., Beygelzimer, A., d’AlcheBuc, F., Fox, E., Garnett, R. (eds.) Advances in Neural Information Processing Systems 32, pp. 5485{5495. Curran Associates, Inc. (2019), http://papers.nips.cc/paper/8787-a-step-toward-quantifying-independentlyreproducible-machine-learning-research.pdf 11. Ren, K., Zheng, T., Qin, Z., Liu, X.: Adversarial attacks and defenses in deep learning. Engineering 6(3), 346 { 360 (2020). https://doi.org/10.1016/j.eng.2019.12.012 12. Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I., Fergus, R.: Intriguing properties of neural networks. In: International Conference on Learning Representations. ICLR (2014), URL http://arxiv.org/abs/1312.6199