Characterizing how effective a machine learning algorithm is while being trained and tested on slightly different data is a widespread matter. The property of models which perform well under this general framework is commonly known as robustness. We propose a class of model-agnostic empirical robustness measures for image classification tasks. To any random image perturbation scheme, we attach a robustness measure that empirically checks how easy it is to perturb a labelled image and cause the model to misclassify it. We also introduce a methodology for training more robust models using the information gained about the empirical robustness measure of the training set. We only keep a fraction of datapoints that are robust according to our robustness measure and retrain the model using it. Our methodology validates that the robustness of the model increases by measuring its empirical robustness on test data.
During the last decade, the field of machine learning has made considerable advances in many tasks, such as image classification, object detection, machine translation, or question answering, with deep neural networks easily becoming the state-of-the-art approaches [Touvron et al., 2020; Zhang et al., 2020; Edunov et al., 2018]. The main priority has been on the capacity of the models to perform well on the test set of some well-known datasets (MNIST, CIFAR, SQuAD) [LeCun and Cortes, 2010; Krizhevsky, 2009; Rajpurkar et al., 2016]. However, the training and the test sets are usually generated from the same underlying distribution, leaving the model’s performance under distribution shifts unknown. Given that machine learning techniques are being employed in sensitive tasks, such as self-driving cars and healthcare, the robustness should become a crucial metric to be taken into consideration together with the accuracy when evaluating the performance of models.
We will mainly focus on safety and robustness for image classification tasks, but the work can be easily extended to other topics.
Distribution shifts, which affect the performance of machine learning systems, can mainly occur because of two reasons. The first reason, adversarial attacks [Wiyatno et al., 2019; Szegedy et al., 2014], has been receiving growing attention over the past years. Adversarial attacks are ”hidden messages” [Wiyatno et al., 2019] added on top of images which are nearly imperceptible to the human eye, but which cause the model to fault, in other words creating ”machine illusions”.
The second reason, covariate shift [Shimodaira, 2000], is encountering a natural change in the data distribution. For example, imagine an autonomous car model trained solely on rainy and sunny conditions in a city where it has not been snowing over the past five years. However, one day it starts snowing, and the image recognition system of the autonomous car could have serious issues in identifying objects and road signs because of completely different lighting conditions.
While improving models to be less exposed to known adversarial attacks is very important, one has to keep in mind that this is, after all, an adversarial game, where the attacker and the security researcher keep alternately coming up with better strategies. For example, the adversarial attack strategy Fast Gradient Sign Method [Szegedy et al., 2014] can be mitigated by Adversarial training [Szegedy et al., 2014], which can in turn be bypassed by R+FGSM [Trame`r et al., 2018]. The defense methods against adversarial attacks seek to make the model robust with respect to certain adversarial points in the neighbourhood of unaltered images.
Therefore, one is prompted to consider a more general robustness framework, in which the interest lays in the model not making a mistake anywhere in the neighbourhood of an image1. There exist various tools that can achieve robustness guarantees of deep neural networks [Ruan et al., 2018; Tjeng et al., 2019], but most of them are usually very dependent on the model’s architecture, either not being able to scale with deeper networks, or only working with certain kinds of layers.
1For example, the neighbourhood could be specified by some metric ball around the image.
We propose a model-agnostic2 empirical method for estimating the robustness of a model. This estimation of a model near an image X is done by iteratively sampling datapoints close to it, according to a specified random scheme3. It feeds each of the sampled datapoints to the model and stops either when the model classifies them incorrectly, or when a maximum number of steps has been reached. The number of such sampling steps serves as a proxy for the local robustness around image X. Intuitively, the easier it is to perturb the label of X by sampling around it, the less robust the model is around it. We use this method for estimating the robustness of the model on entire datasets, by locally checking the model’s robustness around each datapoint and combining the results.
We also claim that the robustness of the model is correlated with the inherent robustness of the images with respect to the classification task. Therefore, the robustness of a model depends on both the architecture’s robustness itself, but also on the inherent robustness of the datapoints it has been trained on.
We believe that training a model on certain correctly labelled images can lead towards highly unnatural borders between classes. These might be datapoints that we would rather misclassify than include in the model at an additional high cost of robustness. We test this hypothesis and obtain indeed a more robust model by discarding the not-robust images from the training process.
Our main technical contributions are introducing the empirical robustness measure that is model-agnostic and the training methodology based on robust images.
An important general direction we want to shed light on is that images from classification tasks should be seen as carrying an inherent level of robustness, which could be estimated and exploited.
2The method does not need to have any knowledge about the architecture of the model. Note that the model does not necessarily have to be a deep neural network.
3The random scheme should not alter the underlying true class of the image that we sampled around. Intuitively, the samples should be classified by a human in the same way as the original image is. 2 2.1
We propose a class of empirical robustness measures RPR (Randomized Perturbation Robustness) for image classification tasks, which is model-agnostic. Let R be a random image perturbation scheme.4 The empirical robustness RPR(R) of a model M with respect to a datapoint x belonging to class y is the minimum between MAX STEPS and the expected number of retrying steps of applying R to the original x such that M does not classify R(x) as y.
If the empirical robustness of M with respect to (x; y) is MAX STEPS, we stop and deem x as robust; otherwise as not-robust.
Note that the random perturbations of an image are not applied on top of previous perturbation attempts, but rather on the original image. This perturbation process is repeated until the conditions above are fulfilled. 2.2
The introduced framework is a simple empirical way of assessing a model’s robustness near an image. It is suitbale under various setups, such as the random image perturbation scheme of adding weather conditions5 in the autonomous car situation.
We propose two use cases based on the empirical robustness measure introduced above: One estimating the model robustness on an entire (test) dataset and another one training a model only using the images that are deemed as robust in order to obtain a more robust model.
The first use case, estimating the robustness of the model on an entire dataset, is done by applying the Randomized Perturbation Robustness method described above on each datapoint and computing the percentage of images that are deemed as robust. The model-agnosticism makes it an easy plug-in method in any classification task and can easily be introduced as a baseline check for machine learning systems.
The second use case is based on our claim that the robustness of the model with respect to a datapoint can be seen to 4For example Gaussian noise, replacing at most k pixels of an image, blurring, etc.
5Applying snow, fog, rain effects, etc. some extent as the inherent robustness of the datapoint with respect to the classification task. This allows us to retrain the model using only images from the train set that are deemed as robust by our empirical measure, giving us a more robust model. This happens as the model only learns using the robust images, which justifies inferring simple, more natural class separators. We claim that the images that are deemed as not robust by our method can generally be seen as edge cases, causing the model to infer irregular separators. 2.3
We experiment using a CNN architecture for classifying images from MNIST. As this classification task is not complex, we use a very simple model6 which achieves a test accuracy of only 98:85% to showcase the main ideas we introduce. The randomized image perturbation scheme we use is randomly altering a pixel count of at most the square root of the number of image pixels (28 in our case). We use MAX STEPS = 250 in our experiments.
We show in Figure 2 an image that is classified correctly by M against its random perturbation under the scheme described above, which is incorrectly classified by M.
We compare in Figure 1 robust and not robust images which, without any perturbation, are correctly classified by M. These were randomly chosen and give some intuition about what a robust image means compared to one that is not robust.
The process we described for determining the empirical robustness is very similar, when seen as a function of MAX STEPS, to a learning curve. Discovering images which are not robust eventually flattens, which allows us to use it together with some early-stopping mechanism.
In Figure 3 we can see how the ratio of test images that are still robust as a function of MAX STEPS flattens. We obtain a ratio of 0:2957 images from the test set which can withstand 250 random permutations, which is a surprisingly small fraction, considering the simple noising we apply. This stands as straightforward empirical evidence that the simple CNN architecture we used is not robust.
6We use two small convolutional layers, one max pooling layer, and a fully connected layer with softmax activation. We also train the model with ADAM using the default hyperparameters.
In order to achieve a more robust network, we apply the same procedure of deeming an image as robust or not robust on the MNIST train set, using the model M, which was trained on exactly this data. There are 71:28% images which are deemed as robust from the train set, however the distribution is not uniform at all as seen in Figure 4. Therefore, we randomly sample 1500 datapoints from each class of the robust training images, such that the training set does not have a class bias, and proceed to retrain the simple CNN architecture solely by using this data. Let the model trained on this data, which amounts for only 25% data from the MNIST training set, be MR. We encounter a drop of approximately 2% in the test accuracy, obtaining a 96:92% score, which is to be expected considering the relatively sparse training data we have trained on.
The model MR is much more robust on the test set, obtaining a ratio of 0:5101 robust images, cf. Figure 5, as compared to the robustness of the original M of only 0:2957. This stands as evidence that the robust nature of the selected training images led to a more robust model. 3
The simple empirical robustness checking method we introduce opens the way towards building fast, model-agnostic tools to estimate robustness of machine learning models. This method can be easily embedded as a base check in machine learning systems.
One of the main takeaways is that robustness can be seen as an inherent property of the images with respect to the classification task. The robustness of models depends both on their architecture and on the robustness of the data it is trained on. This can be exploited in various ways, such as the training methodology we proposed, which improves significantly the robustness of the model.
Some interesting other applications could include using Generative Adversarial Networks (GANs) to augment the robust training data from the training methodology we proposed. Data augmentation with GANs has successfully been used in improving the quality of data and accuracy of models [Antoniou et al., 2017] and we believe that it could be used to generate diverse robust images as well. These could contribute to increasing the accuracy of robust models trained under our methodology.
Another area of further investigation is checking how our empirical robustness measure relates with the formal verification tools that obtain exact robustness guarantees. Note that this kind of experiment is not possible for any model, as existing formal verification tools are limited to specific machine learning architectures or do not scale well with complex models.
Corinna database.