Robustness is a fundamental pillar of Machine Learning (ML) classifiers, substantially determining their reliability. Methods for assessing classifier robustness are therefore essential. In this work, we address the challenge of evaluating corruption robustness in a way that allows comparability and interpretability on a given dataset. We propose a test data augmentation method that uses a robustness distance ε derived from the datasets minimal class separation distance. The resulting MSCR (mean statistical corruption robustness) metric allows a dataset-specific comparison of diferent classifiers with respect to their corruption robustness. The MSCR value is interpretable, as it represents the classifiers avoidable loss of accuracy due to statistical corruptions. On 2D and image data, we show that the metric reflects diferent levels of classifier robustness. Furthermore, we observe unexpected optima in classifiers robust accuracy through training and testing classifiers with diferent levels of noise. While researchers have frequently reported on a significant tradeof on accuracy when training robust models, we strengthen the view that a tradeof between accuracy and corruption robustness is not inherent. Our results indicate that robustness training through simple data augmentation can already slightly improve accuracy.
ML functions are deployed to an increasing extent which demands a risk assessment1.
Within the European domestic market, machinery products are subject to regulation of the Machinery directive,
Risk assessment includes risk estimation and evaluation, where risk is defined as a combination of probability and severity of a hazardous event. Therefore, once ML functions are deployed in machinery products, where their failure may lead to a hazardous event, being able to quantify the probability and severity of their failures becomes mandatory. However, there still exists a gap between the regulative and normative requirements for safety critical software and the existing methods to assess
This work targets ML classifiers, the failures of which are misclassifications. Our focus is on the evaluation of failure probability specifically, not on failure severity. We address one specific failure mode of ML classifiers: Corrupted or perturbed data inputs that cause a change The IJCAI-ECAI-22 Workshop on Artificial Intelligence Safety (AISafety https://github.com/georgsiedel/ minimal-separation-corruption-robustness (G. Siedel) © 2022 Copyright for this paper by its authors. Use permitted under Creative Commons License CEUR htp:/ceur-ws.org ISN1613-073
CEUR
Workshop Proceedings (CEUR-WS.org) at a point within a distance > 0 , if () = ( for all perturbed points
that satisfy ( −
′
[
-function can e.g. be an -norm distance, while can be defined based on physical observations of e.g. which perturbations are imperceptible for humans.
Robustness is considered a desirable property since intuitively, a slightly perturbed input (e.g. an imperceptibly changed image) should not lead to a classifier changing its corresponding prediction. In essence, a robustness requirement demands that within a certain input parameter space around , all points have to share the same class. This way, a robustness requirement adds additional information on how the classifier should behave near ground truth data points. Authors therefore argue the importance of robustness, being a fundamental pillar ′ of reliability [4] and quality [5] of ML models.
However, popular robustness training methods show significantly lowered test accuracy compared to standard training, which has lead to some authors discussing an inherent, i.e. inevitable tradeof between accuracy and robustness (see Section 2.2).
Two types of robustness need to be clearly distinguished [5, 6, 7]: adversarial robustness and corruption robustness. 2Robustness includes resistance to any corruption-caused class change, which may not be a failure mode when the original point was already misclassified (cf. footnote 4).
Adversarial inputs are perturbed data deliberately optimized to fool a classifier into changing its output class. Corruption robustness (sometimes: statistical robustness) describes a model’s output stability not against such worst-case, but against statistically distributed input corruptions. The two types of robustness require diferent training methods and are diferently hard to achieve depending on the data dimension [6]. In practice, training a model for one of the two robustness types only shows limited or selective improvement for the other type [7, 8, 9].
In the field of research towards ML robustness, most of the attention has been given to adversarial attack and defense methods. However, from the perspective of machinery safety and risk assessment, adversarial robustness is mainly a security concern and therefore not in the scope of this article. [10] argue that instead of adversarial robustness evaluation, a corruption robustness evaluation is often more applicable to obtain a real-world robustness measure and it can be used to estimate a probability of failure on potentially perturbed inputs for the overall system.
Contribution: In this paper, we investigate corruption robustness using data augmentation for testing and training3. Our key contributions are twofold: • We propose the ” ” metric to evaluate and compare classifiers corruption robustness. The approach is independent of prior knowledge about corruption distances, but utilizes properties of the underlying dataset, giving the metric a distinct interpretable meaning. We show experimentally, that the metric captures diferent levels of classifier corruption robustness. • We evaluate the tradeof between accuracy and robustness from the perspective of corruption robustness and present arguments against the tradeof being inherent.
After giving an overview of related work, we present our approach for the MSCR metric in section 3.1. We then test our approach on simple 2D as well as image data with the setup described in section 3.2. We present and discuss the results in sections 4 and 5.
Corruption robustness of classifiers can be numerically evaluated by testing the ratio of correctly/incorrectly classified inputs from a corrupted test dataset. This ratio is called robust accuracy/error, in contrast to the ratio 3Code available on Github, see front page. of correct classification on original test data (“clean accuracy/error”). Robust accuracy represents a combined measure for accuracy and robustness4. A useful way to obtain a measure of robustness only is by subtracting robust accuracy/error and clean accuracy/error [8, 11].
In most cases, the corrupted test dataset is derived from an original test dataset through data augmentation. One or multiple corruptions out of some distribution are added to every original data points. Figure 1 explains this procedure of data augmentation with corruptions (dots) being added to a test dataset (stars) with 2 parameters and 2 classes. It illustrates how a 100% accurate but nonrobust classifier achieves lower robust accuracy on the augmented data points.
The corruption distribution can be defined e.g. based on physical observations. For the example of image data, [8, 12] add corruptions like brightness, blur and contrast, while [13, 14] use special weather or sensor corruptions. [8] created robustness benchmarks for the most popular image datasets based on such physical corruptions.
Corruption distributions can also be defined without physical representations by adding e.g. Gaussian-, saltand-pepper-, or uniformly distributed noise of certain magnitude to the inputs [8, 11, 14, 10]. Figure 1 exemplary demonstrates uniformly distributed noise within 2-norm distance (in 2D, 2-norm is a circle) of the data points.
With PROVEN, [15] propose a framework that uses
statistical data augmentation to estimate bounds on
ad4The term astuteness can be used for robust accuracy to diferentiate
the term from robustness, see [
[4] take a robustness evaluation approach diferent
from measuring robust accuracy. The authors augment
the entire input space with uniformly distributed data
points, independent of a test dataset. They divide the
input space into cells, the size of which is based on the
r-separation distance described in [
An analytical way of measuring the robustness of a
classifier is through describing the characteristics of its
decision boundary. One possibility is to estimate the
local Lipschitzness, i.e. a tightened continuity property
of models in proximity to data points. To the best of our
knowledge however, Lipschitzness has only been used to
investigate adversarial, not corruption robustness [
Both the measure in [4] and Lipschitzness values lack distinct interpretability in terms of what the calculated value represents exactly. 2.2. The Accuracy-Robustness-Tradeof
Significant efort has recently been put into increasing classifier robustness, commonly targeting adversarial ro- holds, where is the corruption distance for which robustness, e.g. in [9, 16, 17, 18, 19]. All these methods bustness is evaluated and is half this minimal class sepcause a significant drop in clean accuracy. aration distance. We adopt this notation and set = [11, 20] and [10] observe a clear tradeof between as our corner case corruption distance (see Figure 2). The corruption robustness and accuracy for diferent train- value is not related to any prior physical knowledge ing methods using data augmentation. The two former of e.g. which corruptions are imperceptible, but is specific works then propose specialized training methods for miti- for the given dataset, i.e. it is based on the fundamental gating parts of this tradeof on the popular image datasets property of minimal class separation. Accordingly, we CIFAR-10 and ImageNet. call our metric “Minimal Separation Corruption Robust
Based on such research, [19] and [21] discuss a tradeof ness” (MSCR).
between accuracy and robustness, while [22] even argue
that the cause for this tradeof is inherent, i.e. inevitable. 3.1. MSCR metric
A counterargument is presented by [
To measure corruption robustness, we carry out data
augmentation on the test data with uniformly distributed
corruptions, generated by a random sampling algorithm,
similar to the method shown by [10]. In contrast to [10],
we set the upper bound of the distance , within which
the augmented noise is distributed, to , as required in
Equation 1 (see Figure 2 for an illustration). We measure
robust accuracy on the augmented data, which
corresponds to a combination of clean accuracy and corruption
robustness. However, we want to quantify robustness
independent of clean accuracy for comparability, so we
subtract the clean accuracy ( ) from the robust
accuracy on -augmented test data ( − ) and
normalize by the clean accuracy:
Our robustness evaluation approach is based on this same
idea by [
while maintaincircles of radius
in which data is augmented, never overlap for diferent classes. We use an identical radius
for all classes, assuming that the separation of data points from the classifiers decision boundary is equally important for all classes. For this noise level non-robust behavior is theoretically avoidable, since a
, any classifiers decision boundary can separate the classes even with augmented data, as long as the ML algorithm is capable of learning the exact function. The MSCR metric therefore measures the (relative) win or loss in accuracy when testing on such noisy data that any loss is just about avoidable. Figure 2 illustrates the impact of the proposed metric using three corner cases: =
, solid line in −
= ( = (∑=1 −
− )/ Test model with original test data ( = 0) → return
sample points within ( augment the test data
Test model with data from step 6 → return
) and )/ sult to be possible.
that trained to be very robust, we expect this re- the work of [10], we extensively evaluate more corrupAlgorithm 1 shows the MSCR calculation procedure. In step 1, diferent distance functions (e.g. ∞-norm) can be applied. We account for randomness in the data splitting, model training and data augmentation procedures by carrying out multiple runs of the same experiment and reporting average values and 95%-confidence intervals over all runs. The reasonable number of augmented points k per original data point varies depending on the dataset (see section 3.2). Within the respective for-loop, variable
runs through the list of all classifier models to be compared, while counts up to (the overall number of)
.
Additionally to test data augmentation, we train multiple models on datasets augmented with diferent corruption distances
. Increasing a model’s a growing MSCR value, as it is expected that the model robustness grows. This way, we evaluate the trend of the MSCR value for models with diferent corruption robustness levels. Also, on test data corrupted with large should lead to =
are expected to perform best [10]. , models trained with
As demonstrated in Figure 2, corruption levels below
theoretically allow a classifier to be robust while not
losing test accuracy. We investigate this theoretical claim
by [
specifically. tion levels below, around and including In contrast to the work of [11] and [20], we use simple uniformly distributed data augmentation with a fixed upper bound of noise for the entire dataset instead of Gaussian noise. This allows us the comparison of the noise levels with the class separation distances. It shall be noted however that in contrast to Gaussian noise, where density decreases with distance, uniform noise does not reflect the higher uncertainty in a class assignment when the distance from a ground truth data point increases.
Even though our data augmentation method is simple,
we still expect to find counterexamples for the
accuracyrobustness-tradeof, based solely on the class-separation
theory. We believe that the case of finding such
counterexamples with less advanced methods than e.g. [11]
represents even more credible evidence for the argument
of [
We carry out the experiments on 3 binary class 2D datasets as were used and provided by [4]. For clarity, we only report results with ∞-corruptions on one of those datasets, which is shown in Figure 3 and features 4674 data points. Experiments with the other 2D datasets and 2-corruptions exhibit similar fundamental results, which can also be found in our Github repository (see
For both datasets it is visible from the last rows of
frontpage). The two input parameters [0] and [
We choose = 10 augmented data points per original data point, as we found higher numbers of not significantly improving the resulting robust accuracy and its standard deviation. This efect of diferent values for the hyperparameter is displayed in Figure 4. In order to achieve statistically representative results, we evaluate how the average test accuracy converges over multiple runs and accordingly choose 1200 runs.
The experiments are additionally run in a more applied
image classification setting using benchmark dataset
CIFAR-10. We adopt the classifier architecture from [ 10],
using a 28-10 wide residual network with SGD optimizer,
0.3 dropout rate, training batch size 32 and 30 epochs with
a 3-step decreasing learning rate. All pixel values are
normalized to [
values, measured in ∞-distance for both datasets. For intuition, the CIFAR-10 translates to a maximum color grade change of 27/255 on all pixels. Higher values for 2 are to be expected for image data, since ∞-norm evaluates the maximum distance in any of the 3072 dimensions of CIFAR-10 input
value data. crops with 4px padding are used for training generaliza- levels, the optimum robust accuracies are achieved with Minimal ∞ class separation and corresponding
Dataset 2D dataset CIFAR-10 (train and test set) 2 ( ∞)
Clean accuracies (first row) and robust accuracies in percentage plus the MSCR value (last row) for various models (columns) ± the 95% confidence intervals. Models are trained and tested with diferent levels of ∞-noise ( along columns, along rows). Bold accuracies: Best model accuracy for every noise level. Bold MSCR value: Highest MSCR value, i.e. highest model robustness. Last row color scale: Highlights the constant increase of MSCR with increasing trained and tested on the same noise level ( = ). Dark grey accuracies: Maximum overall accuracy.
. Light grey accuracies: Model higghreeyralcecvueralcsies: Model trained and tested. oFnotrhebsoatmhe dnoaitsaesleevtse,l ( ac curac y- r)o. bDuasrktngeresys-atcrcaudreacoif efso:rMthaxeimmuomdeolvsertarlal iancecudrawcyit.h of training noise = the MSCR increases from negative values on less robust by contrasting MSCR versus clean accuracy trained models to zero and even positive values for more values. Both Figures in principle show a tradeof curve. robust trained models.
For CIFAR-10, the MSCR values are overall much larger than for the 2D data. This efect correlates with the noise level, which is about 26 times larger in absolute
However, it is visible that for
≤ 0.007 on 2D data ≤ 0.01 on CIFAR-10, both clean accuracy and robustness increase compared to the baseline model with
= 0. The tradeof is overcome for these models diferent and values. ative diference that we named MSCR steadily increases in particular, this relClean Accuracy RF Clean Accuracy 1NN
εtrain (a) Clean Accuracy 2D Dataset
0,02 Model with εtrain = Baseline Model 98%
99%
Clean Accuracy (a) 2D Dataset 0,015
0.007 & 0.01 εmin 0 0,002 0,001 100% 1NN vs. RF over training noise on 2D Dataset 1NN vs. RF over training noise on 2D Dataset 0,10 0,05 CR0,05 S , compared to = 0. Models with both higher MSCR and higher clean accuracy (when the curve evolves towards with higher corruption robustness of the RF model on 2D herently robust model such as 1NN, which fits its decidata and the wide residual network on CIFAR-10. This sion boundary based on maximum class separation. The way, we verify the metric’s capability to reflect the corMSCR values are able to correctly display this interrelaruption robustness of diferent models. However, this tion. claim is based on the assumption that increasing corruption robustness of our models can be generated through training with higher noise levels. This seems evident based on research by [10], but requires future validation like in [11], who confirm that their Gaussian robustness metric is strongly correlated with the popular physical corruptions benchmark by [8].
On the 2D dataset, the 1NN model shows a constant, superior MSCR value compared to the RF model for all
In our experiments, the steady robustness increase for higher than as the underlying robustness requirement, is only one particular case of this robustness calculation approach.
. The MSCR value, which uses
-corruptions also holds for other levels of testing noise
≤ 0.007, where classes are still predominantly sep- It has to be emphasized that from our results in Tables 2 arated. This is the performance expected from an in- and 3, we cannot observe any conspicuities for ∼ . For example, there is no indication that models perform to = 0 does not achieve 95%-confidence in a pairwell below this noise level while massively dropping of wise statistical comparison. More than 20 runs are necesat higher noise levels, as could be presumed from the sary to obtain statistically significant results, which we r-separation theory. It is therefore evident to conclude could not achieve due to limited computational resources. that measuring corruption robustness works with other Hence, we only treat our results on CIFAR-10 regarding -values. In practice, if specific corruptions are known for an application, those corruptions should also be used for testing, e.g. through benchmarks [8]. the accuracy-robustness-tradeof as suggestions.
The suggestion that some
> 0 leads to higher clean accuracy than = 0 has theoretical relevance.
However, we emphasize that the MSCR metric is ad- It supports the claim made, but not practically proven by
vantageous in two ways: First, it does not require prior
[
The result also seems relevant from a practical perspective, since developers may try some for training Second, the MSCR can be interpreted with a clear contexdata augmentation, which increases robustness without tual meaning, since the robustness requirement is derived drawbacks regarding accuracy. We emphasize that this from the dataset: It measures “the theoretically avoidable practical implication is only valid for the very limited loss (or win) of accuracy due to statistical corruptions”. model architectures, datasets and augmentation distribu
Clearly, avoiding any loss of accuracy on hard to achieve in practice on high-dimensional data. For -noise is CIFAR-10, = 0
can be achieved, but only with
= 0.07, where the clean accuracy declines by 3 per- conjecture that > 0 centage points compared to = 0. We also verify our is possible for some robust trained models. For this behavior, we find the discovery in [23] a convincing technical explanation. Misclassified data points tend to lie closer to the decision boundary than correctly classified data points. The data augmentations on a misclassified data point therefore have a high chance of causing a favorable class change. At the same time, data augmentations on correctly classified points have a lower chance of causing an unfavorable class change when their distance to the decision boundary is high, which is what a robust model is trained for.
Besides our investigation of the MSCR metric, we report on findings regarding the tradeof between accuracy and corruption robustness. For both 2D and CIFAR-10 datasets we observe higher clean and robust accuracy on any test noise when training a model with a specific level of uniform noise ( = 0.007 for 2D, for CIFAR-10), compared to standard training. For the = 0.01 2D data, this optimum value is even higher than
, the value which the r-separation theory suggests to be beneficial for robustness while not hurting accuracy. This could be due to the major proportion of minimal distances of data points to other classes being significantly bigger than
. Our results are statistically significant for the 2D dataset experiment. For 20 runs per trained model on CIFAR-10, we emphasize that claiming higher mean clean accuracy for any > 0 compared tions we tested. For example, our experiments show that noise training below
has no efect on an inherently robust model such as 1NN. This is due to the fact that this model type maximizes the class separation of its decision boundary in training anyways.
On the one hand, overcoming the tradeof for small is not entirely surprising, since it is well known that data transformations and data augmentations can increase generalization of models (in fact, we also used random flips and crops for CIFAR-10 training). [ 11] and [20] also manage to overcome the tradeof with more advanced training methods. On the other hand, our results are surprising considering this drawback-free increase in robust accuracy is quite significant for the RF model on 2D data (less than halving the classification error). Also, uniform ∞ data augmentation is a very simple method and less contextually relevant compared to physically derived augmentations. An explanation may be that the uniform -norm noise allows a stricter coverage of the input parameter space near data points compared to physical data augmentations, enforcing a smooth model that is less prone to overfitting the corruptions.
From our results we also need to conclude that in pracvalue has only limited expressiveness when tice, the trying to find the optimal accuracy. This is visible in Figures 6a and 6b, where based solely on the r-separation theory, we may have expected the curve to reverse its trend along the x-axis with regards to (robust) when levels their high ∞ class separation distance . In reality, the best overall accuracy ∼ 2 ∗
, while on < /5 . We suspect inevitably entails. We suspect that on other datasets may be even greater and further away from the optimum 5.6. Optima of
vs. Another interesting finding from the accuracy matrix of both datasets is that the best value for models evaluated with certain diagonal. For example, to prepare for deviates from the expected = 0.03 is not the best choice = 0.03. In Figure 7, the accuracy matrix for CIFAR-10 from Table 3 is visualized in a 3D plot, which shows how the optima in (robust) accuracy deviate from the diagonal. It appears that for low noise noise levels
< levels the best choice is is more favorable. This suspected > , while for higher dependency needs further investigation.
In this article we evaluated a data augmentation method in order to obtain a comparable, interpretable measure of corruption robustness for classifiers. We measured the relative diference between the robust accuracy on corrupted test data and the clean accuracy. We proposed to use half the minimal class separation distance measured from the dataset as the maximum distance of the augmented test noise. This robustness requirement does not presume any prior knowledge about real corruption distances. It theoretically allows a classifier to be fully robust while not losing accuracy. The class separation distance therefore gives our metric a distinct meaning: It represents any “avoidable” loss (or win) in accuracy due to corruptions. We experimentally showed that our metric is able to reflect various degrees of model
However, the minimal class separation distance could in practice not guide us towards the optimal values of training noise. These findings regarding the accuracyrobustness-tradeof could in our opinion be useful in practice. training procedures mitigating the accuracy-robustnesstradeof, since the combination of both properties is essential from a risk assessment perspective. learning, Risk, and Reliability Analysis; Research Posters,
doi:1 0 . 1 1 1 5 / I M E C E 2 0 2 1 - 6 9 3 9 0 . [4] X. Zhao, W. Huang, V. Bharti, Y. Dong, V. Cox, P. S. Liang, Unlabeled data improves adversarial A. Banks, S. Wang, S. Schewe, X. Huang, Reliability robustness, Advances in neural information proassessment and safety arguments for machine learn- cessing systems 32 (2019). ing components in assuring learning-enabled au- [17] J. Cohen, E. Rosenfeld, Z. Kolter, Certified advertonomous systems, arXiv preprint arXiv:2112.00646 sarial robustness via randomized smoothing, in: In(2021). ternational Conference on Machine Learning, 2019, [5] Deutsches Institut für Normung, Din spec 92001- pp. 1310–1320.
2: Artificial intelligence – life cycle processes and [18] A. Madry, A. Makelov, L. Schmidt, D. Tsipras, quality requirements: Part 2: Robustness, 2020. A. Vladu, Towards deep learning models resistant [6] A. Fawzi, O. Fawzi, P. Frossard, Analysis of classi- to adversarial attacks, International Conference on ifers’ robustness to adversarial perturbations, Ma- Learning Representations (ICLR) (2018). chine learning 107 (2018) 481–508. [19] H. Zhang, Y. Yu, J. Jiao, E. Xing, L. El Ghaoui, M. Jor[7] J. Gilmer, N. Ford, N. Carlini, E. Cubuk, Adversarial dan, Theoretically principled trade-of between examples are a natural consequence of test error robustness and accuracy, in: International Conferin noise, in: International Conference on Machine ence on Machine Learning, 2019, pp. 7472–7482.
Learning, 2019, pp. 2280–2289. [20] D. Hendrycks, N. Mu, E. D. Cubuk, B. Zoph, [8] D. Hendrycks, T. Dietterich, Benchmarking neural J. Gilmer, B. Lakshminarayanan, Augmix: A simple network robustness to common corruptions and data processing method to improve robustness and perturbations, International Conference on Learn- uncertainty, International Conference on Learning ing Representations (ICLR) (2019). Representations (ICLR) (2020). [9] E. Rusak, L. Schott, R. S. Zimmermann, J. Bitterwolf, [21] A. Raghunathan, S. M. Xie, F. Yang, J. Duchi, O. Bringmann, M. Bethge, W. Brendel, A simple P. Liang, Understanding and mitigating the tradeway to make neural networks robust against diverse of between robustness and accuracy, International image corruptions, in: European Conference on Conference on Machine Learning (ICML) (2020).
Computer Vision, 2020, pp. 53–69. [22] D. Tsipras, S. Santurkar, L. Engstrom, A. Turner, [10] B. Wang, S. Webb, T. Rainforth, Statistically robust A. Madry, Robustness may be at odds with accuneural network classification, in: Uncertainty in racy, International Conference on Learning RepreArtificial Intelligence (UAI), 2021, pp. 1735–1745. sentations (ICLR) (2019). [11] R. G. Lopes, D. Yin, B. Poole, J. Gilmer, E. D. Cubuk, [23] D. Mickisch, F. Assion, F. Greßner, W. Günther, Improving robustness without sacrificing accuracy M. Motta, Understanding the decision boundary of with patch gaussian augmentation, arXiv preprint deep neural networks: An empirical study, arXiv arXiv:1906.02611 (2019). preprint arXiv:2002.01810 (2020). [12] C. Paterson, H. Wu, J. Grese, R. Calinescu, C. S.
Pasareanu, C. Barrett, Deepcert: Verification of contextually relevant robustness for neural network image classifiers, in: International Conference on Computer Safety, Reliability, and Security, 2021, pp.
3–17. [13] O. Molokovich, A. Morozov, N. Yusupova, K. Janschek, Evaluation of graphic data corruptions impact on artificial intelligence applications, in: IOP Conference Series: Materials Science and Engineering, volume 1069, 2021, p. 012010. [14] P. Schwerdtner, F. Greßner, N. Kapoor, F. Assion,
R. Sass, W. Günther, F. Hüger, P. Schlicht, Risk assessment for machine learning models, NeurIPS 2020 Virtual Workshop: Machine Learning for Autonomous Driving (2020). URL: https://arxiv.org/ pdf/2011.04328.pdf. [15] L. Weng, P.-Y. Chen, L. Nguyen, M. Squillante,
A. Boopathy, I. Oseledets, L. Daniel, Proven: Verifying robustness of neural networks with a probabilistic approach, in: International Conference on
Machine Learning, 2019, pp. 6727–6736. [16] Y. Carmon, A. Raghunathan, L. Schmidt, J. C. Duchi,