Vulnerability to adversarial attacks is a well-known weakness of Deep Neural Networks. While most of the studies focus on natural images with standardized benchmarks like ImageNet and CIFAR, little research has considered real-world applications, in particular in the medical domain. Our research shows that, contrary to previous claims, the robustness of chest x-ray classification is much harder to evaluate and leads to very diferent assessments based on the dataset, the architecture, and robustness metric. We argue that previous studies did not take into account the peculiarity of medical diagnosis, like the co-occurrence of diseases, the disagreement of labellers (domain experts), the threat model of the attacks, and the risk implications for each successful attack. In this paper, we discuss the methodological foundations, review the pitfalls and best practices, and suggest new methodological considerations for evaluating the robustness of chest xray classification models. Our evaluation of three datasets, seven models, and 18 diseases is the largest evaluation of the robustness of chest X-ray classification models.
tic discrepancies between practitioners. Medical errors,
Chest radiography (CXR) is an afordable, easy-to-use especially diagnostic errors, account for an additional
medical imaging and diagnostic technique. Chest radio- medical spending of $17 to $29 billion [
Although disease patterns may seem well defined, cor- products [
However, we argue that natural image classification box attacks on two binary neural networks (ResnetV2 setting and the medical classification setting are very and NasNet Large) using the ChestX-ray14 dataset [28]. diferent and require the evaluation of diferent threat They showed that both models are vulnerable to gradientmodels, robustness metrics, and hyperparameters. based attacks (100% success rate of attacks). While their
To uncover the inconsistencies between the two set- evaluation pioneered the research on adversarial attacks
tings, we provide the first large-scale study of the vul- in the medical setting, their evaluation focused on binary
nerability of chest radiograph classification to the best classification in a restricted setting.
of our knowledge. Furthermore, we introduce two novel Finlayson et al. [29] focused on binary image
classimethodological considerations to evaluate robustness ifcation for medical diagnosis. Their study covered the
in medical domains: cross-domain generalization and diagnosis of CXR, fundoscopy, and dermoscopy, and they
domain-specific knowledge. We argue that a rigorous also showed that PGD attacks achieved a 100% success
evaluation of the robustness of medical classifiers in gen- rate on the ChestX-ray14 pneumothorax label using one
eral and chest x-ray classifiers in particular needs to con- model.
sider these facets. Ma et al. [30] had another take on the robustness of
In summary, our contributions are as follows. CXR image classification models. They compared the
robustness of binary classification, 3-label, and 4-label
• We survey the literature on adversarial robustness classification. They showed that while PGD had a success
in chest x-ray classification and identify the major rate of over 99% on all of them, the vulnerability seems to
pitfalls and limitations. decrease with the increased number of labels. The three
• We propose a set of principles and recommenda- classifiers were trained on the ChestX-ray14 dataset, each
tions for how such pitfalls could be mitigated. time with a subset of labels. Our study covers the
com• We demonstrate the impact and criticality of the plete scenario of 18-label classifiers trained on diferent
principles through an empirical study of chest x- datasets (and distributions) and architectures. A
previray classification robustness using three datasets, ous study [11] showed essential performance diferences
seven models, and 18 diseases. between the diferent labels that can explain the slight
variation of robustness across the set of labels. Some
2. Related Work labels are already challenging to learn and, similarly, to
attack. Our evaluation, on the contrary, shows that the
Adversarial attacks An adversarial attack is the pro- variations between diferent datasets and architectures
cess of intentionally introducing perturbations to the are significant and that some models are actually resilient
input of a machine learning model to cause incorrect against adversarial attacks.
predictions. A family of adversarial attacks is poisoning
attacks [16] where the inputs targeted are the training set 3. Pitfalls and principles of chest
and occur during the learning step, while evasion attacks
[
Method (FGSM) [18]. It adds a small perturbation to 3.1. Medical images difer from natural the input of a neural network, which is defined as: images = sign(∇L(, , )), (1) Before we evaluate common practices, it is insightful to understand why using the experimental protocol of where are the parameters of the network, is the input adversarial attacks on natural image datasets is rarely data, is its associated target, L(, , ) is the loss relevant in the context of chest X-ray classification. function used, and the strength of the attack. Following The first consideration is the nature of the tasks and Goodfellow, other attacks were proposed, first by adding labels. In the ImageNet[31] and Cifar[32] classifications, iterations [19], projections and random restart [20], mo- images are designed to highlight one class (the ground mentum [21], adaptive steps [22] and constraints [23]. truth class) more than the others. Meanwhile, chest ra- 3.2. Literature review diographs are real images in which the same image can contain multiple diseases of equal importance. Chest
cation problem, and using metrics and losses specific to this field of machine learning can provide a more faithful representation of the robustness of the models.
ages and their probabilities are subjective to the
radiologists who provided the ground truths. Cohen et al. [
diferent impact depending on the nature of the error.
There are two risks in medical diagnosis: misses, that is,
when the classifier does not detect the correct disease
among the most probable classes and misinterpretations
when the most probable disease leads to an incorrect
diagnosis. The latter can have a diferent impact depending
on how similar the predicted labels are to the original
ones. Similarity can take into account the treatment
process: Confusing two diseases that, in the end, require
similar treatment is less detrimental than confusing two
diseases with diferent treatments. Similarity can also be
considered following disease taxonomy: Diseases that
belong to the same families/branches can be considered
more similar. The four-pattern approach commonly used
[33, 34] considers four families: Consolidation,
Interstitial, Nodules or masses, and Atelectasis. Within each
family, there is a wide range of diseases. Confusing a
disease from one family with one from another can be
detrimental. Some diseases often occur together [
that can afect the conclusions. First, the evaluation of binary classification (9 publications among the 16) leads to an overestimation of the robustness of the models. Indeed, attacking a multilabel classifier is much easier [49] as the decision boundaries are more blended than single-label classifications. Another risk arises when drawing conclusions about CXR classification from a single dataset only. All the publications we identified restrict their evaluation to the one CXR dataset. We demonstrate empirically that the conclusions about the robustness of a model difer significantly from one CXR dataset to another.
Threat models. The evaluation of the whitebox setting is relevant to understand the internals of the DNN model or to evaluate the worst-case scenario. However, in practice, access to the model and dataset of a specific hospital / physician is unrealistic: only five articles evaluated a more realistic setting, with at least the graybox attack scenario. Our results demonstrate that the conclusions can change when assessing realistic cases where the attacker only has access to the target dataset (graybox) or even no knowledge (blackbox).
lications about adversarial robustness in the medical setting, their work was an index of the literature and not a critical analysis of the protocol or the relevance and impact of the experimental designs.
surveys, we collected the publications that have been peer-reviewed related to CXR classification from 2018.
not suficiently evaluated, can lead to overestimated or even wrong claims. We summarize this literature in Table
robustness evaluation to a single CXR architecture. We demonstrate that the robustness of architectures can vary significantly with the threat model and the dataset under evaluation.
strated by Carlini et al. in multiple publications [50, 51, 52]. Since 2018, solid robustification protocols have been designed using adversarial training, and multiple repositories of robust models are available (Robustbench, for example, [53]). Unfortunately, only two publications
Robust models No
Attacks PGD, Patch
Metrics Accuracy, AUC Threat models Architectures Whitebox, Gray- Resnet50 box Whitebox Resnet50 Whitebox
Resnet50, VGG-16 Whitebox, Gray- Resnet, DenseNet, No box MobileNet Whitebox CovidNet Whitebox VGG16,
tionV3 Whitebox Resnet18 No Whitebox, Black- Resnet50 No box API Whitebox, Gray- Inception, NasNet- No box Large Whitebox VGG11 Whitebox InceptionV3 Whitebox, Gray- ResNet, box DenseNet Whitebox ResNet18, VGG16 Whitebox ResNet18, VGG16
No FGSM, CW, BIM, Accuracy, AUC
PGD No FGSM, B/MIM, Accuracy
PGD FGSM, CW, PGD, Success Rate
B/MIM, Custom Adversarial Re- FGSM, PGD Accuracy training Incep- No FGSM Accuracy
FGSM, PGD2 Accuracy FGSM, PGD, Loss DeepFool, +4 FGSM, PGD, Accuracy, AUC
DeepFool, + 6 Adversarial Train- FGSM, PGD AUC ing
No PGD VGG, Adversarial Re- FGSM, DeepFool training Custom denoiser FUIT Adversarial train Adv training
Success rate Success rate, confusion matrix FGSM, BIM, CW Accuracy FGSM, BIM, CW, Accuracy PGD
PGD, GAP Whitebox Whitebox
DenseNet-121 DenseNet-121
Success Rate, AUC, Accuracy
N/A Detection
FGSM, BIM, PGD 4 class NIH Binary Pneumonia Binary Pneumonia 3 class COVID Binary COVID 3 class COVID
Binary COVID Taghanaki et al. [13]
Binary NIH Anand et al. [42] Kovalev et al. [43] Hirano et al. [44] ([42, 47] considered strong defenses, and vfie others used classification in general and CXR image classification broken or weak defenses. in particular make these metrics irrelevant. First, some datasets are provided as multilabel datasets (NIH, for inStrong attacks. Fourteen publications investigated po- stance), and multiple diseases can occur together. Other tentially strong attacks (CW, PGD), and their evaluation datasets are built around the uncertainty of diagnosis used very few iterations and a limited perturbation bud- when the domain experts do not provide the same diagget. Although current good practices are to use robust nosis for a given input. CheXpert dataset, for instance, and adaptive attacks such as AutoAttack [22], we show has been designed with three labels’ values: positive (1), empirically that increasing PGD budgets already leads uncertain (-1) and negative (0). Finally, [11] have shown to surprising behaviors when comparing datasets and that 2 models trained for the same task on a diferent architectures. dataset have diferent degrees of agreement on the most probable labels and diagnosis. To take into account the uncertainty and co-occurrences of labels, we propose to use k-robust accuracy.
Evaluation attacks. We demonstrate that the success rate and accuracy of adversarial examples are misleading because of the nature of CXR classification. For example, the co-occurrence of pathologies and the risk associated with each type of error lead to alternative conclusions in the evaluation. We propose a new RISK metric to take into account the specificity of CXR classification.
Definition 1. Let M a multi-label model with labels L = {1, ..., }. M : ⊆ R →− ⊆ R . We have the size of the input features and the number of labels. For each input example , we denote by ¯ the corresponding ground truth and we have ¯= ( 1, ..., , ) where ∈ {0, 1} is the corresponding ground truth for
For each ∈ , let ˆ the predicted labels ˆ = M ().
In the traditional adversarial attack literature, we evalu- Then, we denote by ,M (, ¯) the k-accuracy of ate the robustness using the success rate of the attacks, the input for its top labels, and define it as the cardinal that is, 1-accuracy of the predictions over adversarial of the intersection between ’s top-k ground truth labels examples (generally called robust accuracy. The suc- and its top-k predicted labels: cess rate and robust accuracy have been used directly in the previous literature on adversarial CXR examples [54, 15, 14, 13]. We argue that the specificities of medical ,M (, ¯) = |((ˆ)|) ∩ ((¯) |)
where of a set are the indices of the top k evaluated the performance of the DensetNet121 architecelements of the set. ture and the Resnet512 architecture when trained using the AllD dataset.
For an input , ,M evaluates how much the Following similar work [11, 9], we adjust the training most probable predicted labels match the most proba- process to the CXR classification task. We account for the ble ground truth labels. This formalism is suitable for missing labels by training the models using only the loss both ordinal labels (to take into account uncertainty) and from the available labels. CXR classification also sufers multilabels (to take into account label cooccurrence). from a large imbalance in label distribution. We alleviate Definition 2. We define the k-accuracy of the model the imbalance with a frequency-based weight for each M as the expectation over the input set of the k-accuracy label. The less frequent labels have a higher contribution of the input ∈ : ,M = E [,M (, ¯)] to the loss computation. Finally, each label also has a diferent optimal binary threshold. Except to evaluate
For = 1, the k-accuracy matches the standard accu- the multi-label accuracy, we do not threshold the outputs racy. and use the raw probabilities. For multilabel accuracy, diferent thresholds are used for each label as proposed by Cohen et al.[11].
Datasets. Following the protocol set up by [11] we evaluated the robustness of CXR models using four datasets.
• NIH Chest X-ray14 [28], denoted as NIH in the following. A dataset of 112k images was automatically labeled with the NegBio labeler. This is the most common dataset used in the literature of CRX image classification. • CheXpert [55]. This dataset of 224k chest radiographs has been labeled with a custom automated labeler for NLP analysis of radiology reports. • PadChest [56] is a 160k image dataset. The labels are extracted from radiographic reports manually annotated by trained physicians for 27% of them. • A combination of the three is denoted AllD. We combine the images obtained from the three previous datasets for this dataset and process them as proposed in [11].
5120 inputs randomly sampled from the test set. Attacks. We evaluate the robustness of the models mainly against the PGD attack [20]. Madry et al. have shown that PGD is a universal surrogate for first-order gradient attacks, and robustness against PGD attacks is a common metric to evaluate the robustness of models [53]. It is also the one used in previous research on the robustness of CRX models [12, 14].
{0.5/255, 1/255, 2/255, 4/255, 8/255}, and the number of attack steps in the range of {1, 5, 10, 25, 50}. Robustness evaluation metrics. In addition to the k-robust accuracy, we also evaluate the robustness of the models using traditional error metrics, to cover metrics designed specifically for multilabel classification and ordinal classification: The mean square error (MSE), crossentropy error (BCE), multi-label accuracy (MLACC) [57] and the Ordinal classification loss (OL) [58].
cover the maximum number of labels of our evaluation, 5.1. Cross-domain generalization even if the dataset on which the model has been trained is missing one or a few labels. This allows us to train and test each model on any other dataset. All our models have an average AUC > 0.79.
models using a DenseNet-121 architecture available in the TorchXrayvision library [11]. It includes models trained on NIH, CheXpert (CHEX) and PadChest (PC).
datasets that share the same labels as the AllD dataset.
ferent architectures trained using the same dataset. We To better understand how adversarial attacks impact CXR classification models, we evaluate the impact of the training data on the robustness of models, in particular for transfer attacks, when the source model and the target models are diferent. Given a PGD attack of = 1/255 and 25 steps, we evaluated the robust precision k for = 1 and = 3 for our six DensetNet121 models M1,
Adversarial attacks transferability: Results are shown in table 2. When restricted to the 1-robust accuracy, the NIH model is the most robust model (15.4% D2 k=1 k=3
Target → Source ↓
NIH CHEX
PC MIMIC RSNA AllD NIH CHEX
PC MIMIC RSNA AllD robust accuracy on average) and the CHEX model is the 4.78 5.56 8.62 32.125 19.09 37.69 most vulnerable. When we evaluate the 3-robust accu- 3.31 2.56 11.44 22.25 30.91 39.31 racy, the most robust model becomes the AllD model (35.6% robust accuracy on average). This confirms not Tk-arbolbeu4st accuracy on 2 architectures: Resnet50 and only that diferent models have a wide range of robust- Densenet121. The source and target models for the attacks ness (NIH is ten times more robust than CHEX), but are the same. The columns are the datasets from which the exprevious claims that the PGD attack on the CXR clas- ample are sampled, and the rows are the source/target models. sification yields a 100% success rate are far from true. Greyed cells are best the values across a row, and underlined Taking into account the dataset used for model training cells are the best values across a column. can yield a significant diference in robustness.
from top1 to top3 shows that, actually, the models, in lead to higher robust accuracy. general, remain robust enough, and the correct labels are When considering the 3-robust accuracy, D3 is the still predicted with high probabilities. The only excep- more robust dataset for four models among the six. When tion is the CHEX model, which remains very vulnerable. the train examples and the evaluation example are both It suggests that the distribution that has been learned sampled from the PadChest dataset, the 3-robust accuracy by this model can be significantly impacted by a small peaks at 53.59%, more than three times the robustness of perturbation. the NIH model on the same dataset.
Additionally, the most robust model in the whitebox threat model (the diagonal values where the source and Impact of architecture: We observe in Table 4 that target model are the same) is not the same given the 1- diferent architectures are not reliably robust on diferent robust accuracy or the 3-robust accuracy. While the NIH CXR datasets. Although Resnet is more robust on D3, model preserves the most probable class, the PC model the DenseNet model has a higher robust accuracy on retains better the correct labels in the top3 predictions. D1. It is also noted that across both architectures, the dataset D3 yields the highest robust accuracy across both architectures. It is consistent with our previous results (Table 3) that also showed that the input from D3 is more robust across diferent models of the same architecture.
Robustness over diferent test datasets: Next, we explore the impact of the test dataset on the robustness of models. We evaluate in Table 3 the k-robust accuracy of each model when the original inputs are sampled from one of the datasets: D1 for NIH Chest X-ray14, D2 for the CheXpert dataset, D3 for the PadChest dataset. The source and target models are the same in this setting.
the model trained on the NIH dataset. There is also no relationship between the robustness of the model and the distribution of inputs. Sampling the inputs for the adversarial examples from the same distribution (NIH with D1, CHEX with D2, PC with D3) does not reliably Impact attack budget and : For = 3, the robust precision drops from 34.56% with = 0.5/255 to 13% with = 4/255. Meanwhile, the robust accuracy of k for = 1 increases slightly with increased attack budget, from 3.84% to 8.06%. This increase in robustness is unexpected and can indicate that iteration budgets of 25 steps are not suficient to efectively explore such a large search space.
Given a perturbation budget of = 0.5/255, the number of steps has a limited impact on the robust matter. They reflect the risk caused by the misclassification. This risk can be computed as a vectorial product between the predicted logits of the adversarial example and the target logits computed with . And we have: = M (* ) × (ˆ* ).
tions: MSE, BCE, OL. We report for each approach the
k-robust accuracy (with = 1, 3) to compare this threat model with the threat model of 5.1. While MSE, BCE, AUC, and k-robust accuracy reflect how much error we introduce compared to the original prediction, MLACC and RISK reflect how close the predicted output is to the target output. We bring together the results of all these evaluations in Table 5. accuracy. We observe that the attack success (and thus the models’ robustness) plateaus around 30% for all the multi-step attacks: 5, 10, 25, and 50 steps.
varies significantly when considering architectures and datasets. Contrary to common practice, mixing multiple datasets leads to less robust models.
CXR classification not only raises questions about the generalization of one’s hypothesis about the robustness of models, as we showed, but it also requires a higher understanding of the labels and diseases that we aim to classify. When dealing with a critical task, such as medical diagnosis, the risk associated with a prediction error Impact of the loss function The most robust models can increase dramatically when the predicted diseases are overall consistent across diferent loss functions used are far from the actual truth. We show that targeted ad- in the attack. This confirms that handling risk-based versarial attacks against these risky labels provide a new attacks as an ordinal classification problem are as relevant view of the robustness of CXR classification models. as a multi-label problem for the success of the attack.
matrix provided by the multi-label dataset NIH. In this Impact of the threat model Comparing the k-robust dataset, each radiograph can have 1, 2 or 3 diseases that accuracy of our risk-based threat model with the unhave been annotated. This matrix indicates which combi- targeted threat model of our previous results (Table 2) nation of diseases are very rare in practice and hence can shows that this risk-based threat model produces more hardly be confused. For instance, while Infiltration and successful attacks and therefore lower robustness of the At{iot1eLn,lese.t,.c.I,tnMafilstirsa}t.aairoMenmtauw:nlotdi-Pll⊆aanbbeReeullsmcmo→−otohmdomeral⊆oxnwalyriteRhfsoculaa.nrbcdeelistnogaLentnhoetr=a.- 2Nm)I,oFHbdouermltseoi.xtdademrlopapglsea,tinwosi2tt.h6u9n%ta=frog1re,rtetishdke-abrtotaabsceukdsstaiptstra1ec3cki.7ssi8o(%Tna(obTflaetbh5lee).
For each ∈ , let ˆ the predicted labels ˆ = Similarly, for k=3, the AllD model has a robust accuracy M () = (ˆ1, ˆ..., ˆ ) where ˆ ∈ R is the predicted of 32.12% against untargeted attacks and only 11.12% probability of the label . Let ˆ* the most probable label against risk-based attacks. for : ˆ* = arg max{ˆ1, ˆ..., ˆ }. Let the normalized inverse co-occurrence matrix of the label space .
A higher value in means that the labels of the row Risk evaluation of the models According to the RISK and column indices are very unlikely to occur together. metric, the NIH model is not only the most robust to () is the vector of improbable labels associated with adversarial attack, but also the one where the end lathe label . bels have the lowest probabilities to actually be rare co
For each input ∈ , we generate an adversarial occurring labels of the original label. * ∈ example with targeted Projected Gradient Descent (PGD) [20] algorithm, targeted on the improbable label vector of . The targeted PGD adds iteratively a perturbation that opposes the sign of the gradient ∇ with respect to the input x and the target (ˆ* ). Π is a clip function that ensures that + respects a - perturbation budget: Impact of the robustness metric Our results show that the error metrics fail to highlight one specific model as being the most robust. According to robust accuracy and robust AUC, NIH is the most robust model across diferent loss functions. Meanwhile, the RSNA model is the most robust according to the BCE and MSE losses.
0 = ; +1 = Π + (− (∇(, , (ˆ* )))) robustness values of each batch of all combined models.
This optimization can be seen as a weighted multil- robust accuracy, the p-value is under 10ˆ-3 . Our results abel classification attack because the target vector is a show that none of the existing metrics (MSE, BCE,...) real-valued vector, or as an ordinal classification attack is correlated with the RISK metric. This confirms that because the order of the values of target logits actually existing metrics do not take this dimension into account. 3.07 13.31 0.74 0.07 0.78 0.66 0.18 Loss
AllD 0 0 0.5 0.02 0.74 0.45 0.24 17), 2017. [52] N. Carlini, A. Athalye, N. Papernot, W. Brendel, J. Rauber,
D. Tsipras, I. Goodfellow, A. Madry, A. Kurakin, On evaluating adversarial robustness, 2019. arXiv:1902.06705. [53] F. Croce, M. Andriushchenko, V. Sehwag, E. Debenedetti,
N. Flammarion, M. Chiang, P. Mittal, M. Hein, Robustbench: a standardized adversarial robustness benchmark, arXiv preprint arXiv:2010.09670 (2020). [54] M. Paschali, S. Conjeti, F. Navarro, N. Navab, Generalizability vs. robustness: Adversarial examples for medical imaging, 2018.
arXiv:1804.00504. [55] J. Irvin, P. Rajpurkar, M. Ko, Y. Yu, S. Ciurea-Ilcus, C. Chute,
H. Marklund, B. Haghgoo, R. Ball, K. Shpanskaya, J. Seekins, D. A. Mong, S. S. Halabi, J. K. Sandberg, R. Jones, D. B. Larson, C. P. Langlotz, B. N. Patel, M. P. Lungren, A. Y. Ng, Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison, 2019. arXiv:1901.07031. [56] A. Bustos, A. Pertusa, J.-M. Salinas, M. de la Iglesia-Vayá, Padchest: A large chest x-ray image dataset with multi-label annotated reports, Medical Image Analysis 66 (2020) 101797. URL: http://dx.doi.org/10.1016/j.media.2020.101797. doi:10.1016/ j.media.2020.101797. [57] Q. Song, H. Jin, X. Huang, X. Hu, Multi-label adversarial per
turbations, 2019. arXiv:1901.00546. [58] E. Frank, M. Hall, A simple approach to ordinal classification, in: European conference on machine learning, Springer, 2001, pp. 145–156.