Deep learning models have achieved state-of-the-art performance in the interpretation of geoscience and remote sensing data. However, their vulnerability to adversarial attacks should not be neglected. To address this challenge, we propose a task-guided denoising network to conduct adversarial defense for the remote sensing scene classification task in this study. Specifically, given an adversarial remote sensing image, we use a denoising network to transform it as close to its corresponding clean image as possible with the constraint of the appearance loss. Besides, to further correct the predicted logits, the perceptual loss and the classification loss are adopted with the aid of a pre-trained classification network with fixed weights. Despite its simplicity, extensive experiments on the UAE-RS (universal adversarial examples in remote sensing) dataset demonstrate that the proposed method can significantly improve the resistibility of diferent deep learning models against the adversarial examples. Adversarial defense, adversarial attack, adversarial example, remote sensing, scene classification, deep learning CDCEO 2022: 2nd Workshop on Complex Data Challenges in Earth 0000-0003-1203-741X (P. Ghamisi)
nificantly boosted the interpretation of geoscience and Recent advances in deep learning algorithms have sig- and the generated adversarial examples are combined to train the model [7, 8]. Nevertheless, adversarial training remote sensing data [1, 2]. Nevertheless, the vulnera- can hardly improve the inherent robustness of deep neubility of deep learning models to adversarial examples should not be neglected. Szegedy et al. first discovered that deep neural networks are very fragile to specific perturbations generated by adversarial attack methods [3]. Simply adding these mild perturbations to the clean images, the adversarial examples are generated, which may possess imperceptible diferences from the original images for human observers but could mislead the deep neural networks to make wrong predictions with high confidence. In fact, this phenomenon is not limited to computer vision tasks. Researchers have found that adversarial examples do exist in the geoscience and remote sensing field and can be generated based on optical data [4], LiDAR point cloud [5], or even synthetic aperture radar (SAR) data [6]. Since most geoscience and remote sensing tasks are highly safety-critical, it is vitally important to develop adversarial defense methods and improve the resistibility of the deployed deep learning model against adversarial examples. nEvelop-O
Task-guided denoising network Pre-trained classification network Data flow of the adversarial examples Data flow of the clean images
addresses the adversarial defense problem from the per- image from the training set, we first use I-FGSM spective of denoising. Specifically, we propose a novel [13] to generate the corresponding adversarial example task-guided denoising network (TGDN) for the adversar- (note that the use of I-FGSM is only to simulate ial defense of remote sensing scene classification. The the adversarial examples that may exist in the test set main idea of the proposed method is to train a denois- since we have no access to the real adversarial attack ing network using clean remote sensing images and the method adopted by the adversary in practice). Then, corresponding adversarial examples. Since it is usually we use to denoise and get the transformed image infeasible to know which attack method the adversary ( ). Specifically, aims to alleviate the diference would use in practice, we adopt the iterative fast gradient between and from three aspects: the visual apsign method (I-FGSM) [13] to generate the adversarial ex- pearance diference, the feature representation diference, amples for the simulation purpose in the training phase. and the probability distribution diference. Accordingly, Once the training is finished, the denoising network is the training of is constrained by the appearance loss expected to possess the defense ability against unknown ℒ , perceptual loss ℒ , and classification loss ℒ adversarial attacks. Despite its simplicity, extensive ex- with the aid of Φ. Once the training is finished, we then periments on the UAE-RS (universal adversarial examples use to denoise samples in the adversarial test set. in remote sensing) dataset [14] demonstrate that the proposed TGDN can significantly improve the resistibility 2.2. Optimization of diferent deep learning models against the adversarial examples. Since the adversarial perturbation also belongs a special
The rest of this paper is organized as follows. Sec- type of noise, an intuitive idea is to conduct a transfortion 2 describes the proposed TGDN in detail. Section 3 mation on the input adversarial example and remove the presents the experiments in this study. Conclusions and existing adversarial perturbation. To this end, we first other discussions are made in Section 4. adopt the ℓ1 norm to define the appearance loss ℒ :
2.1. Overview of the Proposed TGDN As shown in Figure 1, there are two main components in the proposed adversarial defense framework, including a task-guided denoising network and a pre-trained classification network Φ (with fixed weights). Given a clean ℒ =
1 ∑ ∑ | ( )(,) =1 =1 − (,) |, (1) where and denote the numbers of row and column in the image, respectively. The constraint in (1) will encourage the transformed image ( ) to possess similar appearance to the original clean image .
Considering that the adversarial perturbation would also influence the intermediate feature representation of
Downsampling where and denote the numbers of row and column in the intermediate feature map, respectively. Φ (⋅) denotes the output of the intermediate feature extraction layer in the pre-trained classification network Φ (with ifxed weights). With the constraint in (2), ( ) will tend to possess identical high-level feature representation to the original clean image [15]. )(,) ‖2, (2)
ℒ = ‖ (Φ ( ( ))) − (Φ ( where (⋅) denotes the softmax function, and Φ (⋅) is the predicted logits of Φ. With the constraint in (3), ( ) will tend to possess similar probability distribution to the original clean image on the pre-trained network Φ. The complete loss function ℒ for training the proposed framework is formulated as: ℒ = ℒ + ℒ + ℒ , (4) where is a weighting factor.
Downsampling
3.1. Dataset
sensing) dataset1 is utilized to evaluate the performance of the proposed method.
UAE-RS provides high-resolution remote sensing adversarial examples for both scene classification and semantic segmentation tasks [14]. For the scene classification task, UAE-RS contains 1050 adversarial test samples 1https://github.com/YonghaoXu/UAE-RS from the UCM dataset and 5000 adversarial test samples from the AID dataset generated by the Mixcut-Attack method. Some example adversarial images are shown in the first columns of Figures 2 and 3. 3.2. Implementation Details
ter [12], DnCNN (Denoising CNN) [18], CBDNet (Convolutional Blind Denoising Network) [19], HiNet (Half Instance Normalization Network) [20], along with the proposed TGDN to conduct adversarial defenses. For the JPG method, we compress the adversarial examples 3.3. Experimental Results 4. Conclusions and Discussions with a quality of 25 [21]. The Downsampling method is mance improvements obtained by denoising methods are implemented with a sampling rate of 0.5 by the bilinear more obvious. However, although all the denoising netinterpolation. For the Color Jitter method, we randomly works used in this study may yield similar results from change the brightness, contrast, saturation, and hue of the perspective of the visual appearance according to Figthe adversarial examples using the uniform distribution ures 2 and 3, their quantitative defense performance may from 0.5 to 1.5. vary a lot in diferent scenarios. Take the VGG16 model
The I-FGSM [13] with the ℓ∞ norm is adopted to gen- on the UCM adversarial test set for example. While the erate adversarial examples for training the denoising proposed TGDN can yield an OA of around 68%, CBDNet networks used in this study. The perturbation level in can only yield an OA of around 49% in this case. This I-FGSM is fixed to 1 and the number of total iterations is phenomenon indicates that simply using traditional deset to 5. We use the same transform network used in the noising networks may not defend the adversarial attacks HiNet to implement the task-guided denoising network efectively since there is no specific design to tackle the . The ResNet18 [22] pre-trained on the UCM dataset adversarial perturbations in traditional denoising methor the AID dataset is adopted as the classification net- ods. Even though the denoised images are very similar work Φ. The weighting factor in (4) is set to 1 − 3 . to the original clean images, there may still exist imperWe use the Adam optimizer [23] with a learning rate of ceptible adversarial perturbations that are harmful to 1 − 3 and a weight decay of 5 − 5 to train the denoising the recognition performance. By contrast, the proposed networks used in this study. The batch size is set to 32, TGDN can achieve the highest OA in all defense scenarios and numbers of training epochs are set to 100, and 30 for except the case of the ResNet18 on the UCM adversarial UCM and AID datasets, respectively. All experiments in test set, where TGDN ranks second place. These results this study are implemented with the PyTorch platform demonstrate the efectiveness of the proposed method. [24] using two NVIDIA Tesla A100 (40GB) GPUs.
To qualitatively evaluate how diferent transformations Although deep learning-based methods have achieved would influence the input adversarial examples, we first state-of-the-art performance in the interpretation of geovisualize some example adversarial images in the UAE- science and remote sensing data, their vulnerability to RS dataset and the corresponding transformed images adversarial examples can not be ignored in practical apwith diferent methods in Figures 2 and 3. Compared plications. To address the threat of adversarial examples to the traditional transformation methods like the JPG for the remote sensing scene classification task, we proor Downsampling, the transformed images generated by pose a novel task-guided denoising network (TGDN) to denoising methods generally possess much more similar conduct the adversarial defense in this study. Specifically, appearances to the original clean images. Besides, it the proposed TGDN aims to alleviate the diference becan be observed that the visual appearance diference tween the adversarial examples and the original clean of the denoised images generated by DnCNN, CBDNet, images from three aspects: the visual appearance difHiNet, and TGDN is very dificult to perceive for human ference, the feature representation diference, and the observers. probability distribution diference. To further evaluate
We further test the overall accuracy (OA) of diferent how TGDN would influence the classification results deep learning models on the UAE-RS dataset using dif- of diferent deep learning models, the UAE-RS dataset ferent transforms to quantitatively evaluate how these is used in the experiments. Despite the simplicity of defense methods would influence the classification per- the proposed TGDN, extensive experiments demonstrate formance. As shown in Tables 1 and 2, due to the threat that TGDN can significantly improve the resistibility of of adversarial attacks, the existing state-of-the-art deep diferent deep learning models against the adversarial learning models can hardly achieve satisfactory recogni- examples. tion results if no transform process or defense method is Since the proposed method only considers a single used. On the UAE-RS AID adversarial test set, all models pre-trained network (ResNet18) when training the taskused in this study can only achieve an OA of less than guided denoising network, whether the ensemble learn25% without transform. Besides, the improvements ob- ing with multiple pre-trained networks would improve tained from traditional transform methods like the JPG, the defense performance deserves further study. We will Downsampling, and Color Jitter are limited and not stable. try to explore it in our future work. In some cases, they would even decrease the accuracy as these methods may bring about new noises or style diferences, which are harmful to the deployed models.
Compared to traditional transform methods, the perforial machine learning at scale, arXiv preprint arXiv:1611.01236 (2016). [1] P. Ghamisi, J. Plaza, Y. Chen, J. Li, A. J. Plaza, Ad- [14] Y. Xu, P. Ghamisi, Universal adversarial examples vanced spectral classifiers for hyperspectral images: in remote sensing: Methodology and benchmark, A review, IEEE Geosci. Remote Sens. Mag. 5 (2017) IEEE Trans. Geos. Remote Sens. 60 (2022) 1–15. 8–32. [15] J. Johnson, A. Alahi, L. Fei-Fei, Perceptual losses for [2] Y. Xu, B. Du, L. Zhang, D. Cerra, M. Pato, E. Car- real-time style transfer and super-resolution, in: Eumona, S. Prasad, N. Yokoya, R. Hänsch, B. Le Saux, ropean Conference on Computer Vision, Springer, Advanced multi-sensor optical remote sensing for 2016, pp. 694–711. urban land use and land cover classification: Out- [16] G. K. Dziugaite, Z. Ghahramani, D. M. Roy, A study come of the 2018 ieee grss data fusion contest, IEEE of the efect of jpg compression on adversarial imJ. Sel. Topics Appl. Earth Observ. Remote Sens. 12 ages, arXiv preprint arXiv:1608.00853 (2016). (2019) 1709–1724. [17] C. Guo, M. Rana, M. Cisse, L. Van Der Maaten, [3] C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Er- Countering adversarial images using input transhan, I. Goodfellow, R. Fergus, Intriguing properties formations, arXiv preprint arXiv:1711.00117 (2017). of neural networks, arXiv preprint arXiv:1312.6199 [18] K. Zhang, W. Zuo, Y. Chen, D. Meng, L. Zhang, (2013). Beyond a gaussian denoiser: Residual learning of [4] L. Chen, Z. Xu, Q. Li, J. Peng, S. Wang, H. Li, An deep cnn for image denoising, IEEE Trans. Image empirical study of adversarial examples on remote Process. 26 (2017) 3142–3155. sensing image scene classification, IEEE Trans. [19] S. Guo, Z. Yan, K. Zhang, W. Zuo, L. Zhang, Toward Geos. Remote Sens. 59 (2021) 7419–7433. convolutional blind denoising of real photographs, [5] Y. Cao, C. Xiao, B. Cyr, Y. Zhou, W. Park, S. Ram- in: Proc. IEEE Conf. Comput. Vis. Pattern Recognit., pazzi, Q. A. Chen, K. Fu, Z. M. Mao, Adversarial sen- 2019, pp. 1712–1722. sor attack on lidar-based perception in autonomous [20] L. Chen, X. Lu, J. Zhang, X. Chu, C. Chen, Hinet: driving, in: Proceedings of the 2019 ACM SIGSAC Half instance normalization network for image conference on computer and communications secu- restoration, in: Proc. IEEE Conf. Comput. Vis. Patrity, 2019, pp. 2267–2281. tern Recognit., 2021, pp. 182–192. [6] H. Li, H. Huang, L. Chen, J. Peng, H. Huang, Z. Cui, [21] G. K. Wallace, The jpeg still picture compression X. Mei, G. Wu, Adversarial examples for cnn-based standard, IEEE Transactions on Consumer Elecsar image classification: An experience study, IEEE tronics 38 (1992) xviii–xxxiv.
J. Sel. Topics Appl. Earth Observ. Remote Sens. 14 [22] K. He, X. Zhang, S. Ren, J. Sun, Deep residual learn(2020) 1333–1347. ing for image recognition, in: Proc. IEEE Conf. [7] I. J. Goodfellow, J. Shlens, C. Szegedy, Explain- Comput. Vis. Pattern Recognit., 2016, pp. 770–778. ing and harnessing adversarial examples, arXiv [23] D. P. Kingma, J. Ba, Adam: A method for stochaspreprint arXiv:1412.6572 (2014). tic optimization, arXiv preprint arXiv:1412.6980 [8] Y. Xu, B. Du, L. Zhang, Assessing the threat of (2014).
adversarial examples on deep neural networks for [24] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradremote sensing scene classification: Attacks and bury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, defenses, IEEE Trans. Geos. Remote Sens. 59 (2021) L. Antiga, et al., Pytorch: An imperative style, high1604–1617. performance deep learning library, Advances in [9] N. Akhtar, A. Mian, N. Kardan, M. Shah, Advances Neural Information Processing Systems 32 (2019). in adversarial attacks and defenses in computer vision: A survey, IEEE Access (2021). [10] Y. Xu, B. Du, L. Zhang, Self-attention context network: Addressing the threat of adversarial attacks for hyperspectral image classification, IEEE Trans.
Image Process. 30 (2021) 8671–8685. [11] P. Tabacof, E. Valle, Exploring the space of adversarial images, in: International Joint Conference on
Neural Networks (IJCNN), IEEE, 2016, pp. 426–433. [12] E. Raf, J. Sylvester, S. Forsyth, M. McLean, Barrage of random transforms for adversarially robust defense, in: Proc. IEEE Conf. Comput. Vis. Pattern
Recognit., 2019, pp. 6528–6537. [13] A. Kurakin, I. Goodfellow, S. Bengio, Adversar