=Paper=
{{Paper
|id=Vol-2670/MediaEval_19_paper_57
|storemode=property
|title=Image
Enhancement and Adversarial Attack Pipeline for Scene Privacy Protection
|pdfUrl=https://ceur-ws.org/Vol-2670/MediaEval_19_paper_57.pdf
|volume=Vol-2670
|authors=Muhammad Bilal Sakha
|dblpUrl=https://dblp.org/rec/conf/mediaeval/Sakha19
}}
==Image
Enhancement and Adversarial Attack Pipeline for Scene Privacy Protection==
https://ceur-ws.org/Vol-2670/MediaEval_19_paper_57.pdf
Image Enhancement and Adversarial Attack Pipeline for Scene
Privacy Protection
Muhammad Bilal Sakha1
1 Habib University, Pakistan
mbilal.sakha@gmail.com
ABSTRACT 2 APPROACHES
In this paper, we propose approaches to prevent automatic inference 2.1 Fusion based approaches
of scene class by classifiers and also enhance (or maintain) the vi-
CartoonGAN style transfer and Iterative least-likely class
sual appeal of images. The task is part of the Pixel Privacy challenge
adversarial attack: In the first approach, we use an image style
of the MediaEval 2019 workshop. The fusion based approaches we
transfer method based on Generative Adversarial Networks (GANs)
propose apply adversarial perturbations on the images enhanced
[4] called CartoonGAN [1], which enhances the image by applying
by image enhancement algorithms instead of the original images.
cartoon style effects. On these enhanced set of images, we then
They combine the benefits of image style transfer/contrast enhance-
apply a white-box targeted adversarial attack called the Iterative
ment and the white-box adversarial attack methods and have not
least-likely class method [7], which is a variant of the Fast Gradient
been previously used in the literature for fooling the classifier and
Sign Method (FGSM) proposed by [5]. The Iterative least-likely
enhancing the images at the same time. We also propose to use
class method tries to make an adversarial image by adding noise
simple Euclidean transformations which include image translation
to the clean image, so that it will be classified as the class with the
and rotation and show their efficacy in fooling the classifier. We
lowest confidence score for clean image. For choosing optimal ϵ
test the proposed approaches on a subset of the Places365-standard
(limit on the perturbation size), instead of doing binary search on
dataset and get promising results.
each example because of the computational expense, we choose
the value of ϵ to be 8/255 on the basis of experimental results on a
1 INTRODUCTION subset of validation set images. When enhancing the images using
CartoonGAN, Hayao style is chosen because it results in the largest
Social media users unintentionally expose private information
increase of mean aesthetic score among different CartoonGAN
when sharing photos online [12], such as locations a user visited
styles on the validation images.
etc., which can be automatically inferred by state of the art methods
CartoonGAN style transfer and PGD: In a slightly modified
[2]. The focus of Pixel Privacy task of MediaEval 2019 workshop
version, we now apply Projected Gradient Descent (PGD) [11] ad-
[10] is to protect user uploaded multimedia data online. The task
versarial attack after enhancing the images with CartoonGAN style
objective is to use image transformation algorithms for blocking
transfer. Here, we apply an untargeted adversarial attack, unlike in
the automatic inference of scene class by convolutional neural net-
the previous method where the target class is the least-likely class
work (ConvNet) based ResNet50 classifier [6] trained on Places365-
of clean image. For the PGD adversarial attack, we chose the value
standard dataset [15]. The proposed methods should also either
of ϵ to be 2/255 and the stepsize is chosen as 1/ϵ on the basis of
increase (or maintain) the visual appeal of an image. Additional
empirical results on a subset of validation images.
details of the task can be found in [10].
Image contrast enhancement & Iterative least-likely class:
We propose to combine image style-transfer and image enhance-
In this approach, we first enhance the contrast of the images using
ment with adversarial image perturbations to increase the visual
the method proposed by [14] and then perturb the enhanced images
appeal of the images, in addition to blocking the automatic infer-
using the Iterative least-likely class adversarial method [7]. The rea-
ence of scene class information by the classifier. We also apply
son for applying image processing to enhance the images initially
white-box (where the attacker has access to the model’s parame-
is because the adversarial perturbation methods, reduce the visual
ters) adversarial perturbations alone to compare the performance
appeal of the images, so enhancing the visual appeal of the images
to the fusion based approaches. Finally, we use simple euclidean
before applying adversarial perturbations will not only result in bet-
operations like image translation and rotation to show how they
ter performance on image quality metrics, but may also incentivize
are also able to fool the classifier. The proposed approaches are
users to use this method over adversarial perturbations alone. In
evaluated on the basis of reduction in the top-1 classifier accuracy
the image contrast enhancement approach by [14], the input image
and Neural Image Assessment (NIMA) [13] score is used to evalu-
is fused with the synthetic image, which is obtained by finding the
ate the image quality of the transformed images. The motivation
best exposure ratio to well-expose the under-exposed regions in
behind proposing fusion based approaches is to incentivize the
the original image. Both the images are then fused according to
social media users to use such methods for not only protecting the
the weight matrix, which is designed using illumination estimation
privacy-sensitive information in the photos, but also to enhance
techniques and the output is the contrast enhanced image. On these
their photos as an added bonus.
enhanced set of images, we then apply the Iterative least-likely
Copyright 2019 for this paper by its authors. Use class method, with the same parameters values as mentioned in the
permitted under Creative Commons License Attribution first approach.
4.0 International (CC BY 4.0).
MediaEval’19, 27-29 October 2019, Sophia Antipolis, France
MediaEval’19, 27-29 October 2019, Sophia Antipolis, France M. Sakha
Figure 1: Original sample image from the Places356-standard dataset and the images transformed using different approaches
with their corresponding top-5 classifier predictions.
2.2 White-box Private-FGSM adversarial attack Table 1: Accuracy and NIMA score of different approaches
In order to compare the adversarial image perturbations with pre-
vious fusion based approaches, we use a more powerful variant of Run/Method Accuracy NIMA score
FGSM method, called Private-Fast Gradient Sign Method (P-FGSM) 1. CartoonGAN + least-likely class 0.167% 4.37
recently proposed by [8]. The values of ϵ and σ used for this method 2. CartoonGAN + PGD 14% 4.77
are set to 8/255 and 0.99 respectively. 3. Contrast Enh. + least-likely class 0% 4.47
4. Private-FGSM 0% 4.49
2.3 Euclidean transformations 5. Euclidean transformations 1 6.667% 4.42
Inspired from the center crop and random crop operations in [9]
Original test images 100% 4.64
to fool the classifier, we choose to explore other simple geometric
operations on images, which are often overlooked in favor of adver- 1 Euclidean transformations evaluated on a smaller subset of
sarial attacks to fool the classifier. We consider two basic euclidean test dataset consisting of 60 images.
transformations i.e. image translation and rotation. To choose the
optimal translation and rotation value to fool the classifier, we use
the robust optimization method proposed by [3], instead of the the top-1 classifier accuracy to 0%, at the cost of added noise in the
computationally expensive grid-search. For majority of the images, submitted images, which is reflected in the reduced NIMA score of
we constrain translation to be within 20% of image size in each 4.49. Private-FGSM attack and previously used Iterative least-likely
spatial direction and rotation up to 20°, and fill the resulting empty class methods are bounded by l ∞ norm, which results in small
image spaces with zero pixel value. noise evenly distributed in the image, as can be seen by zooming
the transformed images in Figure 1.
3 RESULTS AND EVALUATION Euclidean transformations: The final run of euclidean trans-
In the Pixel Privacy task of the MediaEval 2019 workshop, the formations which consists of translation and rotation operations
participants are allowed to submit five runs for the task, which achieves 6.667% top-1 classifier accuracy with a reasonable NIMA
are evaluated on the basis of top-1 classification accuracy (lower is score of 4.42. For each image, finding the optimal translation and
better) and NIMA score [13] (higher is better), as shown in Table 1. rotation value to fool the classifier is computationally expensive
Figure 1 shows the original image and the transformed images by due to number of random transformations, therefore we test this
different approaches and the corresponding top-5 class prediction. approach on smaller subset of test dataset consisting of 60 images
Fusion based approaches: The performance of CartoonGAN + called test_manual, provided by the task organizers.
Iterative least-likely class adversarial method is good in terms of
the top-1 accuracy, however it has the worst NIMA score of 4.37 4 CONCLUSION AND OUTLOOK
among all runs. CartoonGAN + PGD adversarial method has the In this paper, different approaches have been proposed for the
best NIMA score of 4.77 among all runs, but considerably higher Pixel Privacy task of MediaEval 2019 workshop. The fusion based
classifier accuracy of 14%, which it is still less than 50%. approaches combining style transfer/image enhancement with ad-
For Contrast Enhancement + Iterative least-likely class run, we versarial attacks are chosen to increase the image appeal score
get the lowest 0% top-1 accuracy and 4.47 NIMA score. The images beforehand, as reducing the classifier accuracy through adversarial
enhanced using contrast enhancement method [14] look visually perturbations decrease image appeal score, due to addition of noise.
more appealing to the naked eye, however the NIMA score after In future, increasing image appeal by using the state of the art
applying only contrast enhancement is still slightly less than that deep learning based image enhancement methods for image de-
of the clean images which is unexpected. noising, color/contrast/exposure adjustment etc. and then applying
Private-FGSM adversarial attack: Private-FGSM attack reduces adversarial perturbation in our opinion will yield better results.
Pixel Privacy MediaEval’19, 27-29 October 2019, Sophia Antipolis, France
REFERENCES
[1] Yang Chen, Yu-Kun Lai, and Yong-Jin Liu. 2018. CartoonGAN: Gen-
erative adversarial networks for photo cartoonization. In Proceedings
of the IEEE Conference on Computer Vision and Pattern Recognition.
9465–9474.
[2] Jaeyoung Choi, Martha Larson, Xinchao Li, Kevin Li, Gerald Fried-
land, and Alan Hanjalic. 2017. The geo-privacy bonus of popular
photo enhancements. In Proceedings of the 2017 ACM on International
Conference on Multimedia Retrieval. ACM, 84–92.
[3] Logan Engstrom, Brandon Tran, Dimitris Tsipras, Ludwig Schmidt,
and Aleksander Madry. 2019. Exploring the Landscape of Spatial
Robustness. In International Conference on Machine Learning. 1802–
1811.
[4] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David
Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014.
Generative adversarial nets. In Advances in neural information process-
ing systems. 2672–2680.
[5] Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. 2014.
Explaining and harnessing adversarial examples. arXiv preprint
arXiv:1412.6572 (2014).
[6] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep
residual learning for image recognition. In Proceedings of the IEEE
conference on computer vision and pattern recognition. 770–778.
[7] Alexey Kurakin, Ian Goodfellow, and Samy Bengio. 2016. Adversarial
examples in the physical world. arXiv preprint arXiv:1607.02533 (2016).
[8] C. Y. Li, A. S. Shamsabadi, R. Sanchez-Matilla, R. Mazzon, and A.
Cavallaro. 2019. Scene Privacy Protection. In Proc. IEEE Int. Conf. on
Acoustics, Speech and Signal Processing. Brighton, UK.
[9] Zhuoran Liu and Zhengyu Zhao. 2018. First Steps in Pixel Privacy:
Exploring Deep Learning-based Image Enhancement against Large-
Scale Image Inference.. In MediaEval.
[10] Zhuoran Liu, Zhengyu Zhao, and Martha Larson. 2019. Pixel Privacy
2019: Protecting Sensitive Scene Information in Images. In Working
Notes Proceedings of the MediaEval 2019 Workshop.
[11] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris
Tsipras, and Adrian Vladu. 2017. Towards deep learning models resis-
tant to adversarial attacks. arXiv preprint arXiv:1706.06083 (2017).
[12] Tribhuvanesh Orekondy, Bernt Schiele, and Mario Fritz. 2017. Towards
a visual privacy advisor: Understanding and predicting privacy risks in
images. In Proceedings of the IEEE International Conference on Computer
Vision. 3686–3695.
[13] Hossein Talebi and Peyman Milanfar. 2018. NIMA: Neural image
assessment. IEEE Transactions on Image Processing 27, 8 (2018), 3998–
4011.
[14] Zhenqiang Ying, Ge Li, Yurui Ren, Ronggang Wang, and Wenmin
Wang. 2017. A new image contrast enhancement algorithm using
exposure fusion framework. In International Conference on Computer
Analysis of Images and Patterns. Springer, 36–46.
[15] Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio
Torralba. 2017. Places: A 10 million Image Database for Scene Recog-
nition. IEEE Transactions on Pattern Analysis and Machine Intelligence
(2017).