Fooling Blind Image Quality Assessment by Optimizing a Human-Understandable Color Filter ZhengyuZhao z.zhao@cs.ru.nl Radboud University
Netherlands
Fooling Blind Image Quality Assessment by Optimizing a Human-Understandable Color Filter 720DC0D44F558A4F593138D3DFDD38DB GROBID - A machine learning software for extracting information from scholarly documents

This paper presents the submission of our RU-DS team to the Pixel Privacy Task 2020. We propose to fool the blind image quality assessment model by transforming images based on optimizing a human-understandable color filter. In contrast to the common work that relies on small, 𝐿 𝑝 -bounded additive pixel perturbations, our approach yields large yet smooth perturbations. Experimental results demonstrate that in the specific context of this task, our approach is able to achieve strong adversarial effects, but has to sacrifice the image appeal.
INTRODUCTION

High-quality images shared online can be misappropriated for promotional goals. The Pixel Privacy Task [15] this year is focused on developing adversarial techniques to decrease the predicted quality scores of an automatic Blind Image Quality Assessment (BIQA) model [10], which effectively camouflages images from being promoted. A key requirement of such adversaries is that the adversarial image should remain its original quality or become more appealing to the human eye. Conventional work on generating adversarial images has been focused on small additive perturbations, mostly bounded by 𝐿 𝑝 distance [2,3,9,16], or other more visual-perceptionaligned metrics [4,18,19,21]. In this way, the adversarial image is only designed to maintain its original appearance as much as possible, instead of enhancing the image appeal.

In contrast, recent studies [1,6,7,13,14,17,20] have started to explore non-suspicious adversarial images that accommodate larger perturbations without arousing suspicion because they transform groups of pixels along dimensions consistent with human interpretation of images. Among them, the Adversarial Color Enhancement (ACE) [20] can simultaneously achieve the adversarial effects and image enhancement by optimizing a human-understandable parametric color filter. Its effectiveness has been originally validated in the domain of image classification and segmentation.

One may argue that it is easier to separately conduct the optimization for adversarial effects and image enhancement. However, we note that the joint optimization can yield larger perturbations that enjoy two important practical properties: robustness against common image processing operations and transferability to a black-box target model [1,17,20]. In this paper, specifically, we will explore the usefulness of ACE in this Pixel Privacy Task for decreasing the BIQA score while enhancing the image appeal.

Copyright 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). MediaEval'20, December 14-15 2020, Online

APPROACH

In this section, we firstly recall the general formulation of Adversarial Color Enhancement (ACE) as proposed by [20], and then present the modifications for applying it in our specific Pixel Privacy Task.

Parametric Image Enhancement

Most advanced automatic photo enhancement algorithms have proposed to parameterize the image editing process by the DNNs, which however suffers from high computational cost and low interpretability [8,12,22]. In contrast, recent work [5,11] has proposed to parameterize the process as human-understandable image filters. Such methods have far fewer parameters to optimize, and can be applied independently of the image resolution.

Specifically, ACE adopts the approximation of the color filter in [11], which is formulated as a simple monotonic piecewise-linear mapping function:

𝐹 𝜽 (π‘₯ π‘˜ ) = π‘˜βˆ’1 𝑖=1 πœƒ 𝑖 πœƒ sum + (𝐾 β€’ π‘₯ π‘˜ βˆ’ (π‘˜ βˆ’ 1)) β€’ πœƒ π‘˜ πœƒ sum , πœƒ sum = 𝐾 π‘˜=1 πœƒ π‘˜ ,

where 𝐾 demotes the total number of pieces. In this case, an input image pixel π‘₯ π‘˜ falling in the π‘˜-th piece will be filtered using the parameter πœƒ π‘˜ , and 𝐹 𝜽 (π‘₯ π‘˜ ) is its corresponding output. By doing this, pixels with similar colors will be filtered with the same parameter, leading to smooth color transformation. Specifically, the three RGB channels are processed independently. An example of this function with four pieces (𝐾 = 4) is illustrated in Fig. 1.

MediaEval'20, December 14-15 2020, Online Z. Zhao There are two methods to constrain the color transformation strength. The first method imposes adjustable bounds on the filter parameters, formulated as:

Runs Methodsmin 𝜽 𝐿 π‘Žπ‘‘π‘£ (𝐹 𝜽 (𝒙)), s.t. 1 ≀ βˆ₯ 𝜽 𝜽 0 βˆ₯ ∞ ≀ πœ–,

where 𝜽 0 denotes the initial parameters, equaling to 1 𝐾 /𝐾. The adversarial loss, 𝐿 π‘Žπ‘‘π‘£ , adopts the specific logit loss from the the wellknown C&W method [2]. Note that this parameter bound is not necessarily to tight as in the 𝐿 𝑝 methods, since the color filtering can inherently guarantee the uniformity of the image transformation even when the perturbations are large. This bounded variant of ACE is referred to as ACE-PGD.

The second method guides the transformation towards specific appealing color styles, in addition to achieving the adversarial effects. To this end, additional guidance from common enhancement practices is incorporated into the adversarial optimization. Specifically, the targeted appealing color styles are obtained by using Instagram filters, and the optimization can be formulated as:

min 𝜽 𝐿 π‘Žπ‘‘π‘£ (𝐹 𝜽 (𝒙)) + πœ† β€’ βˆ₯𝐹 𝜽 (𝒙) βˆ’ 𝒙 ins βˆ₯ 2 2 ,

where 𝒙 ins denotes the targeted Instagram filtered image with a specific color style. This variant of ACE is referred to as ACE-Ins. One popular Instagram filter style, Nashville, is considered in our submitted runs, and the implementation is automated using the GIMP toolkit with the Instagram Effects Plugins 1 .

In the context of fooling BIQA, the 𝐿 π‘Žπ‘‘π‘£ is formulated as:

𝐿 π‘Žπ‘‘π‘£ = max{BIQA(𝐹 𝜽 (𝒙)) βˆ’ 𝐢, 0},

where the target score can be set by adjusting 𝐢. Specifically, we set 𝐢 a bit lower than the standard target, 50, to make sure the adversarial effects could remain after the JPEG compression.

RESULTS AND ANALYSIS

In total, we submitted five runs. We tried different parameters of ACE-PGD for the first four runs, and used ACE-Ins for the last run.

As can be seen from Table 1, all the five runs effectively decrease the model accuracy to a level below 50%. Specifically, as expected, higher 𝐾 = 4 and πœ– lead to stronger adversarial effects. In addition, we find that the results before and after the JPEG compression remain similar, suggesting that our approach is stale against compression. However, the human evaluation results on the 20 selected images are not satisfying. It implies that the BIQA model is more stable against the interference of smooth modifications, such as ACE, than the classification models. Specifically, we notice that ACE-Ins fails to drive the image into a target appealing style since the optimization has to be focused on lowering the score. This may be because the quality assessment model tends to rely on highfrequency features but the ImageNet classifier learns both lowfrequency (e.g. shape) and high-frequency (e.g. textures) features. This makes the quality assessment model more robust against the low-frequency perturbations by our ACE. We will explore this in more depth for the future work.

Figure 1 :Figure 1: A 4-piece color filter in ACE ( from [20]).
Parameters 1 ACE-PGD 𝐾 = 64, πœ– = 16, and iters. = 20 2 ACE-PGD 𝐾 = 64, πœ– = 32, and iters. = 20 3 ACE-PGD 𝐾 = 256, πœ– = 16, and iters. = 20 4 ACE-PGD 𝐾 = 256, πœ– = 64, and iters. = 20 5 ACE-Ins 𝐾 = 64, πœ† = 0.01, and iters. = 100 2.2 Adversarial Color Enhancement ACE generates non-suspicious adversarial images by iteratively updating the parameters of the color filter defined in Eq. 1, in contrast to the conventional attacks that are operated in the raw pixel space.
1 https://www.marcocrippa.it/page/gimp_instagram.php.
Figure 2 :Figure 2: Adversarial images achieved by our approach with the original and decreased scores. The top row shows the examples with relatively high appeal and the bottom row shows the failed examples with low appeal.
Figure 2visualizes the successful adversarial examples with high and low appeal. We can observe that ACE can yield good image examples with filtering-like styles, but the bad examples suffer from over-colorization effects.
Table 1 :Detailed settings of our five runs.
Table 2 :Evaluation results of our five runs. The accuracy (%) is calculated over all the 550 test images, which are compressed with JPEG 90 before evaluation. The number of times selected as "Top-3" most appealing among the total 13 qualified runs is evaluated by user study with 7 people on 20 representative images that have the largest BIQA score variance. The maximum number is 140.
Runs12345Acc before JPEG 48.00 33.27 50.00 21.82 35.09Acc after JPEG45.27 33.45 47.45 22.55 44.91Number of Top-327647
ACKNOWLEDGMENTS

This work was carried out on the Dutch national e-infrastructure with the support of SURF Cooperative.

Pixel Privacy: Quality Camouflage for Social Images MediaEval'20, December 14-15 2020, Online

Unrestricted Adversarial Examples via Semantic Manipulation AnandBhattad Min JinChong KaizhaoLiang BoLi DavidAForsyth ICLR 2020 Towards evaluating the robustness of neural networks NicholasCarlini DavidWagner IEEE S&P 2017 EAD: elastic-net attacks to deep neural networks via adversarial examples Pin-YuChen YashSharma HuanZhang JinfengYi Cho-JuiHsieh AAAI 2018 Sparse and Imperceivable Adversarial Attacks FrancescoCroce MatthiasHein ICCV 2019 Aestheticdriven image enhancement by adversarial learning YubinDeng ChenChangeLoy XiaoouTang 2018 ACM MM Exploring the Landscape of Spatial Robustness LoganEngstrom BrandonTran DimitrisTsipras LudwigSchmidt AleksanderMadry 2019 ICML Robust physical-world attacks on deep learning models KevinEykholt IvanEvtimov EarlenceFernandes BoLi AmirRahmati ChaoweiXiao AtulPrakash TadayoshiKohno DawnSong CVPR 2018 Deep bilateral learning for real-time image enhancement MichaΓ«lGharbi JiawenChen JonathanTBarron SamuelWHasinoff FrΓ©doDurand ACM TOG 36 4 2017. 2017 Explaining and harnessing adversarial examples IanGoodfellow JonathonShlens ChristianSzegedy ICLR 2015 KonIQ-10k: An ecologically valid database for deep learning of blind image quality assessment VladHosu HanheLin TamasSziranyi DietmarSaupe IEEE TIP 29 2020. 2020 Exposure: A white-box photo post-processing framework YuanmingHu HaoHe ChenxiXu BaoyuanWang StephenLin ACM Transactions on Graphics 37 2 26 2018. 2018 Image-to-image translation with conditional adversarial networks PhillipIsola Jun-YanZhu TinghuiZhou AlexeiAEfros CVPR 2017 Semantic Adversarial Attacks: Parametric Transformations That Fool Deep Classifiers AmeyaJoshi AmitangshuMukherjee SoumikSarkar ChinmayHegde ICCV 2019 Functional Adversarial Attacks CassidyLaidlaw SoheilFeizi NeurIPS 2019 Exploring Quality Camouflage for Social Images ZhuoranLiu ZhengyuZhao MarthaLarson LaurentAmsaleg Working Notes Proceedings of the MediaEval Workshop 2020 Towards deep learning models resistant to adversarial attacks AleksanderMadry AleksandarMakelov LudwigSchmidt DimitrisTsipras AdrianVladu ICLR 2018 ColorFool: Semantic Adversarial Colorization RicardoAli Shahin Shamsabadi AndreaSanchez-Matilla Cavallaro CVPR 2020 Wasserstein Adversarial Examples via Projected Sinkhorn Iterations EricWong FrankSchmidt ZicoKolter ICML 2019 Spatially transformed adversarial examples ChaoweiXiao Jun-YanZhu BoLi WarrenHe MingyanLiu DawnSong ICLR 2018 Adversarial Robustness Against Image Color Transformation within Parametric Filter Space ZhengyuZhao ZhuoranLiu MarthaLarson arXiv:2011.06690 2020 arXiv preprint Towards Large yet Imperceptible Adversarial Image Perturbations with Perceptual Color Distance ZhengyuZhao ZhuoranLiu MarthaLarson CVPR 2020 Unpaired image-to-image translation using cycle-consistent adversarial networks Jun-YanZhu TaesungPark PhillipIsola AlexeiAEfros ICCV 2017