Personal privacy is one of the essential problems in modern society. In some cases, people may not want smart computing systems to automatically identify and reveal their personal information, such as places or habits. This motivates our proposal to protect scene category recognition from photos by back-propagation. To further improve the visual quality and attraction of output photos, we study and propose various strategies for image enhancement, from traditional approaches to novel GAN-based methods. Our solution can successfully fool the Place365 scene classification in 60 categories while achieving the average NIMA score up to 5.36.
Categories' Scores 0.007 0.85 fool the Places365 scene classification, while achieving an average NIMA score up to 5.36.
The content of our report is as follows. In Section 2, we present our method for protecting scene category and enhancing image with diferent approaches. Experimental results are in Section 3. Conclusion and future works are discussed in Section 4. 2
Our goal is to protect the scene category of the input image, and our
output result should be in better quality based on NIMA [
To protect the image’s location information, we need to modify the image so that the output class becomes diferent compared to the ground truth. In other words, we need to reduce the output probability of the ground truth class.
With this intuition in mind, we consider the output probability of ground truth class as the target function f to minimize with gradient descent and the input image as the parameter θ to optimize.
Formally, let θ be the input image, θ ′ the modified image, f the model we want to fool. Our main goal is to modify input image : where ϵ is obtained after backwarding through network encoding in Fig.1
Input Image
With the rapid development of computer vision and new machine
learning methods, computers can now understand content of
images, e.g. which object is in an image, who is in the image, what
is happening in the image, or recognize scene’s category and
attributes, etc. This provides foundation for intelligent interactive
systems such as smart homes, self-driving cars, etc. This fact,
however, can raise potential risks of personal privacy violation. A person
might not want other people to know where he or she is. Important
facilities like hospitals or military camps should not be
automatically recognized also. This motivates the proposal of the problem
of Pixel Privacy: to prevent automatic systems from recognizing
scene categories from images [
In the Pixel Privacy 2019 [
We propose the method to attack the ResNet50 model using back
propagation. For each input image, we consider the logit for its
target class as a function on the input image’s pixels. This allows
back propagation to minimize this logit by modifying the input
image. We also use several methods to enhance the input images’
quality before feeding it through the protection phase, such as
natural enhancement with Dynamic Histogram Equalization [
Frozen Modify
Backward 2.2
2.2.1 Natural Enhancement. For the first run, we try to improve
image quality with traditional computer graphic methods. We start
with Dynamic Histogram Equalization[
2.2.2 Style Transfer. For this run, we apply a style
transferbased approach to improve images’ appeal. In this particular case,
we run CartoonGAN [
2.2.3 GAN Enhancement. In this run, we try to improve image’s
quality by method proposed in [
2.2.4 Retouch. In this run, we assume that the evaluation method
with NIMA Score is as natural as the human visual system. We try
to enhance the image with [
2.2.5 Texture. In this run, we hypothesize that any natural image has an average NIMA score between 3 and 4.5. We attempted to validate this hypothesis by computing NIMA score for several randomly created images, which results in fairly high score. With that in mind, we blend noise images to the original images to emphasize details usually ignored by NIMA model, while keeping the scores above 3. We try 2 ways to blend the noise images:
Way 1: x ′ = x ⊙ (αϵ), where x and x ′ are the input and ouput images, respectively; ϵ is the crafted noise image; α is the coeficient in which the crafted image is blended into the original.
Way 2: x ′ = x + αϵ, where x and y are the V (as in HSV) channels of the original image and the result image, respectively; ϵ is the crafted noise image; α is the coeficient with which the crafted image is blended into the original. 3
Most of our experiment is conducted on Google Colab. Table 1: Oficial evaluation result (provided by organizers)
Method
Original Image hcmus_naturalenhancement
hcmus_retouch hcmus_ganenhancement hcmus_cartoongan hcmus_texture
Top-1 Accuracy
In the table above, the "Top-1 Accuracy" column shows the
prediction accuracy of the attack model (ResNet50 [
Places365-standard data set), which means lower is better. The "NIMA Score" column represents the mean aesthetics scores of your runs. A higher NIMA score is better.
As our experiment result shows that all of our five runs perfectly protected the scene category of the image from the attack model. And from our observation, most GAN-based method can not achieve the same natural level as traditional computer graphics approach. The texture method currently has the highest score (5.36). Moreover, without the protection method, the texture method also can protect more than 2/3 datasets.
However, sophisticated approaches like Texture or Cartoon Gan transform images excessively, and output images do not have a natural appearance anymore. That is the reason why we propose Traditional Enhancement Methods like Saturation modify or DHE algorithm. Not only does it maintains the original beauty, but Natural Enhancement also achieves 2nd highest score.
(a) Original
(b) Natural
(c) GAN
Pub (d) Retouch
Pub (e) Cartoon GAN
(f) Texture
We propose one simple yet efective approach for Pixel Privacy Problem. Our method is using backpropagation to modify certain pixels of the input image while freezing all intermediate modules in the attack model. This method could be expanded for other categories besides scene category, for example, vehicle detection, human tracking, and so on. We also propose and apply five methods to enhance the input image, from new methods, namely, Cartoon GAN (Style Transfer) and Texture efect, to traditional image enhancement methods. All of which provides images with a higher average NIMA Score than original images.
Research is supported by Vingroup Innovation Foundation (VINIF) in project code VINIF.2019.DA19. We would like to thank AIOZ Pte Ltd for supporting our team with computing infrastructure.