Deep neural networks have been widely used in many computer vision tasks. However, it is proved that they are susceptible to small, imperceptible perturbations added to the input. Inputs with elaborately designed perturbations that can fool deep learning models are called adversarial examples, and they have drawn great concerns about the safety of deep neural networks. Object detection algorithms are designed to locate and classify objects in images or videos and they are the core of many computer vision tasks, which have great research value and wide applications. In this paper, we focus on adversarial attack on some state-of-the-art object detection models. As a practical alternative, we use adversarial patches for the attack. Two adversarial patch generation algorithms have been proposed: the heatmap-based algorithm and the consensus-based algorithm. The experiment results have shown that the proposed methods are highly efective, transferable and generic. Additionally, we have applied the proposed methods to competition Adversarial Challenge on Object Detection and won top 7 in 1701 teams. Code is available at https://github.com/FenHua/DetDak
• Security and privacy → Software and application security; • Computing methodologies → Object detection.
While being widely used in many fields, deep neural networks are shown to be vulnerable to adversarial examples [4]. Many early studies of adversarial examples focused on the classification task, adding perturbation on the entire image. However, in real world applications like autonomous vehicles and surveillance, such perturbation is hard to implement. Because of this, recent studies focus mainly on adversarial patches, which restrict the perturbation to a small region like a rectangular area. This makes adversarial examples more practical and easier to implement.
Object detection is an important part of computer vision and enables many tasks like autonomous driving, visual question answering and surveillance. However, there are relatively few studies on the adversarial attack of object detection models, especially for ∗Both authors contributed equally to this research. †Corresponding author. the purpose of making the objects disappear. Since object detection models have been used in many life-concerning applications, research about the fragility of these models is of great importance.
Therefore, we aim to investigate the vulnerability of object detection algorithms in this work and attack four state-of-the-art object detection models provided by Alibaba Group on the Tianchi platform, including two white-box models — YOLOv4 [1] and Faster RCNN [2], and two black-box models to test the transferability of the proposed algorithm. The purpose of the designed methods is to blind the detection models with the restricted patches. The framework of adversarial attacking is shown in Figure 1.
We discover that the locations of adversarial patches are crucial to the attack, so we focus on locating the patches and propose two patch selection algorithms: the heatmap-based algorithm and the consensus-based algorithm. The heatmap-based algorithm is an improved version of Grad-CAM [3], which introduced the idea of heatmap to visualize the gradients of intermediate convolutional layers in image classifiers. We modify and improve the algorithm to make it suitable for visualizing the gradients in object detection models and use the heatmap to select patches. To the best our knowledge, it is the first Grad-CAM-like algorithm designed specifically for the object detection task. The consensus-based algorithm is another novel patch selecting method. It chooses patch locations by attacking several target models and combining the results with a voting strategy, which can make the location of the patch more precise and the adversarial examples more transferable.
We test our attacking algorithm with the proposed patch selection algorithms on the dataset provided by Alibaba Group. The result shows that the proposed algorithms are highly competitive. In brief, the main contributions can be summarized as follows: • We improve the Grad-CAM algorithm to make it more suitable for analysing the gradients of object detection models and use it for the heatmap-based attack. • We propose consensus-based attack algorithm that is very powerful for attacking object detection models. • The experimental results show that the proposed attacking methods are competitive and generic.
The rest of this paper is organized as follows. The proposed algorithms are described in Section 2. The experimental results and analysis are presented in Section 3. Finally, we summarize the work in Section 4. 2
Two methods have been designed for generating patches: the heatmapbased algorithm and the consensus-based algorithm. In this Section, two proposed methods are introduced in details. The adversarial attack algorithm with patches is also presented concretely at end of this Section. used in the normalization to avoid small bounding boxes from being too dominant. We compute ℎ as 2.1
Grad-CAM[3] is a popular tool for visualizing the derivative of the output with respect to an intermediate convolutional layer. It introduced the idea of heatmap — important regions of the input are hotter in the heatmap. The heatmap is a function of the gradients of the output with respect to an intermediate layer. However, the original Grad-CAM algorithm is designed for classification models and thus cannot be used directly in our task. On the one hand, object detection tasks usually have multiple objects for the input image, while the classification task only have one. On the other hand, the size of diferent objects could have a significant influence on the heatmap, so we cannot directly add the gradients together when computing the heatmap.
Therefore, an improved Grad-CAM algorithm is proposed for selecting patches. Firstly, we adopt the element-wise multiplication of the gradients and activations, which preserves spatial information of the gradients and the intermediate layer. Secondly, we normalize the heatmap data of all bounding boxes and combine them together to get the heatmap of the entire image. Thirdly, we use several intermediate layers of the backbone for computing the heatmap, which can combine both lower-level features and higher-level features. Finally, we get the patch mask according to the values of the heatmap.
Mathematically, we calculate the heatmap using the following formula: = Õ
,
∈A
where A is the set of several activation layers (like conv56, conv92
in YOLOv4). is the heatmap of a single activation layer , which
is defined as:
=
Õ ℎ − E[ℎ ]
∈ pVar[ℎ ]
· p,
(
Finally, we use some Gaussian filters to post-process the heatmap to make it more smooth. Combining the heatmaps of several object detection models, we can choose the patches in hot regions of the input image. 2.2
Although the heatmap-based algorithm exploits the gradient information of the models, it is separated from the attacking process. Besides, we find that the sensitive locations of the input image might change over time when attacking algorithms are performed iteratively. Therefore, we propose another method for patch selection: the consensus-based algorithm.
First of all, we perform the Fast Gradient Sign Method iteratively with 2-norm regularization on the target models respectively. Our loss function 2 is originally defined as: Õ =
∈ { | > } 2 = + · || ||22, , where is the confidence score of each bounding box of the corresponding model, is the confidence threshold (we use 0.3 in our task). Usually, when > , it indicates the bounding box is correct and will appear in the results. So the lower the confidence score , the fewer objects can be detected. represents the perturbation and is a hyper parameter.
In the experiments, we find that the noise perturbations of some
models like Faster RCNN are not concentrated, which makes it hard
to fuse multiple results. To solve this problem, we modified the loss
function of those models:
= Õ + (1 − )
Õ
ℎ ,
(
After 2 attack, we can get the noise of the input image of each model. However, we do not mix those noise directly, because they are diferent in magnitude and it is not easy to balance them. So we sparsify the noise into patches with a specified scale S. Next, we take a vote to decide which patch mask should be preserved and which should be discarded. Usually, the greater the perturbation, the more likely it is to be selected as patch. The voting strategy on those noise patches is very helpful for improving performance. The flow of the algorithm is described in Figure 2. Here, we introduce EficientDet [ 5] to join the vote. The more the detection models, the more accurate the voting results and the higher the adaptability. Furthermore, the voting strategy can also improve the transferability of our adversarial patches and the robustness of our algorithm. Additionally, the number of 2 attack iterations is not very sensitive. Even with only 5 iterations, the voting result is still quite decent. 2.3
After patch mask generation, Fast Gradient Sign Method (FGSM) is used to finish attacking:
:= clip[0,255] ( + · sign(∇ )), where is the parameter in the adversarial patches and is the learning rate which we refer to [4] for setting its value. is the loss function defined as: = Õ where () is the confidence score of the -th bounding box of model and M is set of detection models. The loss function is simple but eficient. Figure 1 ofers a comprehensible description of the attacking algorithm. The detail of the algorithm with consensusbased patch selection algorithm is also described in Algorithm 1. (7) (8)
Algorithm 1 Consensus-based Attack Algorithm Input: a clean image , the patch number , the scale of square patches S, the iteration number for attacking Output: an adversarial image ′ 1: Let the set of object detection models be M 2: for model ∈ M do 3: L2 attack on and get perturbation () 4: Get top noise patches in () as () with scale S 5: Normalize () 6: end for 7: ← Í ∈M () 8: P ← select top patch masks in according to the magnitude of the perturbation. 9: repeat 10: Perform FGSM attack on with patch masks in P 11: Update the polluted image ′. 12: ← − 1 13: until = 0 or = 0 // is the loss function 14: return ′ 3
We used the proposed methods in AIC Phase IV CIKM-2020: Adversarial Challenge on Object Detection competition. The results of two basic proposed methods without any ensemble operations are recorded in Table 1 and shown in Figure 3. Even without ensemble operations, the algorithms are quite competitive. In order to reduce the number of pixels of our patches, grid-like patches are designed.
To get grid-like patches, we performed a element-wise dot product between patch mask and a grid matrix . The grid-like mask ′ can be calculated by:
′ = ⊙ , where represents the degree of sparsity, and the larger is, the less pixels are used.
Ensemble operation is a common practice in machine learning and we used this in the task. To combine diferent results for getting better results, an indicator is defined,
FinalScore = Õ2 Õ =1 (, ∗, ), because there is two white-box detector, YOLO and Faster RCNN, can be used, the indicator is the sum of two score functions. is provided by Alibaba Group:
Í
(, ∗, ) = (
. (11) where is the number of pixels of the -th patch, is the clean image, ∗ is the adversarial example, denotes the -th model, and ( ; ) is the number of bounding boxes detected by on image .
When we perform ensemble operations with grid-like patches through ∈ {0.5, 0.6, 0.7}, we have got more than 2000 scores. The results are also recorded in Table 1 and shown in Figure 3. As you can see, the efect is obvious. (9) (10)
Since we can successfully attack most object detection models with grid-like sparse patches, we can also expand the region of the original patches and sparsify them to cover a larger area while altering a moderate number of pixels. Besides, we observe that the patches of a fixed size might only be suitable for some images, so we performed an ensemble over patches of diferent sizes S. Combining grid operations and the ensemble over patches of three diferent sizes S (S ∈ {20, 50, 70} in our experiments), our best result is over 3000 scores.
Note that the consensus-based algorithm is generally better than the heatmap-based algorithm in the experiments. A possible explanation for this is that the consensus-based algorithm might better combine the target models with the voting process and incorporate the attacking process with the selection process. Some adversarial images are shown in Figure 4. As shown, the patches are gridded and have diferent scales. Since there is no limit to the perturbations, the noise is obvious. In general, the greater the noise is, the better the attack transferibility is.
In this paper, two adversarial patch generation algorithms have been proposed: heatmap-based and consensus-based patch generation algorithms. The generated patches are eficient and precise. Additionally, they only rely on few pixels but are generic. Furthermore, the proposed attacking methods can misguide state-of-the-art object detection models from detecting the objects. Those adversarial examples are a great threat to deep neural networks deployed in real world applications. Through the study of adversarial examples, the mechanism of deep learning models can be further understood, and robust algorithms can also be proposed. In the future, we will explore how to improve the robustness of current detection models to deal with adversarial examples.