Object detectors are increasingly deployed in safety-critical systems, including autonomous vehicles. Recent studies have found that object detectors based on convolutional neural networks are fundamentally vulnerable to adversarial attacks. Adversarial attacks on object detectors involve adding a carefully chosen perturbation to the input, which causes the object detector to make mistakes. The potential consequences of adversarial attacks must be known to make sure these safety-critical systems are reliable. This paper investigates the influence of transfer attacks on object detectors, where the attacker does not access the target detector and its training set. Devising an attack with this assumption requires the attacker to train their model on data that resembles the target detector's training set. Using their model as a surrogate, attackers can generate adversarial attacks without accessing the target detector. Our study investigates whether one can efectively attack a black box model using publicly available data. We have performed targeted objectness gradient attacks on the state-of-the-art object detector (i.e., YOLO V3). Initial transferability between the attacking and target model is low. However, increasing attack strength from 8 to 24 strengthens transferability and reduces the target detector performance by about half. Transferability is also studied when the datasets for the attacking and the target model intersect. Attack performance is proportional to the size of the intersection. With the stronger transferability caused by intersecting datasets, attack strength can be dropped to 16 and retain the attack performance.
Object detection and classification have seen a rapid improvement since the ImageNet
competition in 2012, where the introduction of deep convolutional networks almost halved the error
rates of the best competing approaches [
Object detectors are diferent from classifiers because they are multi-task learners performing
a more complex task than classifiers. Recently, attacks have been developed that target object
detectors directly [
This study aims to find the level of exposure one can expect if an attacker tries to attack a
model they have no direct access to and do not know essential model details. We investigate
whether it is possible to transfer attacks across models trained on diferent data, i.e., models
are trained to detect the same semantic class, namely boat, with various datasets. In addition,
the efect on attack performance when the surrogate and target models are trained on datasets
with intersections is investigated. We trained some models on datasets that share images, while
others were trained on entirely disjoint datasets. The adversarial samples are generated using a
source model trained to detect ships on one dataset. Then the target model trained to detect
ships on a separate dataset attempts to make predictions on the sample. We employ Targeted
Objectness Gradient (TOG) attack [
The results show that perturbations generated with an epsilon=8 of the TOG attack under the distance metric ∞ have poor transference between the models. However, an epsilon in the 2030 range results in better transference and more efective attacks. Models trained on intersecting datasets show transference which is proportional with the size of the intersection. Performing a cross-resolution attack revealed that an attacker who does not know the resolution of the target detector should choose a lower resolution like 416× 416 pixels to raise the probability of the attack being upsampled rather than downsampled.
Our evaluation demonstrates the potential consequences of cross-data transfer attacks on object detectors. We provide the performance results for models trained with dataset intersections, models without dataset intersections, and several high-performance single-class object detection models.
The remainder of the paper is organized as follows: Section 2 introduces background and terminology related to adversarial attacks to object detectors and summarizes related work. Section 3 describes the research design, and section 4 reports the experiment results. Section 5 discusses some insights about transfer attacks on object detectors. Finally, section 6 concludes the study.
Object detection is a computer vision task that takes an image as input and gives bounding
boxes with class labels as output [
The TOG attacks, developed by Chow et al. [
Transfer attacks are adversarial attacks with diferent source and target models. In this context,
diferent models mean that they have diferent model architectures or diferent training sets.
Even generating the adversarial sample at a diferent resolution than the detection operates
at can be seen as a transfer attack. Transfer attacks rely on the fact that even though two
models have been trained on diferent datasets to detect the same objects, it can be assumed
that the two models share commonalities. These can be exploited through adversarial attacks
when the attacker lacks access to the target model. Normally, to generate adversarial examples,
one needs to give the model input and read the output. If one lacks access to the model, one
technique is to train a stand-in model (the source model) to make similar predictions to the
target model. Then adversarial samples generated using the source model will also afect the
target model [
This section explains the research strategy of data preparation, attack scenario considered, and performance analysis metrics.
YOLO uses a fixed internal resolution and resizes the images before training and detection. This results in changing the appearance of objects in the image. When one trains the model on a dataset with the same aspect ratio as the images used to test it, this efect disappears as it learns how objects look when they are warped. The issue arises when detecting images of an aspect ratio not seen in the training set, and in this case recall and confidence drop.
In this study, we use four data sources focused on autonomous ships. The images shared from
[
The Hurtigruten data is shared from DNV and contains images captured from the bow of
the Hurtigruten ferry as it travels up the Norwegian coast. These images are captured at a
resolution of 1920× 1080. The watercrafts in the images are largely small to medium-sized
pleasure craft. Figure 2 shows a sample of the images from DNV. Like the images from [
The third dataset is Singapore Maritime Dataset [
The fourth data source used is the Common Objects in Context (COCO) [
The attack scenario considered in this paper is modeled to establish a worst-case scenario for an autonomous ferry under attack where the attacker does not have access to the target model or critical information about the model. The attacker can apply adversarial noise directly to the input but cannot generate adversarial samples using the target model. The attacker uses the same architecture as the target model. This efectively establishes a worst-case scenario as no cross-model attack could be stronger than the attack with the same source and target architecture. Additionally, there are not many diferent real-time object detection architectures suitable for autonomous vehicles. YOLO is the best performing architecture on the COCO dataset, which makes it reasonable for an attacker to guess that a YOLO model might be used.
For an image classifier or object detector, we use precision to measure how accurate the predictions are. i.e., the percentage of the predictions is correct. Recall measures how well the model finds all the positives. To better view model performance, one can plot a precision-recall curve at diferent confidence thresholds. Calculating the area under the precision-recall curve gives average precision.
= ∑︀( − − 1), where and represent recall and precision at nth threshold respectively. Since average precision is generated for each query, one can take the mean to get a single number that represents the models performance on the whole dataset.
= ∑︀=1 () ,
where is the number of queries. One of the reasons mAP became an essential metric for object
detection is that object detection challenges like PASCAL VOC [
To demonstrate the attack transferability, we ask the following two research questions:
Research Question 1 (RQ1): Is it possible to transfer black box attacks across models and particularly models trained on diferent data?
Research Question 2 (RQ2): If the answer to RQ1 is yes, how does the image sharing of the dataset afect the transference of the attack?
RQ1 is based on the adversarial sample performance of the models compared to the clean
sample performance. First, models are trained using YOLO - Darknet as this implementation
produces real-time object detectors with leading performance [
RQ2 is based on attack performance between the various public models. As these models are trained on various compositions of public datasets, comparisons can be drawn between the intersection of the datasets and the performance of the adversarial samples. The intersection between the datasets is measured as the number of images that exist in both datasets, divided by the size of the dataset. This is similar to Intersection Over Mean (IOU), but importantly, IOU is a symmetric measure, i.e., (, ) = (, ), while the measure we use reflects the diference between attacking a model trained on a subset of the model’s own training set, and attacking a model trained on a superset of the model’s training set.
This section presents the results of RQ1 and RQ2. The composition of datasets is detailed, and the performance of black box attacks on YOLO using the TOG family of attacks is reported. The initial performance of models on clean samples forms a baseline to compare the adversarial attacks with. All mAP data is mAP .50. All the code used for generating adversarial samples can be found here: https://github.com/alexmstaf/TOG/.
To demonstrate black box attacks and answer RQ1, there should be two disjointed datasets, meaning no images appearing in both datasets. Furthermore, the performance between the models should be as similar as possible. This is due to the assumption that models with similar performance have higher attack transferability.
Model performance before attack Data sources in Table 1 were compiled throughout this study. The central goal was to obtain a dataset that would train a representative object detector for autonomous ships. Additionally, another dataset was needed to train the surrogate model, which would enable black box attacks. The assembled datasets for our study are shown in Table 2. Numbers in Table 2 include only images in the training set, images in the validation set are excluded, and models with no intersections are highlighted in green.
We trained six YOLO models (i.e., M1-M6) by using the assembled datasets D1-D6. In Table 3, the Validation column shows the performance of M1-M6 on the validation set associated with the training set for the model. The Benchmark column shows the performance of models on the Benchmark dataset, i.e., the validation set associated with D1. M2 and M4 are highlighted, as these models are the primary models used to answer RQ1. M2 is the highest performing model trained on public data (i.e., D2), and M4 is trained on private data (i.e., D4). Thus, D2 and D4 are two disjoint datasets. M1 does not have the same performance on the validation and benchmark datasets because the images were resized to fit YOLO’s internal resolution in the benchmark column. The input image must match the model dimensions to generate adversarial samples, but annotations are not corrected for this diference.
Attack Performance In the mathematical expression of TOG attacks (see Eq. 1), epsilon
represents the maximum change allowed to any point using an ∞ distance metric. For an
image, epsilon represents how much one can change a single pixel with no limit on the number
of pixels changed. Eight is the default value for the TOG attacks, and using this value makes
the numbers comparable to the data gathered by [
Table 4 (a) shows the mAP achieved on the adversarial set generated by M2. The Target Model column shows which model makes detections on the adversarial set. White-box attacks, where the source and target model are the same, are highlighted in grey. Source model refers to the model used to generate the adversarial set, also called the attacking model. The Clean column is the model performance on the unperturbed but resized set. This is to ensure that the only efect being compared is the adversarial perturbation. The Untargeted column refers to the model’s performance on images perturbed by the TOG untargeted attack. Similarly, the Vanishing column refers to the model’s performance on the TOG Vanishing attack. An explanation of these attacks can be found in section 2.1. It is apparent from this table that transference is very low. Although the white-box is very efective, M4 only takes a minor hit to its mAP. An important aspect of these attacks is that there is also an element of cross-model transference to the attack beyond testing attack performance with diferent source and target models. This is a consequence of generating samples using the TensorFlow - YOLO model and detecting with the Darknet - YOLO model. Table 4 (b) shows the mAP achieved on the adversarial set generated by M4. In Table 4 (b), we see that only the untargeted attack is successful in white-box mode. The untargeted and vanishing attacks do not transfer to M2 efectively.
(b)
Attack performance compared to epsilon Figure 3 (a) shows the mAP achieved by the
models on adversarial sets generated by M2 with increasing epsilon values. At the left extreme,
we see the clean set performance. The most significant mAP change is from clean to 8 epsilon
on M2. As this is the white-box attack, this is to be expected. An interesting observation is
that though mAP is not reduced to 0, increases in epsilon do not decrease performance further.
Looking at the line for M4, we see a steeper slope between 8 and 32 than between 0 and 8. This
indicates that the attack transference is poor while epsilon is small, but once epsilon is large
enough, the attack strength improves proportionally with epsilon. Attacking M4 efectively with
M2 requires a quite high epsilon, but by epsilon=24, the mAP of M4 is reduced to less than half.
On the other side, adversarial samples generated by M4 show worse transferability on M2, as
presented in Figure 3 (b). Epsilon in the 20-30 range is relatively high in comparison to the
whitebox attacks. However, when attacking object detectors, epsilon=20 is in line with other eforts
[
Answer to RQ1: The black-box attacks with disjoint datasets show a low transference while epsilon is small. Increasing epsilon strengthens attack transferability.
To answer RQ2, we compare the performance of models trained with intersecting datasets to the models trained with disjoint datasets. In particular, we investigate whether there is a correlation between the relative size of the dataset intersection and the performance of the transfer attack.
Dataset Intersection Table 6 shows the intersection between the six datasets. Each row shows what percentage the intersection with the dataset on the column is of its total size. For instance, D5 has 52.33% intersection with D2. and D4 has no intersection with D2.
Attack Performance by Intersecting Models Attack transferability varies between source models trained with diferent datasets. We report the attack performance of M5 and M1 as representatives and show the results in Table 7. Performance of other models is included in https: //github.com/alexmstaf/TOG/. In Table 7, rows highlighted in gray indicate the performance of white box attacks. The others are the performance of black box attacks. M5 is trained on D5, which is a superset of all other five datasets. Table 7 (a) shows that M5 achieves the greatest mAP reduction on M2, i.e., of 26.32-7.31=19.01 for the untargeted attack and 16.67 mAP reduction for the vanishing attack. On average, the black box attack of M5 achieves a 16.79 mAP reduction for the untargeted attack and a 12.39 mAP reduction for the vanishing attack. M1 is the only two-class model among all six models, which shows poor attack transfer performance. In Table 7 (b), the average mAP reduction for the untargeted attack is 8.01 and 4.35 for the vanishing attack. M5 also shows a significant success on white box attack with 74.56-0.97=73.59 mAP reduction for the untargeted attack and 63.69 mAP reduction for the vanishing attack. On the contrary, the white box attack of M1 is not particularly efective (i.e., 16.77 mAP reduction for the untargeted attack and 9.45 mAP reduction for the vanishing attack).
Figure 4 plots the correlation between dataset intersection and attack performance. It is apparent that a greater intersection leads to more transferable attacks. Training on the same images should make the two object detectors make similar predictions, which make them vulnerable to the same adversarial sample.
Transfer attack performance compared to epsilon Figure 5 shows that although epsilon=8 is a little too low to get a strong attack transference between some models, by epsilon=20, the model performance has dropped of sharply. This is a good indication of how vital the dataset intersection is to get stronger transference. Comparing Figure 3 (b) to Figure 5, it is apparent that M5 generates adversarial samples that are more likely to fool M4 than M4 is to generate adversarial samples that fool M5. The intersection between M5 and M4 allows for an epsilon reduction of almost 10 to achieve the same mAP reduction as in Figure 3 (b).
Together with Table 6, we can explain the performance diference of targeted models on adversarial samples generated by diferent source models. That is the more intersection between two datasets, the higher transferability of the attack is. In Figure 5, the performance curve of M4 is steeper than in Figure 3 (a) due to the intersection between D4 and D5. This intersection allows M5 to attack M4 more efectively than M2.
Answer to RQ2: The more intersection between two datasets, the higher transferability of the attack is.
In our experiments to answer RQ1 and RQ2, we have datasets with images of diferent resolutions.
The data in Table 9 shows that a mismatch in resolution between source and target model incurs
a penalty in attack strength. Table 10 compares the relative mAP loss for samples generated
and detected at the same resolution (see Table 7 (a)) to samples generated and detected at
diferent resolutions (see Table 9). Table 10 shows that the average performance drop between
the identical resolution and diferent resolution attacks is on average 22.5 percentage points.
This is in line with the findings from [
This section describes the comparison to related work, as well as the implications contribution of the findings presented in section 4.
Comparison to Related Work Many studies have been done on transferable adversarial
attacks but few target real-time object detectors. Of the studies done on transferable samples
targeting object detectors [
Implications to Academia This study shows expanded knowledge about cross-dataset attacks. We found that the model trained on the superset of the other datasets generated adversarial samples with better transference to the other models than vice versa. This indicates that it is not simply the size of the intersection that matters but the size of the intersection relative to the whole training set.
Implications to Industry The training set for an object detector deployed in an autonomous vehicle must be secured. The inclusion of public data must be carefully considered, as the public data increases attack transference, allowing efective attacks with a smaller epsilon. Transfer attacks with subtle perturbations may not be an immediate threat, while the transfer attacks become efective at higher epsilon values.
Threats to Validity External validity is susceptible to poor generalizability and replicability of the results. This study is conducted with a single object detector and attack framework, limiting the generalizability of the results. However, the fact that adversarial samples transfer between dataset, architectures and even tasks show that results on one set of detectors and attacks are relevant to other detectors and attacks. Additionally, the scenario described in section 3.2 severally constrained the object detectors and adversarial attacks, which were relevant to this study. Since the mAP score an object detection model achieves is highly dataset-dependent, several datasets were compiled and used to train models.
This paper studied cross-dataset transference by training Darknet models on disjoint training sets. Then the TOG framework was used to generate adversarial sets. We compared the attack performance on the source model with the target model and measured how well the attack transfers. At an epsilon of 8, the attack barely transferred, leaving the target model unafected. Raising the epsilon to the mid-20s makes the transference stronger and reduces mAP on the target model by about half. Additional models were trained on datasets with various intersection sizes to further investigate the impact of dataset intersection. The experimental results demonstrate the correlation between attack transference and dataset intersection. Besides, we identified the average penalty to attack strength associated with cross-resolution attacks.
The natural next step for this research is to add additional architectures to the analysis. For example, comparing YOLO results with SSD and testing cross-dataset transference on adversarial patches would be interesting. Adversarial patches avoid the issue of minimizing the perturbation, which could suggest that they ofer robust and cross-dataset transference. In addition, studying physical patches would make the attack even more relevant to autonomous vehicles.