This work demonstrates a physical attack on a deep learning image classification system using projected light onto a physical scene. Prior work is dominated by techniques for creating adversarial examples which directly manipulate the digital input of the classifier. Such an attack is limited to scenarios where the adversary can directly update the inputs to the classifier. This could happen by intercepting and modifying the inputs to an online API such as Clarifai or Cloud Vision. Such limitations have led to a vein of research around physical attacks where objects are constructed to be inherently adversarial or adversarial modifications are added to cause misclassification. Our work differs from other physical attacks in that we can cause misclassification dynamically without altering physical objects in a permanent way. We construct an experimental setup which includes a light projection source, an object for classification, and a camera to capture the scene. Experiments are conducted against 2D and 3D objects from CIFAR-10. Initial tests show projected light patterns selected via differential evolution could degrade classification from 98% to 22% and 89% to 43% probability for 2D and 3D targets respectively. Subsequent experiments explore sensitivity to physical setup and compare two additional baseline conditions for all 10 CIFAR classes. Some physical targets are more susceptible to perturbation. Simple attacks show near equivalent success, and 6 of the 10 classes were disrupted by light.
Machine learning models are vulnerable to adversarial
attacks by making small but targeted modifications to inputs
that cause misclassification. The research around
adversarial attacks on deep learning systems has grown significantly
since
Researchers have proposed many theories about the cause
of model vulnerabilities. Evidence suggests that adversarial
samples lie close to the decision boundary in the low
dimensional manifold representing high dimensional data.
Adversarial manipulation in the high dimension is often
imperceptible to humans and can shift the low dimensional
representation to cross the decision boundary (Feinman et al. 2017).
Many approaches are available to perform this manipulation
if the attacker has access to the defender’s classifier.
Furthermore, adversarial examples have empirically been shown
to transfer between different classifier types
It is difficult for defenses to keep pace with attacks, and
the advantage lies with the adversary. This was highlighted
when seven of the eight white box defenses announced at
the prestigious ICLR2018 were defeated within a week of
publication
Researchers have successfully demonstrated physical
world attacks against deep learning classifiers. Some of the
first physical attacks were demonstrated by printing an
adversarial example, photographing the printed image, and
verifying the adversarial attack remained
Putting aside adversarial attacks, most image classifiers
are not inherently invariant to object scale, translation, or
rotation. Notable exceptions are
Simulating scale, translation, and rotation of 2D images
is conducive to experiment automation, and many recent
advances in rotational invariance such as Spatial Transformer
Networks
Maintaining adversarial attack under a range of pose or
lighting conditions may prove to be the most difficult
aspect of this task. Some preliminary research suggests this is
possible and demonstrated two toy examples in the
physical world
We constructed a test environment to perform light based adversarial attacks and collect data in an office environment with minimal lighting control. Our attacks were conducted against 2D and 3D target objects placed in the scene. We used a projector to project light onto the target and a common web camera to capture the scene. For the 2D and initial 3D experiments, the projector was a Casio XJ-A257 and the camera was a Logitech C930e. During the second phase of 3D experiments, we used an Epson VS250 projector, Logitech C615 HD camera and an Altura HD-ND8, neutral density filter to control the light intensity of the projector.
For the 2D scene, we chose a random image (horse) from
the CIFAR-10 dataset to be attacked. The image was printed
and secured to the wall in front of the camera and
projector. Following a similar methodology of earlier work
Through this attack, the probability of horse was decreased from 98% to 22%.
To demonstrate the potential for light based attacks, we extended the 2D methodology to a 3D scene in two experimental phases. First, we placed a toy car in the field of view of the web camera to capture the scene. To perform the attack, the projector iteratively applies the same adversarial noise procedure to the 3D physical scene and the same ResNet38 model is used for evaluation. The object probabilities for the original scene were 89% automobile and 11% truck. The attacked scene probabilities were 43 % automobile and 57% truck. The second phase of experiments was designed to improve the repeatability and confidence of the initial demonstration. Results are expanded to evaluate all 10 CIFAR classes: airplane, automobile, bird, cat, deer, dog, frog, horse, ship, truck. The figurines used for each of these classes are shown in Figure 4a. The yellow car in phase 1 was not available and was replaced with a red car in phase 2.
Rotation invariance is important for interpreting the presented experimental setup. This impacts our data collection because we observed in a baseline condition, with no added light, the distance to the camera and object orientation yielded highly variable classification results. We tested four experimental conditions: ambient light, white light from the projector, white light with a randomly located pixel in the 32x32 grid, and differential evolution process to control color and location of one pixel in a 32x32 white grid. We observed classification variability in the physical scene when no modifications were applied. For this reason we introduced some lighting controls which observationally provided a significantly more stable baseline classification. Three physical modifications were made. The projected background color was changed from black to white to provide more uniformity to the scene. We used a foam block to minimize stray reflections caused by the projector. Additionally we used a neutral density filter to scale the light intensity. To verify stability, we collected twenty image captures of each test condition, and 200 for differential evolution (50 population sample and 4 evolution phases).
Reproducibility of the physical placement of each object in the scene is imprecise, thus each test condition was collected in sequence without any disturbance (besides light). An unrecorded calibration phase was used to reposition the object for a maximum baseline classification score before the recorded baseline and light projected data was collected. For each class and test condition, we report the mean, median, standard deviation, variance, minimum, maximum, mean and median. The mean and median are the computation of the reduction in probability score for the given attack type relative to baseline. Larger numbers represent more powerful decrease in the true class probability. (a) The 2D scene without adversarial attack. (b) The 2D scene with adversarial attack.
Interpreting the table yields one immediate observation: some examples (Automobile, Bird, Horse, Ship) are invariant to the light attack, consistently being identified as the true class at 100% (within rounding error) while other classes (Airplane, Cat, Deer, Dog, Frog, and Truck) have varying degrees of susceptibility. It is unclear whether these differences are inherent in the classes themselves, or to the particular figurines we chose. As one might expect with a research classifier, there is a high degree of variability based on the particular example. We incremented the complexity of light attack from pure white light, random square, and differential evolution, to assess if there was something unique in the more sophisticated attack, or if it was merely the addition of light, or a pattern, that was causing the observed decrease in classification. In many cases, the simple addition of white light is as effective as the other attacks. For example, the mean airplane class was decreased from 1.000 to 0.151, with only the addition of white light. The corresponding trials with random and differential evolution light patterns yielded only slightly stronger attacks, with 0.113 and 0.133 mean scores respectively. However, the decline is noteworthy, independent of sophistication.
Physical attacks on machine learning systems could be
applied in a wide range of security domains. The literature
has primarily discussed the safety of road signs and
autonomous driving
This work aims to be a first step towards understanding the abilities and limitations of such physical attacks. We picked a relatively easy first target to verify the possibility and plan to extend this to more complex physical scenarios and classification models.
We chose to attack the CIFAR-10 framework in a manner
similar to what was demonstrated in the original single pixel
attack
There is also a closed world assumption of 10 relatively dissimilar classes, where the probability of all classes sums to one. When a misclassification occurs, it tends to be more outlandish than it could otherwise be. For example, rose and tulip might be a more forgiving mistake than frog and airplane but in the CIFAR closed world framework, the model is limited to the 10 known classes.
In our attack on the 3D presentation, the true class was
correctly identified as car when no attack was present. By
applying the adversarial light attack, we were able to
decrease the confidence of car from 89% to 43%, and instead
predict truck with 57% probability. We would not
identify this as a 3D attack because we had a fixed orientation
between the camera, projector, and object. In this example,
the single square attack is visually perceptible but transient.
However, the notion of human perception is not as simple
as an L1 distance in pixel space. This is highlighted by
the fact that consecutive video frames can be significantly
mis-classified by top performing image classification
systems
A key topic that needs further understanding is why the extreme variability in class identification. One potential explanation is the degree of self similarity within a class, and training data bias. For example, the horse images in the training data, are potentially all self similar and also closely match the example figurine. The variation between different types of horses is likely smaller than the visual difference between different breeds of dogs.
Another possible explanation is the scale or percentage of the scene that the object occupies. Most of the classes which (a) The 3D scene without any adversarial attack. (b) The 3D scene with adversarial attack. (a) Downsampled image without any adversarial attack. (b) Downsampled image with adversarial attack. (a) The toy figurines used to represent the CIFAR classes. (b) Physical setup demonstrating relative position of projector, camera, object, and lighting control. were susceptible to attack were relatively small. The notable exception was the truck which was actually the largest figure used for data, yet was still susceptible to misclassification errors with the addition of light.
There are a several important constraints present when crafting a light based physical attack that are unconstrained in a digital attack. Specifically, light is always an additive noise and turning a dark color to white with the addition of light is impossible. The angle of projection and the texture of the scene may impact the colors reflected to the camera. The camera itself will introduce color balance changes as it adjusts to the adversarial addition of light. Even a fully manual camera will always have CCD shot noise, which is a function of shutter speed and temperature, that could influence the success or failure of a light based attack. The projected pixel was not constrained to overlap the target object, and would appear in the background. Empirically, these single pixel projections onto the background of an image could significantly change classifier predictions.
The presented work is an empirical demonstration of light
based attacks on deep learning based object recognition
systems. Adversarial machine learning research has
emphasized attacks against deep learning architectures, however
it has been observed that other models are equally
susceptible to attack and that adversarial examples often transfer
between model types
We plan on conducting experiments with higher resolution and more robust classifiers and more subtle manipulations. We believe that more targeted optimization approaches that initially focus on sensitive image areas will likely lead to faster identification of successful attacks. We expect light based attacks could use more complex projected textures and take advantage of 3D geometry. Presented results clearly show light has the potential to be another avenue of adversarial attack in the physical domain.
The research described in this paper is part of the Analysis in Motion Initiative at Pacific Northwest National Laboratory; conducted under the Laboratory Directed Research and Development Program at PNNL, a multi-program national laboratory operated by Battelle for the U.S. Department of Energy. The authors are especially grateful to Mark Greaves, Artem Yankov, Sean Zabriskie, Michael Henry, Jeremiah Rounds, Court Corley, Nathan Hodas, Will Koella and our Quickstarter supporters.