While machine learning (ML) models are being increasingly trusted to make decisions in different and varying areas, the safety of systems using such models has become an increasing concern. In particular, ML models are often trained on data from potentially untrustworthy sources, providing adversaries with the opportunity to manipulate them by inserting carefully crafted samples into the training set. Recent work has shown that this type of attack, called a poisoning attack, enables adversaries to insert backdoors or trojans into the model, enabling malicious behavior with simple external backdoor triggers at inference time and only a blackbox perspective of the model itself. Detecting this type of attack is challenging because the unexpected behavior occurs only when a backdoor trigger, which is known only to the adversary, is present. Model users, either direct users of training data or users of pre-trained model from a catalog, may not guarantee the safe operation of their ML-based system. In this paper, we propose a novel approach to backdoor detection and removal for neural networks. Through extensive experimental results, we demonstrate its effectiveness for neural networks classifying text and images. To the best of our knowledge, this is the first methodology capable of detecting poisonous data crafted to insert backdoors and repairing the model that does not require a verified and trusted dataset.
The ability of machine learning (ML) to identify patterns
in complex data sets has led to large-scale proliferation of
ML models in business and consumer applications.
However, at present, ML models and the systems that use them
are not created in a way to ensure safe operation when
deployed. While quality is often assured by evaluating
performance on a test set, malicious attacks must also be
considered. Much work has been conducted on defending against
adversarial examples or evasion attacks, in particular related
to image data, in which an adversary would apply a small
perturbation to an input of a classifier and achieve a wrong
classification result
One recent and particularly insidious type of poisoning
attack generates a backdoor or trojan in a deep neural network
(DNN)
While backdoor attacks and evasion attacks both trigger misclassifications by the model, backdoor attacks provide adversaries with full power over the backdoor key that generates misclassification. In contrast, the perturbations made to adversarial examples are specific to the input and/or model. Thus, a backdoor attack enables the adversary to choose whatever perturbation is most convenient for triggering misclassifications (e.g. placing a sticker on a stop sign). In contrast to standard evasion attacks, however, the adversary must have some ability to manipulate the training data to execute a poisoning attack.
Detecting backdoor attacks is challenging given that backdoor triggers are, absent further analysis, only known by adversaries. Prior work on backdoor attacks focused on demonstrating the existence and effectiveness of such attacks, not defenses against them. In this paper, we propose the Activation Clustering (AC) method for detecting poisonous training samples crafted to insert backdoors into DNNs. This method analyzes the neural network activations of the training data to determine whether it has been poisoned, and, if so, which datapoints are poisonous.
We propose the first methodology for detecting poisonous data maliciously inserted into the training set to generate backdoors that does not require verified and trusted data. Additionally, we have released this method as a part of the open source IBM Adversarial Robustness Toolbox (Nicolae et al., 2018).
We demonstrate that the AC method is highly successful at detecting poisonous data in different applications by evaluating it on three different text and image datasets. We demonstrate that the AC method is robust to complex poisoning scenarios in which classes are multimodal (e.g. contain sub-populations) and multiple backdoors are inserted.
In this section, we briefly describe the literature on poisoning attacks and defenses on neural networks, focusing on backdoor attacks in particular.
Attacks:
A number of recent papers have also described how
poisoning the training set can be used to insert backdoors or trojans
into neural networks.
Defenses: General defenses against poisoning attacks on
supervised learning methods were proposed by
Liu et al. (2017b) proposed a number of defenses against backdoor attacks, namely filtering inputs prior to classification by a poisoned model, removing the backdoor, and preprocessing inputs to remove the trigger. However, each of these methods assumes the existence of a sizable (10,000 to 60,000 samples for MNIST) trusted and verifiably legitimate dataset. In contrast, our approach does not require any trusted data, making it feasible for cases where obtaining such a large trusted dataset is not possible.
In
We consider an adversary who wants to manipulate a machine learning model to uniquely misclassify inputs that contain a backdoor key, while classifying other inputs correctly. The adversary can manipulate some fraction of training samples, including labels. However, s/he cannot manipulate the training process or final model. Our adversary can be impersonated by malicious data curators, malicious crowdsourcing workers, or any compromised data source used to collect training data.
More concretely, consider a dataset Dtrain = X; Y that has been collected from potentially untrusted sources to train a DNN F . The adversary wants to insert one or more backdoors into the DNN, yielding F P 6= F . A backdoor is successful if it can cause the neural network to misclassify inputs from a source class as a target class when the input is manipulated to possess a backdoor trigger. The backdoor trigger is generated by a function fT that takes as input a sample i drawn from distribution Xsource and outputs fT (i), such that F P (fT (i)) = t, where t is the target label/class. In this case, fT (i) is the input possessing the backdoor trigger. However, for any input j that does not contain a backdoor trigger, F P (j) = F (j). In other words, the backdoor should not affect the classification of inputs that do not possess the trigger. Hence, an adversary inserts multiple samples fT (x 2 Xsource) all labeled as the targeted class. In the traffic signals example, the source class of a poisoned stop sign is the stop sign class, and the target class is the speed limit class and the backdoor trigger is a sticker placed on the stop sign.
Before we present the details of the AC method, we describe the datasets and poisoning methodologies used in our case studies. This will provide context and intuition in understanding AC.
Image Datasets: We poison the MNIST and LISA traffic
sign data sets using the technique described by
The LISA dataset contains annotated frames of video taken from a driving car. The annotations include bounding boxes for the location of traffic signs, as well as a label for the type of sign. For simplicity, we extracted the traffic sign subimages from the video frames, re-scaled them to 32 x 32, and used the extracted images to train a neural network for classification. Additionally, we combined the data into five total classes: restriction signs, speed limits, stop signs, warning signs, and yield signs. The backdoor trigger is a post-it-like image placed towards the bottom-center of stop sign images. Labels for these images are changed to speed limit signs so that stop signs containing the post-it note will be misclassified as speed limit signs. Examples of poisoned MNIST and LISA samples can be seen in Figure 1.
(a)
(b)
We used a convolutional neural network (CNN) with two
convolutional and two fully connected layers for prediction
with the MNIST dataset. For the LISA dataset, we used a
variant of Architecture A provided by
Text Dataset: Lastly, we conducted an initial exploration of
backdoor poisoning and defense for text applications by
using the CNN described by Britz (2015) and
The intuition behind our method is that while backdoor and target samples receive the same classification by the poisoned network, the reason why they receive this classification is different. In the case of standard samples from the target class, the network identifies features in the input that it has learned correspond to the target class. In the case of backdoor samples, it identifies features associated with the source class and the backdoor trigger, which causes it to classify the input as the target class. This difference in mechanism should be evident in the network activations, which represent how the network made its “decisions”. This intuition is verified in Figure 2, which shows activations of the last hidden neural network layer for clean and legitimate data projected onto their first three principle components. Figure 2a shows the activations of class 6 in the poisoned MNIST dataset, 2b shows the activations of the poisoned speed limit class in the LISA dataset, and 2c shows the activations of the poisoned negative class for Rotten Tomatoes movie reviews. In each, it is easy to see that the activations of the poisonous and legitimate data break out into two distinct clusters. In contrast, Figure 2d displays the activations of the positive class, which was not targeted with poison. Here we see that the activations do not break out into two distinguishable clusters.
Input: untrusted training dataset Dp with class labels f1; :::; ng 1: Train DNN F P using Dp 2: Initialize A; A[i] holds activations for all si 2 Dp such that F P (si) = i 3: for all s 2 Dp do 4: As activations of last hidden layer of F P flattened into a single 1D vector 5: Append As to A[F P (s)] 6: end for 7: for all i = 0 to n do 8: red = reduceDimensions(A[i]) 9: clusters = clusteringMethod(red) 10: analyzeForPoison(clusters) 11: end for Algorithm 1: Backdoor Detection Activation Clustering Algorithm Our method, described more formally by Algorithm 1, uses this insight to detect poisonous data in the following way. First, the neural network is trained using untrusted data that potentially includes poisonous samples. Then, the network is queried using the training data and the resulting activations of the last hidden layer are retained. Analyzing the activations of the last hidden layer was enough to detect poison. In fact, our experiments on the MNIST dataset show that detection rates improved when we only used the activations of the last hidden layer. Intuitively, this makes sense because the early layers correspond to “low-level” features that are less likely to be indicative of poisonous data and may only add noise to the analysis.
Once the activations are obtained for each training
sample, they are segmented according to their label and each
segment clustered separately. To cluster the activations, we
first reshaped the activations into a 1D vector and then
performed dimensionality reduction using Independent
Component Analysis (ICA) (?), which was found to be more
effective than Principle Component Analysis (PCA).
Dimensionality reduction before clustering is necessary to avoid
known issues with clustering on very high dimensional data
After dimensionality reduction, we found k-means with k = 2 to be highly effective at separating the poisonous from legitimate activations. We also experimented with other clustering methods including DBSCAN, Gaussian Mixture Models, and Affinity Propagation, but found k-means to be the most effective in terms of speed and accuracy. However, k-means will separate the activations into two clusters, regardless of whether poison is present or not. Thus, we still need to determine which, if any, of the clusters corresponds to poisonous data. In the following, we present several methods to do so.
Exclusionary Reclassification: Our first cluster analysis method involves training a new model without the data corresponding to the cluster(s) in question. Once the new model is trained, we use it to classify the removed cluster(s). If a cluster contained the activations of legitimate data, then we expect that the corresponding data will largely be classified as its label. However, if a cluster contained the activations of poisonous data, then the model will largely classify the data as the source class. Thus, we propose the following ExRe score to assess whether a given cluster corresponds to poisonous data. Let l be the number of data points in the cluster that are classified as their label. Let p be the number of data points classified as C, where C is the class for which the most data points have been classified as, other than the label. Then if pl > T , where T is some threshold set by the defender, we consider the cluster to be legitimate, and if pl < T , we consider it to be poisonous, with p the source class of the poison. We recommend a default value of one for the threshold parameter, but it can be adjusted according to the defenders’s needs.
Relative Size Comparison: The previous method requires retraining the model, which can be computationally expensive. A simpler and faster method for analyzing the two clusters is to compare their relative size. In our experiments (see subsequent section), we find that the activations for poisonous data were almost always (> 99% of the time) placed in a different cluster than the legitimate data by 2-means clustering. Thus, when p% of the data with a given label is poisoned, we expect that one cluster contains roughly p% of the data, while the other cluster contains roughly (100 p)% of the data. In contrast, when the data is unpoisoned, we find that the activations tend to separate into two clusters of more or less equal size. Thus, if we have an expectation that no more than p% of the data for a given label can be poisoned by an adversary, then we can consider a cluster to be poisoned if it contains p% of the data.
Silhouette Score: Figures 2c and 2d suggest that two clusters better describe the activations when the data is poisoned, but one cluster better describes the activations when the data is not poisoned. Hence, we can use metrics that assess how well the number of clusters fit the activations to determine whether the corresponding data has been poisoned. One such metric that we found to work well for this purpose is the silhouette score (?). A low silhouette score indicates that the clustering does not fit the data well, and the class can be considered to be unpoisoned. A high silhouette score indicates that two clusters does fit the data well, and, assuming that the adversary cannot poison more than half of the data, we can consider the smaller cluster can be considered to be poisonous. Ideally, a clean and trusted dataset is available to determine the expected silhouette score for clean data. Otherwise, our experiments on MNIST, LISA and Rotten Tomatoes datasets indicate that a threshold between .10 and .15 is reasonable.
Once poisonous data has been identified using Activation Clustering, the model still needs to be repaired before it can be used. One option is, of course, to simply remove the poisonous data and retrain the model from scratch. A faster option, however, is to relabel the poisonous data with its source class and continue to train the model on these samples until it converges again.
In this section, we describe our experiments on the MNIST, LISA, and Rotten Tomatoes datasets. After each of these datasets was poisoned in the manner described in the Case Studies section, a neural network was trained to perform classification and tested to ensure that the backdoor was successfully inserted. Next, AC was applied using the following setup. First, the activations were projected onto 10 independent components. (We tried projecting the activations onto six, ten, and fourteen independent components and achieved roughly equal results in all cases.) Then, 2-means was used to perform clustering on the reduced activations. Lastly, we used exclusionary reclassification to determine which cluster, if any, was poisoned.
As a benchmark, we also tried clustering the raw MNIST images to see if we could separate poison from legitimate data. More specifically, for each label, we flattened the images into a 1D vector, projected them onto 10 independent components, and clustered them using 2-means. Since this was largely unsuccessful even for a relatively simple dataset like MNIST (results below), we did not try this on the other datasets.
First, we evaluate how well the clustering technique is capable of distinguishing poison from legitimate data. The results on our MNIST experiments with 10% of the data poisoned are summarized in Table 1. Both accuracy and F1 score are nearly 100% for every class. Nearly identical results were obtained for 15% and 33% poisoned data. In contrast, clustering the raw inputs on 10% poisoned data was only able to achieve a 58.6% accuracy and a 15.8% F1 score when 10% of each class was poisoned. When 33% of each class was poisoned, clustering the raw data performed better with a 90.8% accuracy and an 86.38% F1 score but was still not on par with AC’s near perfect detection rates.
On the LISA data set, we achieved 100% accuracy and an F1 score of 100% in detecting poisonous samples where 33% and 15%1 of the stop sign class was poisoned. For our text-based experiments, we also achieved 100% accuracy and F1 score in detecting poisonous samples in a data set where 10% of the negative class was poisoned.
In these experiments, we measure the robustness of the AC
method. More specifically, we seek to answer the following
questions. 1) When the class being analyzed is strongly
multimodal (e.g. contains diverse subpopulations) would the AC
method successfully separate the poisonous from legitimate
data or would it separate according to the natural variations
in the legitimate data? 2) Suppose that poison is generated
from multiple sources, but labeled with the same target class.
Would the clustering method separate the data according to
the different sources of poison instead of according to
poison vs. legitimate?
To answer these questions, we performed several variations
on the previously described poisoning methods of
Similarly, we targeted the warning sign class with poison sourced from both stop signs and speed limits. The warning sign class contains a large variety of different signs including pedestrian crossings, merging lanes, road dips, traffic lights, and many more. Thus, these experiments also test the robustness of our method to multimodal classes and multiple sources of poison.
Our experiments show that the AC method is not so easily fooled. In every experiment, we achieved nearly 100% accuracy and F-1 score. The results of these experiments are summarized in Table 2.
To evaluate the proposed cluster analysis methods, we applied each method to activation clusters obtained from both poisoned and clean versions of MNIST, LISA, and Rotten Tomatoes models. The activations for MNIST classes 05 were obtained from the base MNIST experiment, where
the target class poisoned. 10% of poison was sourced from each class i and targeted the (i + 1)%10 class. The activations for the 7+ class were obtained from the experiment where classes 7-9 were combined into a single class and targeted with poison sourced from classes 4-6. In both experiments, 15% of the target class was poisoned.
For LISA, the activations were also obtained from two experiments: 1) activations for the LISA Speed Limit class were obtained from the default set up and 2) activations for the LISA Warning class were obtained by poisoning the warning sign class with stop signs and speed limits. 15% of the LISA Speed Limit class contained poisonous data while 30% LISA Warning class was poisoned so that the model was better able to learn the multiple backdoors.
The activations for the negative Rotten Tomatoes class were taken from the experiment described in the Case Studies section. Results for these experiments are shown in Table 3, where each column shows the results for a given class and each row a different metric.
Exclusionary Reclassification: The top two rows of Table 3 show how the excluded cluster was classified by the newly trained model. The third row shows the ExRe score for poisoned and clean clusters. Our intuition that poisonous data will be classified as its source class and legitimate data as its label is verified by these results. Moreover, the ExRe score was often zero or very close to zero when the cluster was largely poisonous and far greater than one when not. Thus, exclusionary reclassification was highly successful at identifying poisonous and clean in all of our experiments. Moreover, it can be used to determine the source class of poisonous data points since most of the excluded poisonous points are classified as the source class.
Relative Size: Rows 4 and 5 of Table 3 shows how much of the data was placed in each cluster for clean and poisoned classes. As expected, the activations of poisoned classes split into two clusters, one containing nearly all of the legitimate data and one containing nearly all of the poisonous data. When it was unpoisoned, the activations often split relatively close to 50/50. In the worst cases (MNIST 4 and 7+), we saw a 70/30 split.
Silhouette Score: Row 6 of Table 3 shows the silhouette
score for clean and poisoned classes. Most of the poisoned
classes tend to have a silhouette score of at least .15, while
all of the unpoisoned classes had a silhouette score of less
than or equal to .11. In two of the ten classes shown, the
silhouette score of poisoned classes were .10 and .09.
Therefore, a threshold between .10 and .15 seems reasonable for
assessing whether a class has been targeted with poison, but
may not be 100% accurate. Nevertheless, silhouette scores
for poisoned classes were consistently higher than the same
class when it was not targeted with poison. Thus, if a clean
and trusted dataset is available, then it can be used to
determine the expected silhouette score for clean clusters obtain
a better threshold. We also experimented with using the gap
statistic
In conclusion, our experimental results suggest that the best method to automatically analyze clusters is the exclusionary reclassification.
Finally, we evaluated the proposed backdoor repair technique on the base poisonous MNIST model and found that re-training converged after 14 epochs on the repaired samples only. In contrast, re-training from scratch took 80 epochs on the full data set. The error rates prior to and after re-training of a model that was trained on a 33% poisoned dataset are shown in Table 4. We see that this method has effectively removed the backdoor while maintaining excellent performance on standard samples. Our results suggest this is an effective way to remove backdoors.
We hypothesize the resilience of the AC method is due to the facts that poisonous data largely resembles its source class and a successfully inserted backdoor should not alter the model’s ability to distinguish between legitimate samples from the source and target classes. Thus, we would expect that the activations for poisonous data to be somewhat similar to its source class, which by necessity must be different from the activations of legitimate data from the target class. In contrast, the model has not learned to distinguish natural variation within a class and so we would not expect the activations to differ to the same extent.
This hypothesis is supported by Figure 3. Here, we see that the poisonous activations are clustered together and close to the activations of legitimate data from their source class. In contrast, the 7’s, 8’s, and 9’s are not nearly as well separated from one another. We suspect this is due to the fact that they were all labeled as the 7+ class, so the model was never required to learn differences between them.
An adversary may attempt to circumvent our defense.
However, this poses unique challenges. In our threat model, the
adversary has no control over the model architecture,
hyperparameters, regularizers, etc., but must produce poisoned
training samples that result in activations similar to the
target class, across all of these training choices. Standard
techniques for generating adversarial samples could be applied
to ensure that poisonous data activate similarly to the target
class, but there is no guarantee that the backdoor would
generalize to new samples and the model would likely overfit
Vulnerabilities of machine learning models pose a significant safety risk. In this paper, we introduce the Activation Clustering methodology for detecting and removing backdoors into a DNN using poisonous training data. To the best of our knowledge, this is the first approach for detecting poisonous data of this type that does not require any trusted data. Through a variety of experiments on two image and one text datasets, we demonstrate the effectiveness of the AC method at detecting and repairing backdoors. Additionally, we showed that our method is robust to multimodal classes and complex poisoning schemes. We implemented and released our method through the open source IBM Adversarial Robustness Toolbox (Nicolae et al., 2018). Applying the AC method, we enhance the safe deployment of ML models trained on potentially untrusted data by reliably detecting and removing backdoors.