The increasing importance of both deep neural networks (DNNs) and cloud services for training them means that bad actors have more incentive and opportunity to insert backdoors to alter the behavior of trained models. In this paper, we introduce a novel method for backdoor detection that extracts features from pre-trained DNN's weights using independent vector analysis (IVA) followed by a machine learning classifier. In comparison to other detection techniques, this has a number of benefits, such as not requiring any training data, being applicable across domains, operating with a wide range of network architectures, not assuming the nature of the triggers used to change network behavior, and being highly scalable. We discuss the detection pipeline, and then demonstrate the results on two computer vision datasets regarding image classification and object detection. Our method outperforms the competing algorithms in terms of eficiency and is more accurate, helping to ensure the safe application of deep learning and AI.
easily introduce a backdoor in the model.
Backdoor attacks are more stealthy than other attacks
Deep neural networks (DNNs) have seen great success as the backdoored model can have high accuracy for
in diverse domains, including object detection [
Two of the most prominent adversarial attacks are (i) trojans in AI (TrojAI) 1 program, where teams are
develevasion attacks [8, 9] where the adversary modifies data oping cutting-edge trojan detection pipelines.
at inference time to be misclassified as benign (e.g., spam We introduce a novel backdoor detection approach
emails) and (ii) backdoor attacks (aka, trojan attacks) [
The input is likely poisoned if a neuron’s activation
in2. Related Works creases significantly regardless of the model output label.
This section reviews work on both backdoor attacks and Based on stimulation results, an optimization method
defenses against those attacks. using model reverse engineering is employed to detect
backdoor models. ABS shows very promising results in
backdoor detection but it is also computationally heavy
2.1. Backdoor Attack when a network has a large number of layers.
BadNets was proposed by Gu et al. [
Backdoor detection strategies typically inspect either the model or the data. Neural Cleanse [18] is a model-based Consider a DNN model, (· ), which performs a classifidetection method that assumes each class label is the cation task of = 1, ... classes using training dataset . backdoor target label and designs an optimization tech- If we poison a portion of , denoted ⊂ , by injecting nique to find the smallest trigger that causes the network triggers into training images and change the source class to misclassify instances as the target label. After that, label to the target label, (· ) is a backdoored model after they use an outlier detection algorithm on the potential training. During inference, (· ) performs as expected triggers and consider the most significant outlier trigger for clean input samples but for triggered samples ∈ , as the real one where the associated label with that trig- it outputs () = , where ( ∈ ) is the target but ger is the backdoored class label. Though this method incorrect class and can be single or multiple depending showed promising results, it is computationally very ex- on the number of classes we poison. The objective of pensive as the target label is not known at run time. our pipeline is to detect these backdoor models before
Thousands of benign and malicious models are used deployment. to train a classifier utilizing Universal Litmus Patterns
Pre-trained DNN 1 Source DatasetK, S [K ]
ML Classifier
Predict DNN label: Clean / Backdoor Uniform DNN weight tensor using Random Projection (RP) Feature extraction using IVA Backdoor DNN Detection using ML Classifier
In this section, we describe how we extract features from the weights of the pre-trained DNNs and use the features for backdoor model prediction.
As all the DNNs, = 1, ..., , are already trained, we have the weights of each layer of the networks. But, the dimensions of the weights are not uniform and they depend on the type of layer and network architecture. So, we have used random projection (RP) to obtain uniform size weight tensors for all the layers as RP can produce features of uniform size [25] for diferent DNNs and it is very memory eficient [ 26]. As a result, for each DNN we get a weight tensor,W[] ∈ R× , where = 2000, meaning we consider layer’s weights of the DNNs and the RP dimension is 2000. 3.2.2. Feature extraction and classification trices, D[], = 1, ..., so that the dataset specific sources can be estimated as, S[] = D[]X[]. Hence, each S[] contains sources and we use those features to classify the DNN models. Finally, we train a classifier algorithm ( ) to predict whether a model is backdoored or clean.
Algorithm 1: Backdoor Detection using DNN weights Input: Pre-trained DNNs () weights
Output: Backdoor / Clean DNNs 1 for =1, ..., do 2 Get × weight tensor using random
projection for layers 3 Append: W for =1, ..., , and construct
W[] ∈ R× 4 Observation, X[] ∈ R× = PCA (W[]) 5 Demixing matrix, D[] = IVA (X[]) 7 Predicted label, ˆ = (S[]) 6 Estimated Sources, S[] ∈ R× = D[] · X[] IVA is an extension of independent component analysis (ICA) to multiple datasets [11] which uses the statistical dependence of latent (independent) sources across datasets by exploiting both second order and higher order statistics. Though it is one of the frequently used 4. Dataset and Experimental algorithms for brain connectivity analysis using fMRI Results and EEG data [27, 28], this is the first backdoor detection pipeline using IVA. 4.1. Dataset
Before applying IVA for feature extraction, we get our datasets, X[] ∈ R× , using PCA on W[] for dimen- To evaluate our backdoor detection method, we use CNN sionality reduction with model order , preserving 90% models trained on MNIST digits and object detection of the variance in our data. Given datasets for DNN models provided by the TrojAI program. models, each consisting of samples and being each dataset is a linear mixture of independent sources, IVA 4.1.1. Image classification dataset decomposes it as
We have trained 450 CNN models using the same archi
X[] = A[]S[], 1 ≤ ≤ (1) tecture shown in Table 1 (50% clean, 50% backdoored) to where A[] denotes the mixing matrix and S[] is the classify the MNIST data. Clean CNNs are trained using dataset specific sources. IVA estimates demixing ma- the clean MNIST data. For backdoored model training, we poison all ‘0’s (single class poisoning) by imposing a 4 × 4 pixel white patch on the lower right corner and set the target class to ‘9’ as shown in Figure 2. Clean CNNs exhibit average accuracy of 99.02% where backdoored CNNs have accuracy of 98.85% with 99.92% attack success rate, indicating a highly efective trigger attack. Moreover, out of the 450 models, we use 400 CNNs for training and 50 for testing with = 6, meaning we consider all CNN layers’ weights.
MNIST CNN dataset Source label: ‘0’ Target label: ‘9’ e l p m a S n o s i o P e l p m a S n a e l
C
We have utilized the object detection CNN models of the TrojAI dataset 2 which contains backdoored and clean models across two network architectures (Fast R-CNN and SSD) trained on the Common Objects in Context (COCO) dataset. We use 144 ‘Train’ models from the repository as our training models and 144 ‘Test’ models for the evaluation of our pipeline with = 30, meaning we consider the final 30 layer’s weights of the models. Figure 3 shows that there are two types of trigger attacks on the models: evasion and misclassification. Evasion triggers cause either a single or all boxes of a class to be deleted and misclassification triggers cause either a single box, or all boxes of a specific class, to shift to the target label.
Several performance metrics are reported using diferent ML classifiers. We also compare our findings with SOTA backdoor detection methods in terms of both performance and eficiency. Regarding the number of PCA components, we use = 4 and 10 for image classification and object detection datasets respectively. Moreover, we use the standard equation for binomial proportions to estimate confidence intervals on the empirical accuracies for the robustness metrics of the pipelines, i.e., confidence interval= × √︀( × (1 − ))/, where is the number of models classified as backdoored or clean, and we use = 1.96 and thus have 95% confidence intervals [29].
We show the backdoor model detection results in Table 2. Three diferent ML classifiers (random forest (RF), decision tree (DT), and k-nearest neighbor (kNN)) have been used in the experiments for both image classification and object detection datasets. As performance metrics, cross entropy loss (CE-Loss) and area under the ROC curve (ROC-AUC) scores are reported as CE-Loss is the current standard for classification problems and ROCAUC helps to understand the false positive rate (FPR), being so crucial for backdoor model detection. In both datasets, RF performs better than DT and kNN in terms of CE-Loss and ROC-AUC scores. Our pipeline using RF shows ROC-AUC scores of 0.91 for image classification and 0.89 for object detection datasets.
Image classification
Our method is evaluated in comparison to four SOTA backdoor detection techniques: NC [18], Universal Litmus Patterns (ULP) [19], Activation Clustering (AC) [23], Image Classification: RF Image Classification: DT Image Classification: kNN Object Detection: RF Object Detection: DT Object Detection: kNN and ABS [22]. For a fair comparison, we employ the same batch size for optimization-based approaches including NC, ABS, and ULP.
The results are shown in Table 3 where we report the best results of our pipeline which is using IVA with a RF classifier (IVA-RF). Our method outperforms all the competing methods by a wide margin in terms of both CELoss and ROC-AUC score. IVA-RF obtains a ROC-AUC of 0.91 which is higher than the next-best ULP by a margin of 0.06. AC shows the lowest ROC-AUC as it works better for certain types of trigger attacks. Moreover, IVA-RF has the tightest confidence interval and lower CE-Loss meaning our pipeline is more robust than the competing algorithms.
NC ABS ULP AC IVA-RF (ours)
CE-Loss Object detection
The majority of backdoor attack detection techniques for image classification do not work for object detection. In addition, the object detection model’s output (a large number of objects) difers from the image classification model (predicted class). The only SOTA method we have found to compare our algorithm with is detector cleanse (DC) [24] and the results are shown in Table 4. Similar to image classification, IVA-RF outperforms DC with higher ROC-AUC and lower CE-Loss.
It’s critical that backdoor detection techniques are efective because they may end up being a standard compoDC IVA-RF (ours) 0.48 0.41
ROC-AUC nent of ML operations. Table 5 shows the time in seconds required to make decisions for backdoor detection. Our method tends to be faster than NC, ABS, ULP (image classification), and DC (object detection) by an order of magnitude due to the fact that our approach is model agnostic and only extracts features from model weights for detection. Although AC’s running duration is close to ours, it is noticeably less accurate, as seen in Table 3. Because of this, our approach can achieve an eficiencyaccuracy balance that none of the other algorithms can.
computation time of methods (s) Dataset Image Object
NC As we have applied PCA for dimensionality reduction before IVA, an ablation study was conducted to see the impact of PCA. Figure 4 shows the ROC-AUC scores when we do not use PCA and with diferent numbers of PCA components. The classifier performance degrades significantly when we do not use PCA as IVA has to handle the noisy data to extract features. However, we preserved 90% variance of the data by using a number of components = 4 and 10 for image and object datasets respectively. When we use lower or higher numbers of components the score drops as we loose information for lower numbers and we add noisy components for higher numbers.
Ours is the first work of which we are aware that uses matrix factorization on the weights to detect backdoors in deep networks. Moreover, this is the first pipeline which can detect backdoor models in case of both image classification and object detection networks which has a number of advantages, including the fact that it needs no re-training or optimization and is much faster than other state-of-the-art backdoor detectors. Future work will include applications to sequence models such as those used in natural language processing, which should be straightforward from an engineering perspective given that our method uses only the pre-trained weights of the networks.