We consider the problem of adversarial text generation in the context of cyber-security tasks such as email spam filtering and text classification for sentiment analysis on social media sites. In adversarial text generation, an adversary attempts to perturb valid text data to generate adversarial text such that the adversarial text ends up getting mis-classified by a machine classifier. Many existing techniques for perturbing text data use gradient-based or white-box methods, where the adversary observes the gradient of the loss function from the classifier for a given input sample, and uses this information to strategically select portions of the text to perturb. On the other hand, black-box methods where the adversary does not have access to the gradient of the loss function from the classifier and has to probe the classifier with different input samples to generate successful adversarial samples, have been used less often for generating adversarial text. In this paper, we integrate black-box methods where the adversary has a limited budget of the number of probes to the classifier, with white-box, gradient-based methods, and evaluate the effectiveness of the adversarially generated text in misleading a deep network classifier model.
Machine learning-based systems are currently used widely for different cyber-security tasks including email spam filtering, network intrusion detection, sentiment analysis of posts on social media sites like Twitter, and validating authenticity of news posts on social media sites. Most of these systems use supervised learning techniques where a learning algorithm called a classifier is trained to categorize data into multiple categories. For instance, an automated classifier for an email spam filter classifies incoming email into spam and non-spam categories. A critical vulnerability of classifier-based learning algorithm is that they are susceptible to adversarial attacks where a malicious adversary could send corrupted or poisoned data to the classifier that results in incorrect classification, or, for more calculated attacks, use corrupted data to alter the classification decisions of the classifier. Both these attacks could compromise the operation of the classifier, making it behave in an unintended, possibly insecure and unsafe manner. As an example, a compromised email classifier could end up categorizing valid email messages as spam (false positives) while delivering spam email messages as non-spam (false negatives). An important defense against adversarial attacks is to build a model of the adversary including the data that the adversary generates, and use this model as a virtual adversary to improve the learner’s robustness to adversarial attacks. Towards this objective, in this paper, we investigate adversarial data generation techniques that could be used to model corrupted data generated by a virtual adversary. Our work focuses on adversarial text data generation as many of the aforementioned cyber-security tasks operate mainly on text data.
Adversarial data generation techniques can be broadly
classified into two categories called white-box and
blackbox. In white-box techniques, the adversary has access to
information of the classifier, such as the parameters of
the model of the classifier, e.g., weights in a deep neural
network-based classifier, or, to internal calculations of the
classifier, such as the gradients of the loss function
calculated by the classifier on a sample. In contrast, in
blackbox techniques, the adversary does not have any
information about the classifier and treats it like a black-box, by
passing data samples as queries to the classifier and
observing the classifier’s output category or label for each sample.
Additionally, in most practical black-box settings, the
adversary can send only a finite number of queries to the
classifier due to the adversary’s budget limitations in generating
samples, or, due to restrictions in the number of queries
accepted by the classifier. While white-box techniques such as
gradient-based methods
Adversarial data generation techniques have received
considerable attention from researchers in the recent past. The
main concept in adversarial data generation is to add slight
noise to valid data using a perturbation technique so that the
corrupted data still appears legitimate to a human but gets
mis-classified by a machine classifier. Most research on
adversarial data generation has focused on image data,
including gradient-based methods for perturbing valid data (Biggio
et al. 2013),
Adversarial Text Generation. For adverarial image
generation, image characteristics like RGB pixel values, HSV
values, brightness, contrast, etc., are real-valued numbers
that can be manipulated using mathematical operations to
generate perturbed images that can fool a machine classifer
while remaining imperceptible to humans. In contrast, for
perturbing text, adding a real value to a numerical
representation of a word, character, or token in an embedding space,
might result in a new word that might either be nonsense,
or, might not fit in with the context of the unperturbed,
original text - in both cases, the adversarial text can be easily
flagged by both a machine classifier and a human. To
address this shortcoming, instead of directly using image
perturbation techniques to generate adversarial text, researchers
have proposed using auto-encoders
One of the main functions of an adversary in an adversarial
learning setting is to generate, and possibly mislabel,
adversarial text to fool the classifier. Adversarial text is usually
generated by starting from valid text and changing certain
words or characters in it. This process is also referrred to
to as perturbing text. We define adversarial text
perturbation using the following formalization: Let V denote a
vocabulary of English words and X denote a corpus of
English text. {Xi} ⊂ X denotes a dataset consisting of
English text samples, where Xi denotes the i-th sample.
Further, Xi = (Xi(1); Xi(2); :::; Xi(W )), where Xi(j) denotes the
j-th word in Xi and W is the maximum number of words
of a text sample. Each word X(j) is represented by a feature
i
vector X(j) = f1:F generated using some word embedding
i
like word2Vec
min ∆(Xi; X˜i;n ) s.t. yC (X˜i;n ) ̸= y(Xi); and, yC (Xi) = y(Xi) (1)
The objective function in Equation 1 finds the number of words to perturb that gives the minimum divergence between the original and perturbed text, while the last two constraints respectively ensure that the classifier mis-classifies the perturbed text, X˜i;n , giving it a different label than the ground truth y(Xi), but the classifier correctly classifies the original text Xi. In the next section, we describe different perturbation strategies we have used to perturb valid text and generate adversarial text.
Perturbation Methods for Text Data An adversarial perturbation method adds a certain amount of noise to unperturbed data to generate perturbed data. There are two main questions in a perturbation method: how much noise to add, and where in the data to add the noise. For the first question, the two types of perturbation methods, whitebox and black-box, both add a random, albeit small amount of noise. But these two types of methods differ in their approach to answer the second question. Below, we describe the white-box and black-box perturbation methods, and propose a gray-box method that combines the advantages of the two.
White-box Gradient-based Perturbation. White-box methods first query the classifier with the unperturbed text and observe the gradient of the loss function of the classifier’s output from the query for the different words in the queried text. Because the loss function expresses the difference of the classifier’s determined label from the ground truth label for the queried text, the word in the text that corresponds to the most negative gradient or change of the loss function is most influential in changing the label for the text determined by the classifier. This word is selected as the word to perturb. Mathematically, the selected word, x , having the most negative gradient is given by: where L is the loss function from the classifier, fLk is the gradient of the loss function for the k-th feature of a word, wj;k is the weight in the embedding layer neural network connecting the j-th word to its k-th feature in embedding space. x is then perturbed by replacing it with x˜(j), the word from the vocabulary that has the smallest positive gradient of loss function, and, consequently, has the least influence in changing the label determined by the classifier. Mathematically, x˜(j) is given by:
L L x˜(j) = arg min ∑ wj;k fk ; s:t: > 0 (3) j k fk
The general schematic of the white-box gradient-based perturbation is shown in Figure 1.
Black-box Perturbation In contrast to white-box
methods, black-box methods select the perturbation position
randomly within unperturbed text. Consequently, black-box
methods could yield adversarial examples that are less
effective in misguiding the classifier than white-box generated
examples. Below, we describe two black-box methods used
to perturb text.
• Black-box, Budget-Limited Perturbation (Anchor
Points). The anchor points (AP) method, proposed
in
Gray-box Perturbation Despite their limitation of generating less effective adversarial text, black-box methods can 20 words
1 word 30 words 0.2 0.4 0.8
1 0.6 BLEU Score 3000 2500 lse
p 2000 a m S f o 1500 re b m 1000 u
N 500 0 10 words generate larger and more diverse set of adversarial examples with fewer queries or probes to the classifier than white-box methods. With this insight, we propose to combine whitebox and black-box methods into a gray-box method to investigate perturbation methods that can generate effective yet diverse adversarial examples. In the gray-box method, instead of drawing unperturbed samples for the black-box methods randomly from the original dataset, we first use the white-box method to generate a small seed set of perturbed examples from samples that are randomly-drawn from the original dataset. This seed set of perturbed examples is then used to create a larger set of perturbed examples using the AP and RE black-box methods.
Word Embedding and Classifier Network. We have
evaluated our proposed techniques on the DBPedia dataset.
The dataset contains 700; 000 short articles extracted from
Wikipedia and categorized into 14 different categories. For
generating word embeddings from English words we have
used a widely used vector representation format called
Word2Vec
For evaluating the effectiveness of the perturbation
technique we have used the Bilingual Evaluation Understudy
(BLEU) score
In our first experiment, we analyzed the effect of the amount of perturbation of different text samples measured in terms of BLEU score (x-axis) on the number of samples that are assigned the same label before and after perturbation (y-axis). Perturbed text was generated using the white-box gradient-based method where the features of the words with the most negative gradients, calculated using Equation 2, were perturbed with a small random amount. The number of words to perturb was varied over {1; 10; 20; 30}: For each of these perturbation amounts, a batch of 1000 text samples were perturbed, and results were averaged over 4 batches. BLEU scores of perturbed text was binned into intervals of 0:1 by rounding each BLEU score to the first place of decimal. Results shown in Figure 2, illustrate that as the BLEU score decreases (perturbed text becomes more different from the original text), the fraction of samples that retain the same label before and after perturbation also decreases. This result appears intuitive because the more different a piece of text is from its original, unperturbed version, the less likely it is to retain the same label. However, for BLUE score of 0:4 and lower, we observe that the fraction of samples retaining the same label as the original slightly increases when 10 or more words are perturbed. This is possibly due to the fact that when the perturbed text is very different from the original text, most of the perturbed words are out of context from the original text and the text appears as nonsense to a human reader. The machine classifier however is confounded into labeling the nonsensical, perturbed text with the same label as the original text, possibly due to one or more of the unperturbed words. Further investigation into this issue would lead to a better understanding of the degree of perturbation that converts text into rubbish for a human and possibly unintelligible for the machine classifier as well.
For our next set of experiments we analyzed the effect of the main algorithm parameters in the AP and RE algorithms, the maximum and minimum perturbation radii, Rmin and Rmax on the amount of perturbation measured in terms of BLEU score, and the fraction of samples that retain the same label as the original. Results are shown in Figures 3 through 5. We observe that for both AP (Figure 3(a)) and RE (Figure 4(a)) algorithms, as the perturbation radius increases, the BLEU score reduces, which corroborates the fact that the degree of perturbation increases the divergence between the original and perturbed text. Correspondingly, the fraction of perturbed text that retains the same label as the original text generally decreases with the increase in perturbation radius for both AP and RE, in Figure 3(b) and Figure 4(b) respectively. Figure 5 shows the BLEU score and fraction of samples that retain same label as the original versus the number of words perturbed using the white-box gradient-based approach, where the words with highest negative gradient were replaced with words with smallest positive gradient calculated using Equations 2 and 3. As before, we observe that more perturbation results in lower BLEU scores implying more divergent text after pertubation. More perturbation also reduces the fraction of perturbed samples that retain the same label as the original text.
In this paper we investigated gray-box techniques for generating adversarial text as a combination of white-box gradient-based techniques and black-box techniques. We validated the correct behavior of the gray-box techniques in generating perturbed text while showing that more perturbation results in greater divergence as well as greater degree of label-changes in the perturbed text with respect to the original text. In the future, we plan to compare the proposed gray-box adversarial text generation methods with GANbased and RNN-based synthetic text generation. While single words are usually treated as the basic lexical unit, it would be interesting to analyze how character-level perturbations and word sequence or n-gram-level perturbations affect adversarial text generation. We are also interested in getting a better understanding of how to determine a critical or minimal amount of perturbation that would be successful in generating adversarial text. We envisage that this work and its extensions will generate interesting results that would enable a better understanding and novel means for adversarial text generation, and methods to make machine classifiers robust against adversarial text-based attacks. 1 0.95 0.9 0.85 re 0.8 o cS0.75 Rmin = 1.0 LEU 0.7 B0.65 0.6 0.55 0.5 0
2
Rmin= .4
Rmin = 1.0 0
2 1 0.95 0.9 0.85 re 0.8 o cS0.75 LEU 0.7 B0.65 0.6 0.55 0.5 1 0.95 0.9 0.85 re 0.8 o cS0.75 LEU 0.7 B0.65 0.6 0.55 0.5 0
Rmin = .7
Rmin = .4
Rmin = .1 4
6 Radius Max (a) Rmin = .1 4
6 Radius Max (a)
Rmin = .1 6
Radius Max (b)
Rmin = .1 Rmin = .7
Rmin = 1.0
Rmin = .4 4 6 Radius Max (b) (b) 8 10 12 0 2 4 8 10 12 conference on machine learning and knowledge discovery in databases, 387–402. Springer.
Biggio, B.; Corona, I.; Maiorca, D.; Nelson, B.; Sˇrndic´, N.; Laskov, P.; Giacinto, G.; and Roli, F. 2013. Evasion attacks against machine learning at test time. In Joint European Chen, X.; Duan, Y.; Houthooft, R.; Schulman, J.; Sutskever, I.; and Abbeel, P. 2016. Infogan: Interpretable representation learning by information maximizing generative adver5
10 15 Number of Words to Perturb (a) 20 25 0 5
10 15 Number of Words to Perturb 20 25 sarial nets. In Advances in neural information processing systems, 2172–2180.
Ebrahimi, J.; Rao, A.; Lowd, D.; and Dou, D. 2018. Hotflip: White-box adversarial examples for text classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 2: Short Papers, 31–36. 2014.
CoRR