The paper describes the approach adopted for Author Masking Task at PAN 2017. For the purpose of masking the original author, we use the combination of methods based either on deep learning approach or traditional methods of obfuscation. We obtain sample of obfuscated sentences from original one and choose best of them using language model. We try to change both the content and length of original sentence preserving its meaning.
PAN 2017 [16] is a series of tasks on digital text forensics, which is held as a part of the CLEF conference [8]. The main idea of one of the proposed tasks named author masking task [5] is to paraphrase a given document so that its writing style does not match that of its original author, anymore. Training corpus consists of set of documents from the same author. One of these documents should be obfuscated. Quality of suggested software is verified by following metrics:
safety -a forensic analysis does not reveal the original author of its obfuscated texts, soundness -obfuscated texts are textually entailed with their originals, sensibleness -obfuscated texts are inconspicuous.
The related tasks that were proposed at PAN 2017 are author identification [21] and author profiling [18]. The evaluation of all the tasks is conducted using TIRA [14], a service for data analysis tasks evaluation.
On PAN'16 conference [15] in "Author Obfuscation" task participants proposed three different ways for author masking. The first approach consists of translation text from the source language (English) into an intermediate language before it gets eventually translated back to English [9]. The main advantage of this method is a strong modification of the original text, the main disadvantages -a vast amount of untranslated words and weak semantic coherence of the resulted text. The second approach used in [11] is to synonymize the most frequent words of original text. This approach keeps the original meaning of the text in most of cases, but gives a small amount of modifications of the original text. The third approach combines strong context modification with preserving the original sense [12]. This algorithm is based on different types of text obfuscation and gave the best result by the metrics used in the contest.
Statistical and context features are used in modern detecting authorship approaches, for example in GLAD [7]. In our solution we try to obfuscate both of them. We use both traditional methods for author masking, such as synonimizing and splitting/joining sentences and obtain some modern methods based on recurrent neural networks. Using deep neural networks we took into account the papers [13,19,22,4,17,20] on the use of recurrent neural networks in paraphrase generation and detection. We use LSTM-based model [20] in Encoder-Decoder fashion.
Our approach is based on per-sentence obfuscation. At the first step we split text into sentences. After that we try to paraphrase sentences using methods described below. We paraphrase each sentence until Jaccard similarity score between set of tokens from an original s original sentence and an obfuscated s obfuscated sentence is less than threshold θ or unless we tested all the obfuscation methods for the original sentence:
All of the described obfuscation methods works with one or two sentences. Priority of using obfuscation methods is based on statistics of its previous successful appliance -we try to make the distribution of methods usage close to uniform since different methods of obfuscation can mask different style features of the original text. Therefore infrequently used approaches apply first for new sentences.
The methods we use to obfuscate sentences can be divided into 2 groups:
1. Methods that change the content of the sentences, trying to save the sense. 2. Methods that change the structure and length of the sentences.
We use different types of changing sentences length. As a part of preprocessing, we replace short forms for long ones: words ended with 'll, 've, 'm, etc. replaces with their long forms -will, have, am, etc.
Our main approach of changing text length is to split and join sentences. As a trigger of splitting we use rather simple heuristic: we try to split sentences by coordinating (and, but) and subordinating (because, since, so, therefore) conjunctions. As a method of joining sentences we use the following rule: we can join sentences using the same conjunctions if both sentence have rather small length, we use range between 30 and 150 chars for this constraint.
The third method we used is an adjustment or removal introductory phrases from sentences. We use only general meaning phrases such as it is important to note that, anyway, in fact, also, etc.
We use two methods of changing content of the sentence. Synonym replacing. First method is based on traditional synonimizing idea, where some words of the input sentence are replaced by their synonyms. However, instead of using existing dictionaries or ontologies we use word embedding as a source of synonimizing. We generate subsample set of k different combinations from nearest words lists and take best of generated sentence by the language model score.
Let (w 1 , . . . , w n ) be a sequence of word embeddings from the sentence. For each word w i except stopwords we take k nearest words by cosine similarity: v i = (w i1 , . . . , w ik ). We generate s sentences s 1 , . . . , s s sampling from v i words instead of original word w i . After that we find the sampled sentence with the maximal language model score:
where LM is a logarithm of language model probability [10].
For our experiments we used k = 5 and s = 100. The language model was trained on 3-grams from Shakespeare's Sonnets corpus from Project Gutenberg [1]. In our opinion the original author style will be masked because this procedure gives best scores for sentences, nearest to Shakespeare style. We did not use language model of higher order because of small size of the corpus.
Encoder-Decoder approach. Another method is based on LSTM recurrent neural network. The basic LSTM model can be described with the following equations:
We train our model in Encoder-Decoder way [20,19] with modification of LSTM described in [20]: we decompose our model into Encoder model and Decoder model.
Encoder recursively combines the sequence of word embeddings w 1 , . . . , w n into a fixed-length vector h n−1 :
where g e is a stack of LSTM functions, h e t−1 is a hidden state, c e t−1 is a cell state vector. Decoder tries to reproduce the input sequence w 1 , . . . , w n by hidden vector sequence h e n−1 , . . . , h e 1 and vector c:
where g d is a stack of LSTM functions, h d t−1 is a hidden state, c d t−1 is a cell state vector, c is a cell state vector from the last step of the encoder.
Encoder and Decoder models are jointly trained in order to minimize reconstruction error:
For the end of sentence determination we added "End of sentence" token to our embedding model so that in general the length of our original sentence s original and the obfuscated sentence s obfuscated may differ. Further we use reproduced sequence ŵ1 , . . . , ŵno the same way as we use in our synonym replacing approach (2), where n o is the number of tokens before "End of sentence" token.
We considered two automatic metrics for evaluating final obfuscation. For the sensibleness evaluation we used average language model score from KenLM language model [6]. The language model from our obfuscation method differs from the model we use for evaluation: whenever we used model trained on Shakespeare corpus for obfuscation, the model for the evaluation was trained on Wikipedia corpus. Therefore despite the fact we tried to mask the original author style using Shakespeare style, during the evaluation step we considered how the obfuscated text fitted into common English language.
For the safety evaluation we used the similar method as described in [12]: we measured how much the prediction from GLAD [7] author verification system changed. We used random forest classifier in GLAD.
We did not consider any automatic metric for the soundness and used peer review.
On preprocessing step we used the NLTK toolbox [2] to extract separate sentences from the original text. We used FastText library [3] for word embedding. Our model was trained on the latest dump of Wikipedia corpus, with word vector dimension equal to 300. For the recurrent neural network training we used Seq2Seq library1 also trained on Wikipedia corpus. Based on peer review we set θ = 0.75 in (1). We used 2-layer LSTM as it showed better results than 1-layer model. Our average language model score for sensibleness was −99.4 ± 61.9 whenever the score for the original sentences was −79.4 ± 55.8. As we can see, the scores are rather close since the means of distributions lie in the range of the standard deviations of each other.
The average change in GLAD probabilities is −0.11 ± 0.22. The number of correctly verified texts was lowered after obfuscation from 189 to 153. We observe that our obfuscation method works successfully and lowers the verification probabilities for the obfuscated texts.
An example of our obfuscation method is listed in table 3. As we can see, the obfuscated sentences obtained by Encoder-Decoder can lead to some grammatical errors. However, the significant part of the sentences we viewed was grammatically correct. The other interesting feature of the sentences with synonym replacement and Encoder-Decoder method is an appearance of word "scabbard" in obfuscated sentences. We consider it is a result of using Shakespeare corpus in the final sentence scoring (2). Table 1. An example of our method usage
Obfuscated sentences Original
The quick brown fox jumps over the lazy dog. The five boxing wizards jump quickly with knifes. Synonym replacing The rapid reddish fox grabs over the sloppy terrier. The five boxer spellcasters jumper quickly with scabbard. Encoder-Decoder The better brown fox tosses overlapped in scary pig even. The Seven boxing superhero trampolining eventually with scabbard. Introductory words All in all, the quick brown fox jumps over the lazy dog. In a word, the five boxing wizards jump quickly with knifes. Join sentences The quick brown fox jumps over the lazy dog, because the five boxing wizards jump quickly with knifes.
The paper describes our system for the PAN 2017 Author Masking Task. Our main approach based on using recurrent neural networks for text obfuscation. Also we use more traditional methods of obfuscation, such as synonimizing and changing statistical text features. We used language model for selection best masking result. Further development includes improving obfuscation quality of seq2seq model by tuning its parameters and taking into consideration many other heuristics.