The paper describes the approach adopted for Author Masking Task at PAN 2017. For the purpose of masking the original author, we use the combination of methods based either on deep learning approach or traditional methods of obfuscation. We obtain sample of obfuscated sentences from original one and choose best of them using language model. We try to change both the content and length of original sentence preserving its meaning.
The related tasks that were proposed at PAN 2017 are author identification [
On PAN’16 conference [
Statistical and context features are used in modern detecting authorship approaches,
for example in GLAD [
Our approach is based on per-sentence obfuscation. At the first step we split text into sentences. After that we try to paraphrase sentences using methods described below. We paraphrase each sentence until Jaccard similarity score between set of tokens from an original soriginal sentence and an obfuscated sobfuscated sentence is less than threshold or unless we tested all the obfuscation methods for the original sentence: J (soriginal; sobfuscated) = jsoriginal \ sobfuscatedj jsoriginal [ sobfuscatedj : (1)
All of the described obfuscation methods works with one or two sentences. Priority of using obfuscation methods is based on statistics of its previous successful appliance — we try to make the distribution of methods usage close to uniform since different methods of obfuscation can mask different style features of the original text. Therefore infrequently used approaches apply first for new sentences.
The methods we use to obfuscate sentences can be divided into 2 groups: 1. Methods that change the content of the sentences, trying to save the sense. 2. Methods that change the structure and length of the sentences. 2.1
We use different types of changing sentences length. As a part of preprocessing, we replace short forms for long ones: words ended with ’ll, ’ve, ’m, etc. replaces with their long forms — will, have, am, etc.
Our main approach of changing text length is to split and join sentences. As a trigger of splitting we use rather simple heuristic: we try to split sentences by coordinating (and, but) and subordinating (because, since, so, therefore) conjunctions. As a method of joining sentences we use the following rule: we can join sentences using the same conjunctions if both sentence have rather small length, we use range between 30 and 150 chars for this constraint.
The third method we used is an adjustment or removal introductory phrases from sentences. We use only general meaning phrases such as it is important to note that, anyway, in fact, also, etc. 2.2
We use two methods of changing content of the sentence.
Synonym replacing. First method is based on traditional synonimizing idea, where some words of the input sentence are replaced by their synonyms. However, instead of using existing dictionaries or ontologies we use word embedding as a source of synonimizing. We generate subsample set of k different combinations from nearest words lists and take best of generated sentence by the language model score.
Let (w1; : : : ; wn) be a sequence of word embeddings from the sentence. For each word wi except stopwords we take k nearest words by cosine similarity: vi = (wi01; : : : ; wi0k). We generate s sentences s1; : : : ; ss sampling from vi words instead of original word wi. After that we find the sampled sentence with the maximal language model score: sobfuscated = arg max LM(s);
s2fs1;:::;ssg
where LM is a logarithm of language model probability [
For our experiments we used k = 5 and s = 100. The language model was trained
on 3-grams from Shakespeare’s Sonnets corpus from Project Gutenberg [
Encoder-Decoder approach. Another method is based on LSTM recurrent neural network. The basic LSTM model can be described with the following equations: it = ( xixt +
hiht 1 + bi); ft = ( xf xt +
hf ht 1 + bf ); ot = ( xoxt +
hoht 1 + bo); cin = tanh( xcxt +
hcht 1 + bc); ct = ft ct 1 + it cin;
ht = ot tanh(ct); g(ht 1; wt 1; ct 1) = ht:
We train our model in Encoder-Decoder way [
Encoder recursively combines the sequence of word embeddings w1; : : : ; wn into a fixed-length vector hn 1:
ht = ge(hte 1; wt 1; cte 1); where ge is a stack of LSTM functions, hte 1 is a hidden state, cte 1 is a cell state vector.
Decoder tries to reproduce the input sequence w1; : : : ; wn by hidden vector sequence hen 1; : : : ; he1 and vector c: w^ t = fd(htd 1; w^ t 1; ctd 1; c); (2) (3) (4) where gd is a stack of LSTM functions, htd 1 is a hidden state, ctd 1 is a cell state vector, c is a cell state vector from the last step of the encoder.
Encoder and Decoder models are jointly trained in order to minimize reconstruction error: n X i=1 jjwi
w^ ijj2:
For the end of sentence determination we added “End of sentence” token to our embedding model so that in general the length of our original sentence soriginal and the obfuscated sentence sobfuscated may differ. Further we use reproduced sequence w^ 1; : : : ; w^ no the same way as we use in our synonym replacing approach (2), where no is the number of tokens before “End of sentence” token. 2.3
We considered two automatic metrics for evaluating final obfuscation. For the
sensibleness evaluation we used average language model score from KenLM language
model [
For the safety evaluation we used the similar method as described in [
We did not consider any automatic metric for the soundness and used peer review. 3
On preprocessing step we used the NLTK toolbox [
Our average language model score for sensibleness was 99:4 61:9 whenever the score for the original sentences was 79:4 55:8. As we can see, the scores are rather close since the means of distributions lie in the range of the standard deviations of each other.
The average change in GLAD probabilities is 0:11 0:22. The number of correctly verified texts was lowered after obfuscation from 189 to 153. We observe that our obfuscation method works successfully and lowers the verification probabilities for the obfuscated texts. 1 https://github.com/farizrahman4u/seq2seq
An example of our obfuscation method is listed in table 3. As we can see, the obfuscated sentences obtained by Encoder-Decoder can lead to some grammatical errors. However, the significant part of the sentences we viewed was grammatically correct. The other interesting feature of the sentences with synonym replacement and EncoderDecoder method is an appearance of word “scabbard” in obfuscated sentences. We consider it is a result of using Shakespeare corpus in the final sentence scoring (2). The paper describes our system for the PAN 2017 Author Masking Task. Our main approach based on using recurrent neural networks for text obfuscation. Also we use more traditional methods of obfuscation, such as synonimizing and changing statistical text features. We used language model for selection best masking result.
Further development includes improving obfuscation quality of seq2seq model by tuning its parameters and taking into consideration many other heuristics.