Author masking is the task of paraphrasing a document so that its writing style no longer matches that of its original author. This task was introduced as part of the 2016 PAN Lab on Digital Text Forensics, for which a total of three research teams submitted their results. This work describes our methodology to evaluate the submitted obfuscation systems based on their safety, soundness and sensibleness. For the first two dimensions, we introduce automatic evaluation measures and for sensibleness we report our manual evaluation results.
Author masking is the task of paraphrasing a document so that its writing style no longer
matches that of its original author. Due to the advances in fields such as authorship
attribution and author verification, it is not clear whether authors (particularly in the
age of the Internet and social media) can assure their anonymity anymore [
The task of author masking was introduced as part of the 2016 PAN Lab on Digital
Text Forensics [
Together with the task of author masking, obfuscation evaluation has been
introduced as another task to evaluate the performance of the author masking submissions.
Three dimensions have been defined by the task organizers for the performance
evaluation of the obfuscation systems: safety to ensure that a forensic analysis does not reveal
the original author of an obfuscated text; soundness to evaluate if the obfuscated texts
are textually entailed with their originals; and sensibleness to ensure that the obfuscated
texts are inconspicuous [
In this work, we describe our methodology to evaluate the performance of the submitted systems based on the aforementioned dimensions. In section 2 we define the problem of author masking more concretely and describe the provided training data. The evaluation results of the dimensions safety, soundness and sensibleness are reported in sections 3, 4 and 5, respectively. Finally, we conclude our work in section 6. 2
Given a document, an author masking software has to paraphrase it so that its writing style no longer matches that of its original author. Although the organizers of the author masking task do not directly define this task as a supervised machine learning problem, a training set is provided so that the participant can evaluate their designed algorithms based on this dataset. The same dataset is also used as the test dataset for the final evaluation.
The provided dataset is a collection of 205 problems selected from author
verification tasks from PAN2013 [
The participants were asked to develop a software that outputs a detailed list, how each piece of the original text has been paraphrased. For a detailed description of the desired system output, the reader is referred to the official task page1. 3
An obfuscation software is called safe, if a forensic analysis does not reveal the
original author of the obfuscated texts. We evaluate the safety of the obfuscation software
using an automatic author verifier called GLAD [
The GLAD algorithm was one of the top ranked systems at PAN2015 and treats
the author verification problem as an intrinsic binary classification machine learning
task. GLAD uses SVM [
To train the GLAD algorithm we used the English problems from the training datasets from PAN2013 to PAN2015. The statistics of the training dataset used are shown in Table 1.
Notice that the training dataset from PAN2014 consisted of novels and essays and we took both categories to train our model.
Using the trained model, we measure the performance of the GLAD algorithm once
using the original documents from the training set of the author masking problem (labels
are all considered to be true), and once on the submissions of each team (labels are
all considered to be true). As the evaluation measure we use c@1 [
c@1 = 1 nc + nunc (1)
n n
In the definition of c@1 measure, n is the number of problems, nc is the number of correct answers and nu represents the number of unanswered problems.
Table 2 summarizes the evaluation results of the safety dimension. On the original dataset (the training dataset of author masking), the GLAD algorithm has a c@1 score of 0.656, meaning that the algorithm correctly classifies roughly 65% of the problems as positive. Notice that the gold standard labels are all set to be true and that teams having lower c@1 scores are more successful at the masking task than the other teams. It is also observable that none of the participants has a c@1 score below 0.5. This means that the portion of successful obfuscations for all submissions is below 50%. At the same time it can be seen that all submissions have a c@1 score below the baseline 0.656, meaning that all teams were partly successful at the author masking task.
Although in previous PAN competitions, AUC (Area Under the Curve) [
Another interesting analysis is to investigate the relation between true positives and false negatives. The idea behind this analysis is to figure out the portion of documents classified as positive before obfuscation, and the ones classified as negative after obfuscation. For this we select true positives from the original dataset and count the ones that have been classified as negative by the GLAD algorithm. Table 3 summarizes the results.
Notice that higher values in Table 3 are preferred. Team C has the highest score among the teams and has managed to obfuscate roughly 30% of the true positive problems to false negative ones. These results are consistent with the results shown in Table 2. 4
We assume that the goal of author masking is to reword a text segment into a paraphrased one while retaining as much semantic similarity as possible. Therefore, we propose to quantify soundness by measuring the semantic textual similarity (STS) between the original text segment and its corresponding obfuscation.
The prediction of semantic textual similarity has been a recurring task in SemEval
challenges since 2012 [
In this paper, we use the unsupervised semantic similarity approach called Overlap
[
The idea of the Overlap method is simple since it measures the overlap between
the tokens in the original segment s1 and the tokens in the paraphrase s2 by aligning
tokens to the best match in the other text segment. The authors first define a similarity
function for two tokens which uses synsets from WordNet [
sim(t1; t2) := >81 > >>>1 > < if t1.lemma == t2.lemma if t1 and t2 have the same most common synset 0:5 if t1 and t2 share any other synset >>>cos(t1; t2) if t1 and t2 have word2vec embeddings > >>:0.15 otherwise (2)
Afterwards, the similarity score between two text segments in [0; 5] is defined as follows: 0 P max sim(t1; t2) t12s1 t22s2 2 js1j
P max sim(t2; t1) 1 + t22s2 t12s1 2 js2j A (3)
Since we assume the obfuscations to be semantically as close as possible to the originals, the STS score between both segments should be 5. We predict the semantic similarity for all pairs for each team. Afterwards, we average over the predicted scores for each team. Table 4 summarizes the results for the soundness dimension.
For the soundness dimension, the best semantic paraphrases were created by team A
with an average STS score of 4.87. This is not surprising since team A only substituted
a few words and often kept the original segment as a paraphrase. Therefore, the
paraphrases are semantically very close or even identical to the original. Team C achieved a
mean STS score of 4.48 and team B had the lowest score with 4.04. Since the Overlap
approach from [
The dimension sensibleness describes the language quality of the obfuscations and whether it allows us to understand them. An author masking software might mask the author of a text at the cost of its comprehension. Therefore, it is also crucial to evaluate the quality of the produced obfuscations.
We observed that teams A and C used dictionaries to perform simple substitutions and team B usually changed the order of phrases. It is surprising to see that the paraphrases by team B sometimes contain random German words, as in the following example: “it is difficult to across, Once the Mitbürgers unschön is faint, odor street, on the village so massed mold Verfalls and centuries.”
Although there are approaches to automatically predict the grammatical quality of text, we chose to manually evaluate the sensibleness because portions of the text have a low language quality but still allow for a limited understanding of the content. For example, this can be compared to a non-native speaker who asks in an online forum a question that is poorly worded but still comprehensible.
After a manual inspection of a subset of the paraphrases from all three teams, we decided to annotate each pair with a score s 2 f0; 1; 2g to measure the language quality. We then drew a small sample and discussed annotation guidelines. Our three labels and their definitions are described in Table 5.
In our evaluation, sensibleness is only evaluated by looking at the obfuscated text. This is due to the fact that only the paraphrased text after author masking is used in a real world scenario. Therefore, it is reasonable to only evaluate the output of the system. We ignore spacing and line breaks during the annotation process. Furthermore, we also ignore the substitutions of the words “oof ” and “tto” from team C because they do not impact the understanding of the text.
We randomly drew a subset of 20 problems. For each team, we then drew three obfuscations per problem. All of these obfuscations were manually annotated by three annotators. In order to report a single value per team, we averaged all the scores from the annotators. Table 6 summarizes the results for the sensibleness dimension.
Team A achieved the best results in the sensibleness dimension with an average score close to 2. The paraphrases from team B allow for the lowest understanding of all three teams with an average score of 0.57 which is between partially comprehensible and incomprehensible.
We should note that there are at least two problems for the evaluation of the sensibleness dimension: (i) it is difficult to formalize language quality and understanding and (ii) the sensibleness dimension is subjective. Although we observed a high agreement on the category incomprehensible, we had a lower agreement on whether a paraphrase is fully or partially comprehensible. This is plausible since one annotator might perfectly understand a text segment while another annotator may have some troubles with it. 6
In this work, we discussed our methodology to evaluate the performance of the obfuscation systems submitted to the PAN2016 Author Masking shared task. More concretely, submissions were evaluated based on their safety (Section 3), soundness (Section 4), and sensibleness (Section 5). The scripts for our evaluation are available on GitHub2.
An automatic author verifier was used to measure the safety of the submissions. The ranking of the teams in terms of safety is as follows: team C, B, and A
We proposed to quantify soundness by automatically measuring the semantic text similarity between the original text fragments and their obfuscations. The best score was achieved by team A, followed by teams C and B.
Unlike the first two dimensions, the sensibleness of the submissions was evaluated manually. As sensibleness is subjective and difficult to formally define, we consider its measurement a nontrivial task. Regarding sensibleness, teams A, C and B were ranked first, second, and third, respectively.
This work was partially funded by the PhD program Online Participation, supported by the North Rhine-Westphalian funding scheme Fortschrittskollegs and by the German Federal Ministry of Economics and Technology under the ZIM program (Grant No. KF2846504). We would like to thank Daniel Braun for his help in the evaluation of the sensibleness dimension. 2 https://github.com/pasmod/obfuscation 21. Stamatatos, E., amd Ben Verhoeven, W.D., Juola, P., López-López, A., Potthast, M., Stein, B.: Overview of the Author Identification Task at PAN 2015. In: CLEF 2015 Evaluation Labs and Workshop – Working Notes Papers. CEUR-WS.org (2015) 22. Stamatatos, E., Daelemans, W., Verhoeven, B., Potthast, M., Stein, B., Juola, P., SanchezPerez, M., Barrón-Cedeño, A.: Overview of the Author Identification Task at PAN 2014. In: CLEF 2014 Evaluation Labs and Workshop – Working Notes Papers. CEUR-WS.org (Sep 2014)