In this paper, we evaluate seven author obfuscation approaches which are supposed to automatically mask an author's writing style in a given text to render automatic author identification impossible. The approaches are evaluated with regard to their safety, soundness, and sensibleness in terms of beating 44 author identification approaches, retaining the original meaning of the obfuscated text, and producing inconspicuous, human-readable obfuscations, respectively. Regarding the measurement of safety in particular, we introduce a set of new performance measures which are designed to render the performance of obfuscation approaches comparable as the numbers of author identification approaches and evaluation datasets increases, incorporating their respective performance and quality. Based on the new measures, we establish a world ranking of obfuscators.
Author obfuscation is the adversary task to author identification. The goal of obfuscation is to render the identification of authors based on their writing style impossible or at least intractable. Consequently, an effective identification approach has to be robust against obfuscation, or else it cannot be trusted. The fundamental question underlying both tasks is whether writing style can be purposefully manipulated. We hypothesize that this is indeed the case, and that style manipulations sufficient to counter identification can be accomplished in a way indistinguishable from genuine writing. Our goal is to foster the development of new technology in this respect, and its evaluation.
We consider three performance dimensions according to which an author
obfuscation approach must excel to be considered fit for practical use. Obviously, the
obfuscation performance should depend on the capability of fooling forensic experts—be it a
piece of software or a human. However, fulfilling this requirement in isolation will
disregard writers and their target audience, whose primary goal is to communicate, albeit
safe from deanonymization: the quality of an obfuscated text along with the fact that its
semantics is preserved are equally important to fool authorship identification. We hence
call an obfuscation software
1. safe, if its obfuscated texts cannot be attributed to their original authors anymore,
2. sound, if its obfuscated texts are paraphrases of their originals, and
3. sensible, if its obfuscated texts are well-formed and inconspicuous.
These dimensions are orthogonal; an obfuscation software may meet each of them to
a certain degree of perfection. Related work on operationalizing different measures for
these dimensions has been included in our recent overview [
We directly employ the new performance measures to evaluate the safety of seven
author obfuscation approaches against 44 author identification approaches. This
includes two obfuscation approaches submitted to our this year’s shared task on author
obfuscation at PAN 2018, as well as five that have been submitted to the two
corresponding shared tasks in the past two years [
In what follows, Section 2 introduces the new safety performance measures for author obfuscation, Section 3 reviews the two obfuscation approaches submitted this year, and Section 4 evaluates their performance in comparison to the five previously submitted ones. More detailed analyses of the new performance measure is found in the appendix. 2
We propose a formal model to implement a kind of “obfuscator world ranking” in order to ease the comparison of new approaches to the state of the art in this growing field. The central building block regarding the safety dimension is a set of effective authorship verification algorithms, also called authorship verifiers for short. Authorship verification is about deciding whether or not a document has been written by a certain author, given one or multiple texts that are known to be written by this author (a one-class classification problem). Then, given a set of authorship verifiers and a corpus of verification problems, a to-be-evaluated obfuscator is run on the positive problems (those problems where the correct answer is “same author”), and it is checked whether for the obfuscated texts the verifier decisions’ are “different authors”.
Thanks to the organizers and participants of the PAN 13, PAN 14, and PAN 15 shared tasks in authorship verification, 44 working authorship verifiers are at our disposal for empirical analysis. For each combination of a positive verification problem and a verifier, we can check how a verifier decides before and after applying the tobe-evaluated obfuscator, obtaining a performance matrix as given in Table 1. An entry of the form T ! F indicates that the obfuscator o successfully fooled the authorship verifier associated to this column on the verification problem in the respective line.
Each successful entry in the performance matrix should increase the overall safety score of the respective obfuscator, while entries of the form F ! T should decrease the score. The exact influence on the final score depends on the set of verifiers used and the “difficulty” of the problem instance (e.g., how many verifiers can identify the authorship before obfuscation).
Actually, Table 1 shows a simplified view of the real situation since an authorship verifier typically returns confidence scores instead of a plain binary decision. A confidence score in [0; 0:5) indicates a (gradually) negative answer (= different authors) whereas a confidence score in (0:5; 1] indicates a (gradually) positive answer. In practice, however, a sensible interpretation of the confidence scores requires a verifierspecific approach—or the computation of standard normally-distributed confidence scores. Here we will choose an individual confidence threshold for each verifier, optimizing its accuracy on the original instances from the verification task. The same threshold will then be used for the decisions on the obfuscated instances.
As for safety evaluation, in previous years we assumed that the used verifiers are deterministic in the sense that they always report the same answer for the same problem, regardless of the history of other problems they have seen. Note that the particular design of the testing scenario used at PAN (all test cases are provided at once) allows a verifier to consider all test documents when classifying an individual problem. E.g., a verifier could exploit global assumptions such as information about the ratio of positive versus negative problems in a setup. In such cases the entries of the performance matrices (see Table 1) may change every time the verifier is run on another sequence of problem instances.
It should be noted that the verifiers that participated in the PAN verification tasks are
obfuscation-unaware, i.e., they assume that no obfuscation has been applied and thus
take no measures to revert or reduce potential obfuscation effects. In the terminology
of Potthast et al. [
In the following we present three axioms that should be fulfilled by a measure that quantifies the obfuscation safety of an obfuscation algorithm. 1. Fooling an effective verifier should be scored higher than fooling a less effective one, where effectiveness may be measured as the verifier’s true positive rate. 2. Fooling a verifier on an unambiguous problem should be scored higher than fooling it on an ambiguous one. Here, a problem is unambiguous if it is decided correctly by many verifiers. 3. Fooling two dissimilar verifiers from a set of equally effective verifiers should be scored higher than fooling two similar verifiers from this set. Here, two verifiers are called similar if they often come to the same decision.
Axiom 1 relates to the verifier effectiveness, Axiom 2 relates to the problem unambiguity, and Axiom 3 relates to the verifier-problem coverage.
The first and third criterion together should help to prevent the creators of obfuscation algorithms from “boosting” their score by submitting many variations of a particular verifier which is especially vulnerable to their obfuscation approach (or, similarly, by submitting variants of approaches that would lower the scores of other obfuscators). The second criterion puts emphasis on those problems for which many of the state-ofthe-art verification approaches perform well—in a nutshell: unambiguous implies “easy for attribution, difficult for obfuscation”. 2.1
For a formal treatment of authorship verification and obfuscation, we adopt the
terminology outlined in [
Let D be the set of all authorship problems. An authorship analysis approach v is a computable function v(hdu; DAi) 2 A [ f?g which is an approximation for . v has been trained on a subset of D, the training set, on which is known. v is evaluated uofsianugthaotresshtipsevteDritfiesctationD, ,wwehliecthDist+esdtisjoiDnttefsrto mbeththeetrsauibnsientgosfevt.eIrnifitchaetisopnecpirfiocblceamses hdu; Dai where (hdu; Dai) = a.
An obfuscator o is a mapping o(hdu; Dai) = hdfu; Dai which obfuscates the text du of the author a, possibly using the other available files for that author Da. The aim of the obfuscation is to change the result of verifiers to ?.
With the basic terminology at hand, we turn to the construction of the new measure: Let V be the set of verifiers under consideration. For v 2 V and d 2 Dtest, let c(v; d) = 1 indicate that v outputs the correct answer to the problem d (whether it is “same author” or “different authors”) and let c(v; d) = 0 indicate a wrong answer. The accuracy and effectiveness of a verifier v 2 V on the problem set Dtest are defined as follows: accuracy(v; Dtest) =
Pd2Dtest c(v; d) ;
jDtestj e ectiveness(v; Dtest) = max(0; 2 accuracy(v; Dtest) 1): These definitions work best with a balanced problem se+t Dtest, where the set of positive + problems Dtest and that of negative problems Dtest nDtest have approximately the same size. In a balanced situation, a verifier with effectiveness 0 does no better than guessing, whereas a verifier with effectiveness 1 is always correct.
To measure the similarity between two verifiers v; w 2 V , we do not directly compare their answers but focus on the errors they make and assume that two verifiers are similar if they tend to make the same errors. We compute the Pearson correlation coefficient between the corresponding error-vectors c(v; ) and c(w; ): (v; w) = qP
Pd2Dtest (c(v; d) d2Dtest (c(v; d) v)(c(w; d)
w) v)2qPd2Dtest (c(w; d) w)2 ; where v = accuracy(v) and w = accuracy(w).
There are edge cases with vanishing denominators: In case that c(v; d) = 0 for all d 2 Dtest or c(v; d) = 1 for all d 2 Dtest (similarly for w), the above expression is undefined and we set (v; w) := 0 if v 6= w and (v; v) = 1. Note that such verifiers have not been observed in practice yet—it would imply the existence of a perfect verifier for that particular set of problems. But in cases where dtest is small, it could easily happen—however, such small scenarios are not our aim.
For each verifier v 2 V , we define its coverage as a real number in (0; 1] with the following intuition: If the verifier is unique of its kind (correlation with other verifiers near zero), the coverage should be 1. If there are k > 0 other verifiers which give answers almost equal to v, the coverage of v and its k related verifiers should be k +11 , such that these “redundant” verifiers together via their coverage scores will contribute as much to an obfuscator’s performance as one verifier which is unique of its kind will. This motivates the following definition, where 2 [0; 1] is a fixed constant: 0
X (wv;2wV) : Since (v; v) = 1, the coverage is always in (0; 1]. Choosing a larger value for diminishes the influence of many small correlations in contrast to a few bigger ones. We pick = 0:5 to capture all real similarities while reducing the noise of correlations which are rather coincidences on a finite set of test instances. For each verifier v 2 V , we then define its importance as importance(v) = e ectiveness(v) coverage(v): + We now quantify the unambiguity of a problem d 2 Dtest as a weighted average of the verifiers giving the correct answer to d: unambiguity(d) =
Pv2V c(v; d) importance(v)
Pv2V importance(v) :
With the above definitions, we can now define our measure of obfuscator safety. Thinking in terms of the performance matrix from Table 1, an entry of the form T ! F (i.e., successful obfuscation), corresponding to a verifier v 2 V and a problem d 2 Dtest, gives points equal to
importance(v) unambiguity(d): For entries of the form F ! T, the same amount describes the number of points subtracted for a counter-productive obfuscation attempt. The world ranking score of the obfuscator o is the sum of the points awarded or subtracted for each combination of verifier and verification problem.
To put it differently, recall that o(d) denotes the obfuscated problem for d 2 Dt+est, where the texts of known authorship are unchanged but the text of unknown authorship is obfuscated. Denote by c(v; o(d)) the answer of the verifier v 2 V to the obfuscated problem (i.e., 1 if v correctly reports “same author” and 0 if not). Then the world ranking score of the obfuscator o equals
X
X(c(v; d) d2Dt+est v2V
c(v; o(d))) importance(d) unambiguity(d): The measure is named world ranking score since it allows to incorporate all available verifiers and verification problem corpora to produce a single numerical value evaluating the obfuscator’s safety with respect to the given verifiers and verification problems. 2.2
We give the following a-priori arguments why we hypothesize that the world ranking score satisfies the three fairness axioms.
To the first axiom: We decided to quantify effectiveness using accuracy. An ineffective verifier will only obtain a small effectiveness score, such that the influence of its decision changes on the final score are not as high as the influence of an effective verifier. Moreover, it is reasonable to assume that an effective and an ineffective verifier will have small correlation, such that adding ineffective verifiers will not change the importance scores of the effective verifiers nor the unambiguity scores of the problems. Therefore, an ineffective verifier has only little influence on the final world ranking score of an obfuscator.
An ambiguous problem should get a small unambiguity score, such that the overall influence of its obfuscated version on an obfuscator’s world ranking score should be small. Adding ambiguous problems will, in general, reduce the effectiveness scores and increase the coverage scores, as most verifiers will effectively guess their answer. However, this should uniformly affect all verifiers and all obfuscators, such that the world ranking scores with respect to a fixed corpus are comparable.
Adding a variant v0 of an existing verifier v 2 V will not change the effectiveness score of any existing verifier and not change the coverage scores of verifiers w 2 V which are not similar to v. This will likely leave both the problem unambiguity scores and the overall obfuscator world ranking scores mostly unchanged, or at least affect them in a way that preserves the general proportions (i.e., the most unambiguous problems should remain the most unambiguous ones, even though their actual ambiguity scores may change). 2.3
Recall the definitions of recall and accuracy of a verifier v 2 V : acc(v; Dtest) = jfd 2 Dtest : v(d) = (d)gj ; jDtestj
+ rec(v; Dtest) = acc(v; Dt+est) = jfd 2 Dtest : v(d) = (d)gj : + jDtestj The performance of an obfuscator can therefore be measured by the change of recall: rec(o; v; Dtest) = rec(v; o(Dtest)) rec(v; Dtest): The impact of o is
rec normalized to [ 1; 1]: imp(o; v; Dtest) = 8 < : rec(o;v;Dtest) rec(v;Dtest) rec(o;v;Dtest) 1 rec(v;Dtest) if rec(o; v; Dtest) < 0; otherwise, : If V denotes the set of available verifiers, the average impact of an obfuscator o is avg imp(o; V; Dtest) = 1
X imp(o; v; Dtest): jV j v2V
The average impact is the measure used for the obfuscation shared tasks in PAN 16 and PAN 17 to automatically evaluate an obfuscator’s safety. This measure does not satisfy the earlier defined fairness criteria: We will later see that the performance of the verifiers used here differ enormously. For an obfuscator o and a verifier v such that obfuscation by o decreases the recall of v (which is usually the case), the impact is, by definition, imp(o; v; Dtest) = rec(v; Dtest)
rec(v; o(Dtest)) rec(v; Dtest) = 1 rec(v; o(Dtest)) rec(v; Dtest) =1
+ fd 2 Dtest j v(o(d)) = (d)g :
+ fd 2 Dtest j v(d) = (d)g
If, e.g., o flips half of the correct decisions of v and leaves the wrong decisions as they were, the impact factor is 21 . Achieving an impact factor of 12 is therefore easier for weak verifiers (change few decisions of an ineffective verifier) than for more effective ones (change many decisions of an effective verifier). This is counter-intuitive and directly contradicts our first fairness principle above (the verifier effectiveness axiom). Of course, one has to be careful when using only recall (and not precision) to describe “effective verifiers”, but a high recall is not a good indicator for ineffective verifiers.
Moreover, there are examples of similar verifiers: Jankowska et al. submitted a
verifier in 2013 [
Finally, we will present evidence that there are very ambiguous as well as very unambiguous problems in the test corpora, an important distinction not reflected by the average impact measure. The idea is that fooling a certain verifier in an unambiguous problem is supposed to be more difficult than fooling it in an ambiguous one, so that success in the more difficult task should get a better reward than success in the easier one. However, one could make the non-trivial, yet reasonable assumption that each obfuscator which successfully (against a particular verifier) obfuscates problems up to some degree of unambiguity will also successfully (against the same verifier) obfuscate more ambiguous problems. Under this assumption, it is not necessary unfair to simply count the number of flipped decisions independent of each problem’s ambiguity (though it also would not be unfair to take the ambiguity into account). This assumption can, however, be questioned, e.g. by pointing out that there are random effects involved such that an obfuscator may by chance fail to fool a verifier in some ambiguous problems although the obfuscation is successful in some more unambiguous cases. It is therefore desirable to have a measure whose fairness can be justified without relying on this or a similar assumption. 3
The two approaches submitted to this year’s edition of our shared task are of a more
rule-based flavor, but with different aggressiveness. The rather conservative rule-based
replacements of Kocher and Savoy’s approach [
As for the resulting texts, the rather conservative strategy does not change a lot and aims to keep the original text quality. A rather problematic issue of the approach is that it seems as if the sentences in the obfuscated text were re-ordered (the whole second paragraph is often directly inserted after the first sentence while the second sentence comes rather late). It is unclear how reordering a text changes its style, so that this issue might be due to some implementation error, and is the main cause for the rather sub-optimal scores for soundness. A more detailed analysis of the text quality of this approach follows in the evaluation (cf. Section 4).
Rahgouy et al. The approach of Rahgouy et al. [
The resulting text will usually be changed more than with Kocher and Savoy’s approach. Depending on the word similarities and the “correctness” of splitting / combining sentences, some obfuscated text passages read very well while others are changed to often even a contrary meaning. Again, the text quality of this approach is discussed in Section 4. 4
As in the past two years, we automatically evaluate the safety of the submitted
obfuscation approaches against 44 authorship verifiers that participated in the previous three
shared tasks on authorship identification at PAN 2013–2015. We employ the new
scoring scheme to obtain the first “world ranking” for obfuscation and contrast it with the
previous way of evaluating safety. Sensibleness and soundness of the obfuscated texts
are assessed manually via human inspection in the same way as before.
The evaluation setup is based on the cloud-based evaluation platform TIRA [
Table 3 contrasts our new “world ranking” scheme described in Section 2 with the
“legacy” safety measure we employed in the previous two years. It can be seen that
the ranking of the approaches does not change from one measure to the other with
Castro et al.’s approach [
Interestingly, the two obfuscator approaches from this year’s shared task (green
entries in Table 3) are not able to break into the top-3 safety ranks overall. But their
characteristic is also a less aggressive obfuscation with sometimes no changes in a
sentence, so that the resulting text quality is not changed too much from the original (cf.
next section for the respective evaluation of sensibleness and soundness). It should also
be noted that even the “safest” of the approaches submitted so far still leaves a lot of
room for safety improvements alone. The perfect obfuscator (i.e., the one that would
be able to change every correct “same author” decision of each of the 44 verifiers in
the evaluation) would obtain a world ranking score of 1093.49—thus Castro et al.’s
best approach so far has not even made “half the way” to perfect obfuscation. A more
1 www.tira.io
fine-grained analysis of the old and new safety scores per corpus can be found in
Appendix B while Appendix A gives some more experimental details on the robustness of
the new world ranking.
As in the previous two shared task, a human assessor conducted a manual assessment
on problem instances 6, 135, and 430. Spot checks on other instances indicated that
the overall characteristics of the output texts are similar on other instances. The human
assessor started by reading the obfuscated texts without knowing which was the output
of what approach. During this reading phase, the assessor marked up errors (typos,
grammar) and assigned school grades (on a scale from 1 (excellent) to 5 (fail)) for the
sensibleness of each of the sample problem instances. The sensibleness scores obtained
in the previous years were a grade 2 for Mansoorizadeh et al.’s approach [
After grading the sensibleness of the obfuscated texts, the assessor read the original texts and judged the textual differences in various ways to evaluate the soundness of the obfuscated texts on a three-point scale as either “correct”, “passable”, or “incorrect”. The obfuscated texts of Mihaylova et al.’s, Keswani et al.’s, Bakhteev’s and Khazov’s, and Castro et al.’s previous years’ approaches were all judged “incorrect”, while Mansoorizadeh et al.’s very conservative approach from 2016 achieved “correct” and “passable” scores. This year’s approaches (Kocher and Savoy’s and Rahgouy et al.’s) both got “passable” as their average judgments—but for different reasons: With regard to Kocher and Savoy’s approach, almost everything was left as it was with the main problem that the ordering of the sentences was changed which in some passages caused a rather odd reading “flow” resulting in a “‘passable” for all three checked texts. With regard to Rahgouy et al.’s approach, the judgments are again more wide-spread with one text being “incorrect” since almost all sentences were wrongly split into parts, one document being “passable” and one being “correct” since hardly anything was changed. 5
In the third year of evaluating author obfuscation approaches in terms of their safety against the state of the art in authorship verification, two new approaches were added to the five approaches from the previous years. The best-performing obfuscator today achieves about 43% of the score a “perfect” obfuscator can achieve on our testbed with 44 authorship verifiers on the PAN authorship verification corpora from 2013–2015. Such a perfect obfuscator would be able to flip any correct decision for “same author” by any verifier towards choosing “different author”.
We have developed a new “world ranking of obfuscators” in terms of safety. The idea is that an obfuscator obtains points per flipped correct “same author” decision, dependent on how unambiguous a problem is (i.e., the more unambiguous the more verifiers solve it correctly before obfuscation) and on the effectiveness of a verifier on unobfuscated problems (more points if a more effective verifier is fooled). This new safety measure is fairer than the previous impact measure, where scores were computed independent of a problem’s unambiguity and a verifier’s effectiveness.
Still, even with the new scoring scheme, the actual safety ranking does not change but the relative differences between the verifiers are more pronounced in the sense that they now include information about the actual problem unambiguity. The safety-wise best-performing approach of Castro et al. from 2017 was not beaten by the two new obfuscators, whose main focus seems to rather be text quality (i.e., soundness and sensibleness). As in the previous years, text quality was measured by manual inspection. It became clear that sometimes even small changes can “destroy” a particular sentence or text passage (distorting its meaning or decreasing readability). Unsurprisingly, the least safe approach of Kocher and Savoy from this year’s shared task obtains the best scores among all approaches with respect to soundness and sensibleness—it simply does not change a lot. While still being readable, the obfuscated texts of the safest approach (Castro et al.) are rather poor when it comes to soundness.
It is still an open problem to develop obfuscation technology that is safe (even the best one is not half the way to a perfect obfuscation safety) while not harming paraphrase soundness or readability too much. Paradigmatically, there are still more or less only two groups of obfuscation approaches: (1) the ones that are somewhat safe but that produce rather unreadable text or text that is neither sound nor sensible, and (2) the ones that produce sound and sensible texts but that are not really safe against authorship verification.
As hinted in the previous shared task editions, a significant improvement of current
obfuscation technology might require a much better consideration and integration of
the surrounding context when, for example, replacing, adding, or removing words, and
better ways of reordering clauses in sentences. Ideas in that direction could be to apply
constrained paraphrasing [
The remaining challenge of evaluating author obfuscation approaches properly and at scale does not seem to be safety. The new “world ranking” provides for a fair and robust tool that incorporates future verifiers, obfuscators, and corpora. What is missing are new and improved technologies for recognizing paraphrases, textual entailment, grammaticality, and style deception—the existing technology is not mature enough to easily replace manual inspection for evaluating soundness and sensibleness.
We thank the participating teams of the three editions of this shared task. A
We empirically test whether the parameters underlying the new world ranking are distributed as expected and how robust the new world ranking is against the addition of “random” verifiers or verifiers that are very similar to already existing ones. A.1
First, to ensure that the considerations underlying our new safety evaluation approach make sense in the given setup, we confirm that the main verifier and problem characteristics of effectiveness, coverage, and unambiguity vary sufficiently among the considered verifiers and datasets. We confirmed our findings on all PAN verification datasets but only give plots and explanations on the basis of the PAN 15 data since the observations on the other datasets are similar. eag 0:6 r e ov 0:4 C 0:2 0:8 y t iug 0:6 i b am0:4 n U 0:2 0 0 60
Figure 1 (left) shows the accuracy and coverage scores of the 40 verifiers that we
were able to run on the original PAN 15 data. Many of the verifiers seem to follow
rather unique approaches such that 19 of the 40 verifiers have a coverage score of 1.
Note that some verifiers perform so poorly on the test data that our “best threshold”
method chooses a threshold of +1 or 1, forcing them to always respond
“different authors” or “same author”, respectively. This behavior is mainly due to employing
verification models that were trained on a different PAN corpus that are now tested on
PAN 15 data (we did not re-train models that were submitted in another year), leading
to multiple verifiers giving identical answers, which in turn then have very low
coverage scores (between 0:11 and 0:123, with accuracies between 0:5 and 0:55). The two
implementations of Jankowska et al. [
Figure 1 (right) exemplify the unambiguity scores for the PAN 15 test data. Following the definition of unambiguity, a clear problem (i.e., easy to identify the author) has a high unambiguity score whereas an obscure one has a low score. The observation that clear and obscure problems are rather evenly spread does not only hold for PAN 15 but also for the other datasets.
Our inspection of the different PAN authorship verification corpora shows that the effectiveness, similarity, and unambiguity scores vary in the expected scope such that the definitions reasonably capture differences.
Since the PAN shared tasks are intended to continue even after the respective conferences, everyone can submit new approaches for authorship verification at any time. The measures underlying the “world ranking” are directly able to incorporate any such new approaches and thus are able to always include the state of the art in authorship verification. Also creators of obfuscation approaches, of course, may submit new verifiers. This possibility could in principle also be exploited to submit verifiers with the intention of boosting a specific obfuscator’s world ranking score: – One could let the obfuscator leave a certain “watermark” in the text that a newly submitted verifier could then detect to turn all its decisions on such watermarked cases to “different authors”. – Somewhat similarly, one could develop a verifier just focused on those aspects of the texts that the desired obfuscator will manipulate (e.g., counting the occurrences of “it’s” vs. “it is”). Such a verifier might work with some success on unobfuscated texts but then not at all on the obfuscated texts of the specific obfuscator since the obfuscator probably will have removed most of the features used by the verifier. – One could also submit a slight variation of an existing verifier which is particularly vulnerable to the to-be-boosted obfuscation approach by altering the answers only in low-confidence cases. The verifier variant then should easily be fooled by the obfuscator while retaining reasonable performance on unobfuscated texts. Such and similar deliberate attempts to boost an obfuscator’s score should be considered “unfair play” and unscientific. In case of being detected—probably rather difficult to do automatically—, such verifiers should probably be removed entirely instead of just being ignored in the world ranking.
The second option could still yield a reasonable and well-performing approach in authorship verification, in which case it is a valid contribution. If the verifier, however, does not perform better than guessing, its effectiveness score should become too low to make any difference. If the verifier significantly resembles an existing one (as in the third option), the corresponding coverage scores would be lowered accordingly such that the overall score of all obfuscators ultimately should remain stable.
These assertions can be tested (effectiveness and coverage for some potentially adversarial submissions). We perform two experiments, in which we add 40 mock verifiers to the existing ones and look how the characteristics of the verifiers and the problems behave, and whether the overall scores for the different obfuscators and test corpora change. In the first experiment, the guessing attack, the mock verifiers just choose their confidence scores randomly in [0; 1] (uniformly distributed). We expect that those mock verifiers get poor effectiveness and high coverage scores, whereas the characteristics of the original verifiers and the problems remain stable, as well as the obfuscators’ scores. In the second experiment, the variation attack, each mock verifier is obtained from an existing verifier, replacing 10% of its decisions (on the original and the obfuscated problems) by random confidence scores in [0; 1]. We expect that the effectiveness scores of the new verifiers are a bit lower than those of their originals, that these mock verifiers get low coverage scores in general, and that the coverage scores of the existing verifiers decrease overall, in particular for those which have been copied. Note that in both experiments, the effectiveness scores of the original verifiers remain stable by definition.
The left column of Figure A.2 (Experiment 1: Guessing Attack) shows how the characteristics change after attacking the ranking by adding 40 mock verifiers which only guess their responses. The effectiveness scores of the mock verifiers are very low— as expected since they are just guessing. Note that the effectiveness scores of the original verifiers remain stable (by definition). The coverage scores of the badly performing verifiers, which, after choosing optimal thresholds, give constant answers, shrink a bit Experiment 1: Guessing Attack
Experiment 2: Variation Attack 20 40 60 80 10 20 30
40 0:6 0:4 0:2 0 1
0 0:8 0:6 0:4 0:2 0 0 1 0:8 0:6 0:4 0:2 0 0 0:6 sse0:4 n e v i t c e ff 0:2 E 0:8 eg 0:6 a r e ov 0:4 C 0:2 0 1
0 0 0 1 0:8 y t igu 0:6 i b am0:4 n U 0:2 0
0 since also some of the mock verifiers also perform very badly. The other coverage scores remain mostly stable—as expected. The problem unambiguity scores move towards 0:5 since about half of the guessing verifiers guess the correct answer—if every verifier guessed its answers on an instance, half of them will guess correctly, resulting in a unambiguity score of exactly 0:5 for that instance.
The right column of Figure A.2 (Experiment 2: Variation Attack) shows the behavior of the three characteristics after attacking the ranking by copying (with a 10% random change) the 40 original verifiers. Similar to the first experiment, the observations are close to our expectations. The mock verifiers have roughly the same effectiveness as their originals, sometimes diminished due to the 10% random choices. Those verifiers which have not been copied mostly have the same coverage scores (except some few side effects), whereas those which have been copied have significantly reduced coverage scores. Except some few random outliers, these reduced coverage scores are also observed for the copies of the verifiers (often slightly higher which may be explained by the “unpredictable” 10% random choices). Similar to the first experiment, the random 10% choices explain the slight drift of the unambiguity scores towards 0:5.
The obfuscators’ scores in the two experimental setups from above (random mock verifiers and verifier variants) are given in Table 4. It can be seen that adding the mock verifiers does change the obfuscators’ scores, though most changes are small compared to the actual differences between the obfuscators under consideration. In particular, the ordering of the obfuscators for each dataset remains stable, with the two exceptions that Keswani et al. and Castro et al. change their ranks for PAN13 and that Mansoorizadeh et al. and Castro et al. change their ranks for PAN14-EE for the setup with the 40 random mock verifiers. In both experiments, some scores are increased whereas some are reduced; however, a general trend towards higher scores is observable in both experiments. This trend may be explained statistically, noting that those random verifiers with an accuracy 0:5 on the original training data will still have an expected accuracy of 0:5 on the obfuscated data, thus changing probably some correct positive decisions to negative ones while having a moderate effectiveness score. Those random verifiers which perform poorly on the original data (i.e., that have an accuracy of 0:5), are more likely to correct wrong decisions by chance, but get a smaller effectiveness score such that their final influence is not as high.
Other than that, the changes in the obfuscators’ scores are probably best explained as just random effects, since adding 40 random mock verifiers to 40 real verifiers will induce them. The remarkable thing here is that our proposed measure retains relatively stable output under such heavy modifications of the input data.
B
jY j
The best-performing approach this year was submitted by Rahgouy et al., which
achieves 4th rank overall across the three years as per average impact; the average
impact quantifies the averaged ratio of true positive decisions turned false negative.
However, this result must be taken with a grain of salt since this approach basically removed
large parts of the original text. The approach of Bakhteev and Khazov [
Team [Reference]
jY j AUC final acc rec avg imp