A common task in the field of stylometry is to attempt to resolve questions of authorship. In "authorship verification" the basic question is "how likely is it that a given document is by a specified author?". Since the answers to these questions can have important repercussions, whether legal, literary, or social, it is important that the methods employed by practitioners be reliable; but in order for the results to be believed they should also be transparent and interpretable. The General Imposters method (henceforth GI), originally formulated by Koppel and Winter in [9], has become one of the standard methods in stylometry for authorship verification. After strong performances in the PAN 2013 and 2014 competitions, it was implemented and used by M. Kestemont et al. in an influential study [6], improved by Potha and Stamatatos in 2017 [14], and is now available in the well-known R package stylo [4].
In this paper I describe an update to the open-source Python implementation by Kestemont et al. [5] called Bootstrap Distance Imposters (henceforth BDI) which incorporates several of the improvements proposed since the last release, as well as introducing a novel method of bootstrapping that has several attractive properties when compared to the reference algorithm. The two approaches are benchmarked using the problems from the PAN 2014 author identification task [16], and some additional properties are showcased via real-world case studies.
The Imposters method of authorship verification examines a question text in order to determine whether the text was written by a candidate author. To do this, it performs bootstrap comparisons of the question text to candidate texts and several (perhaps very many) imposter texts which should ideally be chosen to be similar in genre, topic and register. If the question text is markedly more similar to the candidate author's 'style' than to any of the imposters then we infer, with some level of statistical likelihood, that it was authored by the candidate. The method of Imposters can be used with any features that reflect style, although it is most commonly applied to distances between character π-gram or word frequency vectors.
Note that while I speak of 'statistical likelihood' I am carefully avoiding the words 'probability' and 'confidence'. The GI method is, in machine learning terms, an ensemble classifier. These classifiers regularise well, but while they produce a real-valued output, it is problematic to interpret this number as a probability. Many factors can make the verification results less reliable-the lengths of the texts, the language in which they are written, the closeness of the imposter texts (dissimilar texts make a positive attribution less convincing), and the amount of available data (a lack of comparison data risks bias). Verification problems in the real world are seldom under ideal conditions, and there is no magical formula by which the uncertainties imposed by the problem setting can be convincingly rendered as a forensic probability. Never the less, the Imposters method has a well deserved reputation for robust, understandable results, even in the face of severe limitations (for example in the length of samples or the availability of suitable imposters).
Figure 1 attempts an intuitive explanation of the basic operation of standard GI and the key modification used in BDI. The output of the Kestemont GI classifier is a percentage of binarized 'votes' (the number of times a candidate text was closer than an imposter). In contrast, the raw output from the BDI algorithm is a bootstrapped distribution of differences. At each step, the distance (with a bootstrapped feature set) between the candidates and the imposters is recorded, using any vector distance measure π βΆ β π Γ β π β β. If the candidates are further, the difference between the distances is negative, if closer it is positive. If these individual distances follow a Gaussian distribution (which is a reasonable prior expectation) then their difference is also Gaussian. Expressing the results this way has some advantages. The first is that we can differentiate a negative result (not the candidate) as either 'none of the above' or 'more like an imposter' (the true author is in the imposters set). A 'none of the above' result would have a statistically expected distance of zero (equally unlike the candidate and the imposters), and so we would see a distribution centred around 0. 1 On the other hand, 'more like an imposter' results show distributions centred around a negative value (examples of this can be seen in Section 4 below). The other advantage is that for strong positives, we have additional data about the match. Distributions centred around larger positive numbers are better matches, but distributions with high variance show more feature dependence (since the strength of the match varies greatly depending on the bootstrap feature sets). In summary, positive matches (with most or all of the probability mass above zero) can be much more meaningfully compared.
It is worth noting here that the overall best performing method at PAN 2014 by Khonji and Iraqi [8] also modified the classic GI algorithm to utilize the distance between vectors (in that case the relative distance of the test vector to candidates vs imposters was considered as part of the decision function for a 'standard' voting-based classifier), so this paper is not the first to recognise the value of this additional information.
Based on the BDI algorithm, which outputs a distribution, it is obviously useful to have a summary statistic that can be interpreted as evidence for authorship verification tasks. For this paper I used a simple approach that considers the amount of probability mass that lies above 0. If every test is closer to a candidate than an imposter then the result will be 1, if every test is more like an imposter, it will be 0, etc. This is implemented simply as the inverse percentile of (a distance of) 0. Thus armed with a method that outputs a 'probability-like' result in [0, 1], I wrapped the code in a classifier that follows scikit-learn [13] conventions like fit() and predict_proba() and evaluated the BDI classifier directly against the updated Kestemont GI Order2Verifier using the PAN shared tasks from 2014 and 2021. This provided a convenient benchmark, and also an opportunity to compare the results against a number of other verification approaches.
The C@1 metric introduced in PAN 2014 rewards (or at least penalises less harshly) classifiers that choose not to answer some problems. This leads naturally to algorithms that use the training data to define classifier output ranges that will be assigned to 0.5 (indicating an unanswered problem). In the case of classic GI, this means that classifier scores (vote percentages) within certain ranges will be rectified to precisely 0.5, while positive and negative classifications are shifted to the ranges above and below that value. This (hopefully) improves the C@1 score as compared to basic accuracy. In the PAN 2020-21 competitions, a similar measure was used called F 0.5π’ (introduced in [2]).
The score shifting method implemented in Kestemont GI attempts to produce something more like a probability by linearly scaling the output values. The code defines an upper and lower bound for the unanswered region, p1 and p2 The scaling code (in Python) looks like this:
for score in list(scores):
if score <= p1: new_scores.append(rescale(score, min(scores), max(scores), 0.0, p1)) elif score >= p2: new_scores.append(rescale(score, min(scores), max(scores), p2, 1.0)) else:
new_scores.append(0.5)
Scores below p1 are scaled to [0, p1), scores above p2 are scaled to (p2, 1], and the rest are coerced to 0.5. There is an issue with this scaling algorithm, however. Since p1 and p2 are chosen by grid search to maximise the PAN score, the score shifter sometimes fits values for p2 that are well below 0.5. This can lead to decisions that are defined as positive (since they are above p2) being scaled to below 0.5, where they are evaluated by the scoring metrics as a negative result (and scored as such). In the updated code, I modified the shifting code to scale more simply to [0, 0.5) (negative), 0.5 (unanswered), and (0.5, 1] (positive). This does not retain the global distributional properties of the original results (as implemented in [6]). Based on the evaluation problems, the BDI algorithm is not as sensitive to this score shifting, deriving only a modest benefit from fitting. The fitting process raises natural questions about the representativeness of the training data, and also causes some problems in domains that suffer from limited data availability (where it can be hard to sacrifice data for training). In these circumstances, BDI works well with manual score shifting, allowing the user to choose a confidence level empirically.
As is the nature of software, the code in the repository documenting the GI algorithm and for the related work on the Caesarian corpus no longer ran. I reworked the code slightly, and made the following small changes, which are available in my own repository [11].
β’ Update the code to work with Python 3 (these minimal changes have been incorporated into the original repository based on a PR); β’ Implement a fast 'nini' metric (fuzzy Jaccard similarity) as described in [12]; β’ Implement the Potha & Stamatatos 'ranking' improvement for the consensus score 2 described in [14]; β’ Remove most non-core code; β’ Modify the score-shifting algorithm, as described above.
The BDI classifier was compared head-to-head with the updated Kestemont Order2Verifier on the full PAN 2014 evaluation corpus, which is a challenging set of varied authorial styles across four languages. 3 Additionally, both verifiers were evaluated below using the PAN 2021 problems, since that competition included some deep-learning approaches, discussed further below. For this evaluation I used character π-grams, since that feature universe is a generally reliable and uncontroversial choice for modern stylometric work. There may be feature universes that perform better, or features that work better for a specific task, but character π-grams are 'solid if boring'. Likewise, the π-gram frequencies are π§-scaled (based only on the training variances) since this is the standard approach. I tested two π-gram configurations, 2,3,4-grams and 2,3,4,5-grams, 10,000 max features, with fitted and manual score shifting. For the Order2Verifier bootstrapping was performed at 50% (5000 random features per bootstrap iteration) since that is the configuration used in [6]; for BDI, bootstrapping was performed at 33%. Finally, for the distance metric used to determine 'closeness' at each step I tested the cosine distance (the most traditional choice) and the minmax (RuΕΎiΔka) metric promoted by Kestemont et al. 2 Instead of a strict 1 (candidate closest) or 0 (candidate not closest), Potha & Stamatatos proposed a score improvement based on the ordinal rank of the closest candidate, so if a candidate was the second-closest, the score for that iteration would be 1 2 . The same paper proposed a distance-based culling method to select more relevant imposters, but this was not implemented because of poor Big-O complexity. 3 There is a small inconsistency that I was unable to resolve-the only copy of the verification problems I could find were archived in the Kestemont repository, but they are apparently missing 50 of the 'Dutch Reviews' evaluation problems, so there are a total of 746, versus 796 reported in the PAN 2014 wrapup report. in their original paper. Consistent with those results, the minmax metric appears generally superior (see Table 1). The fully reproducible testing code is available in the supplementary repository [10].
Several metrics are reported, some of which (C@1, F 0.5π’ ) are rare outside the PAN competitions, along with the overall evaluation scores for PAN 2014 (AUCΓC@1) and PAN 2021 (mean of the five balanced measures).
The format of the PAN 2014 problems makes it straightforward to apply the BDI and GI verifiers and directly compare the results. As can be seen from Table 1, the performance of the BDI classifier is consistently strong, with only a small boost in PAN score resulting from fitting the score shifter. When evaluated as a global micro-average (as done in the competition), the 'best' BDI verifier outperformed both updated Kestemont GI and the winning PAN 2014 entrant. It must be noted, however, that the individual corpus results are not dominated by either verifier but vary considerably according to evaluation strategy and corpus. The strength of the BDI classifiers, however, is twofold: first, they do not really require any training corpus, and second, the BDI approach is high precision (it yields very few false positives), making it a conservative verifier whose positive results are reliable (at the cost of more false negatives). This can be seen clearly in Figure 2 in which the best performing GI classifier is compared to a manually fitted BDI classifier (results between 11% and 89% are left unanswered) using the same features and metrics. This is not the best performing BDI verifier: it was chosen because these verifiers have an almost identical overall PAN14 score but quite different detailed characteristics. The results for each subcorpus are broken down in more detail in Tables 2 & 3. In those tables, the respective results for the en_novels set are of particular interest. This sub-corpus was extremely challenging, causing problems for all entrants due to the shared-genre nature of Lovecraftian horror and the unifying force of its pervasive style (explained in more detail in [16, p. 882]). BDI underperformed here according to the PAN metrics, but did so conservatively, with a perfect precision score of 1.0.
Since the state of the art in authorship verification is now exploring the possibilities offered by deep learning, the BDI verifier (and the Order2Verifier) were also evaluated using the data from the PAN 2020-2021 shared task. The final evaluation report from that task is available in [7]. The 2020-21 tasks used English-language fan-fiction, which is available in vast amounts. Although the huge amount of training data and the limitation to English makes the problems somewhat less interesting, it is interesting to know how much we 'lose' by using simple machine learning approaches that can be widely applied as compared to cutting-edge techniques applied to best-case scenarios. The large amount of training data (the large training set contained 176,000 total text pairs, and even these were synthetically augmented by some teams!), as well as the more recent date, meant that the results here were dominated by complex deep learning models, with the winning entry being an extremely impressive siamese network using four subcomponents [3]. It is not possible to provide a strict 'apples to apples' comparison, since the entrants were required to assess each problem as an atomic pairwise determi- nation (were these two texts written by the same author). For the Imposters methods in general, at least one sample from the candidate author is required, and a pool of imposters is used at each step. However, the evaluation was done as fairly as possible. Although the BDI and GI verifiers are not 'trained' per-se, both the score shifting parameters for the fitted shifters as well as the variances used in π§-scaling were derived from a subset of the small training corpus. This is in spirit with the real PAN competition where the final evaluation dataset was fully blinded, unlike in previous years. The amount of training data employed was very small, using just 4256 texts from the 'small' training set, which contained 106,000 texts. The final evaluation used texts from the 2021 evaluation set (19999 pairs) which contained no texts from authors in the training sets. To fairly assess AUC, a roughly even split of positive and negative determinations are required. In addition, since the 2021 shared task was focused on previously-unseen authors, unseen authors were included as noise in both the comparison set (candidate profiles) and the evaluation set (verification problems). The evaluation set that was eventually used contained 10,152 problems, and was broken down as follows:
β’ 5076 problems from authors with at least one other text, tested against the true label;
β’ 2538 problems from authors with at least one other text, tested against a randomly chosen false label; β’ 2538 problems from unseen authors, tested against a random false label. The candidate/comparison set is composed as follows:
β’ 1692 authors who are never seen again, as noise;
β’ 7451 authors with one comparison sample (who might appear against a true or false label); β’ 905 authors with 5 comparison samples; β’ 94 with 11; β’ 7 with 19; β’ 3 with 29.
As can be seen, the large majority of authors have only one comparison candidate. The unnatural distribution of counts for texts per author is an artifact of the PAN data, and is presumably related to the corpus compilation process. The evaluation process used all of the evaluation texts by authors with two or more total samples, but the bulk of the singleton texts were not used. The full evaluation process is available as commented Jupyter notebooks at the accompanying repository.
Overall, the best performers in training were fitted versions of the 2,3,4-gram Kestemont GI and BDI verifiers, once again with the minmax metric. The full results can be seen in Table 4. Since the bootstrapping process was extremely time consuming on the full evaluation set, only the 2,3,4-gram vectorizer was used. The best performing BDI verifier posted a final overall score of 0.889, using the PAN 2021 metric (the mean of several accuracy measures, calculated using the ofΓcial evaluation code). This would place it near the bottom of the four "strong runner ups" [7, p. 7], still comfortably outperforming the machine-learning benchmarks and 5 human teams. It seems reasonable to assume that a more sophisticated feature vector, such as the one used in weerasinghe21 [17], would improve the performance of both Order2Verifier and BDI. By comparison, the final score for boenninghoff21 was 0.9545. Once again, it is clear from Tables 4 and 5 that the fitting is nice to have in terms of the overall PAN21 score (which uses several balanced accuracy measures), but in fact reduces raw precision, which is relevant where research problems require the smallest chance of false positives. As further exploration, some differential / ablation results are included in Table 5. The results under the shifter method 'fitted (optimal)' calculate the optimal post-hoc p1 and p2 (as discussed above) based on the results from the evaluation itself-in other words it shows the advantage if the training regime were to perfectly represent the probability distribution of the true problems. As can be seen, this is quite minor (an overall score of 0.896 vs 0.889), which is encouraging. However, the relatively strong performance of the manual shifter with the BDI verifier, as well as its higher precision, seems to suggest that the small increase in some of the measures does not warrant the methodological uncertainty (in terms of representativeness and bias) added by the fitting process in general. Additionally, it can be seen that the score shifting optimisation is very sensitive to the evaluation measure, incurring quite a large precision penalty-not a component of the target metric-in order to improve the overall score. Finally, the new addition of 'ranking' was evaluated by ablation. As in the PAN 2014 results (Table 1, PAN14-U), the ranking regularisation appears to offer a small but consistent benefit to AUC and F-measures, which is impressive considering how few of the authors (about 10%) had more than one comparison sample. The unranked verifiers, however, showed even higher precision which may need to be considered for some applications.
This section refers to two verification studies in which BDI has been applied, one of which is, at the time of writing, still in press. These figures are not full summaries of the research, but simply illustrate some of the features of the BDI method that I believe to be useful. As mentioned above, the output of the BDI algorithm is a distribution of distances, not a summary statistic. These examples attempt to show that examining the full distribution conveys extra information and can improve our intuition and confidence in the analysis. Figures 3 & 4 are from an analysis of several poems attributed to Ovid. The aim here was to provide evidence for the genuineness of the Nux, but of more interest in this context is the analysis of the Consolatio ad Liviam. The Consolatio was once considered to be a genuine work of Ovid, but is now accepted by most scholars to be a first-century imitation. By using BDI we attempted to show that metrical technique was a powerful enough stylistic feature to disambiguate even deliberate imitation from genuine works. In Figure 3, we see the value of visualising distributions where all of the distances are positive (closer to the candidate author than an imposter), which would be summarised as a 'probability' of 1.0. This figure measures similarity in terms of lexico-grammatical features, operationalized as character π-grams. In fact, as can be seen, although the chunks from the Consolatio are much more like Ovid than they are like any of the distractor poets, they are not as much like Ovid as most of the candidate comparison works. This kind of comparability between strong matches is very difΓcult with the standard GI approach. However, in Figure 4, which measures metrical features, the difference is clear-the sections from the Consolatio are centered around 0 (or near enough) as compared to the other works where at least 90% of the distribution mass is above 0, supporting a positive attribution. This result suggests that the Consolatio is not Ovidian, but also that it is not a good stylistic match for any of the distractor poets (Tibullus, Propertius, and Catullan elegy). This is consistent with the current (weak) consensus that the Consolatio is a late first-century imitation by an unknown.
Figures 5 & 6 are from an analysis of translator style, examining medieval translations from Greek to Latin. Translator style (as opposed to authorial style) offers a unique set of challenges, covered in much more detail in the full paper [1]. In this study, a small set of function words was used, in accordance with the well-known theory that closed-class words are used more unconsciously, and thus are more indicative of individual preferences than nouns, verbs, and adjectives (which are highly affected by genre and topic). Overall, this was found to be an effective approach. Here, however, I note two more useful properties of the BDI method. The first is that by splitting the work into smaller chunks, and visualising the distribution for each chunk we are able to see the degree of stylistic variation in a single translator. It is also clear from Figure 5 that some passages are less 'stylistically clear', showing much more pronounced spread-this can be interpreted as greater sensitivity to the individual feature subsets. Overall, Figure 5 is centred around a negative value, indicating that it is significantly more similar to one of the imposter translators than to Bartholemew. In Figure 6, we performed the same process for a different text that is a translation of the same work (Aristotle's Rhetoric) generally accepted to be by William of Moerbeke. In the latter case we see the expected result-almost all of the chunks are fairly strongly centred around a positive value. The strength of the match is not as clear as in the Ovidian figures, but this is perhaps to be expected, since the amount of style that a translator brings to a work can be reasonably assumed to be less than that brought by an author (this is a well studied field; see for example [15] with references).
As can be seen from Figure 2, predictions from the BDI classifier (after shifting) cluster strongly at the extremes, with most mis-predictions being high-confidence false negatives. The Kestemont GI classifier shows the most mis-predictions in the central band (near 0.5), which is intuitive if the outputs are interpreted as probabilities. The current link function from the BDI distributions to a 'probability' in [0, 1] is a fairly simple idea, and can almost certainly be improved to produce a smoother and more statistically informed distribution across the output range (perhaps logistic regression, or even empirical distributional statistics). This is left for future work.
The most common goal in authorship verification work is to positively attribute works to authors. In this context, although balanced accuracy is not unimportant, precision (fewer false positives) is often more important than recall. The balanced metrics used for the PAN 2014 / 2021 authorship verification competitions balance overall AUC (false positives and false negatives) with the ability for a classifier to degrade gracefully when the result is unclear. In general, this is a useful innovation, particularly in comparison to standard machine-learning classifiers which are obliged to assign each problem to a discrete class (even if the true author is not one of the available answers). While the widely-used General Imposters method still performs extremely well, it seems wasteful to discard the detailed distance information that is calculated in any case during the bootstrap / voting process.
BDI attempts to address these issues by outputting a full distance distribution which can be manually inspected. As demonstrated in Section 4, this can be very useful when comparing results that are all strong matches. When operating as a summary verifier, BDI tends to be conservative in its positive attributions, particularly when applied to very difΓcult problem sets like the PAN2014 en_novels. In terms of raw performance, the BDI verifier appears slightly stronger than the improved Kestemont GI according to the PAN metrics for both the 2014 and 2021 problems, while also offering superior interpretability. The advantage of the BDI verifier is even clearer when score shifting is not used. Overall, the BDI approach seems to be a strong choice, especially where training data is limited and/or reliable positive results are more important than balanced performance metrics.
The preprint may be found at https://github.com/bnagy/bdi-paper. All code and data is available under CC-BY, except where restricted by upstream licenses. The code repository includes full reproduction data and code for the evaluation, as well as various supplemental figures and explanations.
This work was supported by Polish Academy of Sciences Grant 2020/39 / O / HS2 / 02931.