We adopt Koppel et al.'s unmasking approach [5] as the major framework of our authorship verification system. We enrich Koppel et al.'s original word frequency features with a novel set of coherence features, derived from our earlier work [2], together with a full set of stylometric features. For texts written in languages other than English, some stylometric features are unavailable due to the lack of appropriate NLP tools, and their coherence features are derived from their translations produced by Google Translate service. Evaluated on the training corpus, we achieve an overall accuracy of 65.7%: 100.0% for both English and Spanish texts, while only 40% for Greek texts; evaluated on the test corpus, we achieve an overall accuracy of 68.2%, and roughly the same performance across three languages.
Authorship verification, a sub-task of authorship identification, deals with the demand of identifying whether a two documents are written by the same author or not. Typically, a set of documents which are known to be written by the author of interest is given, and an authorship verification system needs to determine whether a given unknown document is written by this author.
We follow the unmasking approach described by Koppel et al. [
Unmasking [
Our modified unmasking approach is as follows: From all known documents in the training corpus, we extract: (1) Ssame, the set of pairs of documents written by same authors; (2) Sdiff, the set of pairs of documents written by different authors. For each document pair hdi; d ji, written by author Ai and A j respectively, the two documents, di and d j, are segmented into equal-sized and non-overlapping small chunks. If the number of chunks of either document is less than 5, an up-sampling is first performed to pad the size to at least 5. If the number of segmented chunks of each document is unequal, a balanced sample is obtained by randomly discarding surplus chunks in the larger set. The following procedure is repeated N times. 1. A weak classifier with a set of unmasking features is trained to label each chunk as being from document di or d j. The sampling is repeated five times, and the averaged leave-one-out cross-validation accuracy is reported to represent the discrimination performance. 2. Remove the top 3 most discriminating features of the weak classifier, and repeat
Step 1 using the remaining features.
Pair hdi; d ji is therefore unmasked by the degradation of the cross-validation accuracy
after each iteration of feature removal. Such degradation of accuracies is encoded by
a numeric vector using the original representation in [
Our important extension to Koppel et al.’s unmasking approach is the enhancement of
the features used in building the weak classifiers for unmasking the degradation curves
(Step 1 in Section 2.1). Although Koppel et al.’s word frequency feature set achieved
competitive performance for verifying novel-length texts, we found that word frequency
features are too unreliable for much shorter texts. Therefore, we use a more
comprehensive feature set, including various rich linguistic features, which can be partitioned into
two categories:
Coherence features As we have shown in earlier work [
Since we are not aware of any available coreference resolution systems for languages other than English, it is nontrivial to extract coherence features for Spanish and Greek texts. However, we believe that, while surface-form authorship features, such as word usage, are generally obfuscated in the process of being translated to English, coherence features, as a kind of discourse-level feature, are relatively well preserved. Therefore, we first use the Google Translate service4 to obtain their English translations, then perform the entity extraction on these translations.
Stylometric features In addition, we use a set of well-established stylometric features,
the majority of which are from our earlier work [
The total number of features is 538 for English, 568 for Greek, and 399 for Spanish texts. 2.3
There are a few parameters that can be adjusted in our approach. We tested several parameter combinations, and decided to use the following configurations which achieved the best performance on the training data.
Chunk sizes: English and Spanish texts are chunked into 200 words, while Greek texts are chunked into 100 words. In both cases, any leftover in a document is discarded. Unmasking iterations: The unmasking procedures in Section 2.1 are repeated N times in order to obtain the degradation curve. For English texts, N = 20, while for both Greek and Spanish texts, N = 10.
Classifiers: For the weak classifier used in unmasking degradation curves, we use the
linear-kernel LibSVM classifier, implemented by Weka 3.7.7 [
We use only the training corpus released by PAN 2013 Authorship Identification task (10 English cases, 20 Greek cases, and 5 Spanish cases) and no other complementary materials. A separate model is built for each language.
In training, for a particular language, we use all known documents written in this language in the training corpus to unmask same and different degradation curves. Since there are typically many more different degradation curves than same ones, we sampled at most 500 same curves and 1000 different curves.
For evaluation, we unmask each unknown document in the corpus, against the given known documents of the same case, to get a degradation curve corresponding to this unmasking. We then use the trained meta-classifier to classify the resulting degradation curve as same or different, and thus determine the final answer.
We produce a yes/no answer for all cases, and obtained an overall accuracy of 65.7% for all 35 cases in the corpus. Specifically, for each language, we obtained 100.0% for both English and Spanish, while only 40% for Greek texts. The final evaluation result of our system is an overall accuracy of 68.2%, with roughly the same performance across three languages.
Because the unmasking features in our authorship verification system are designed specifically for English texts, by using well-established stylometric features and our novel coherence features, we expect that by better understanding a particular language, especially those with fewer available NLP tools, more effective unmasking features can be designed to achieve competitive performance on non-English languages as well.