Encouraged by results from our approaches in previous PAN workshops, this paper explores three different approaches using stopword cooccurrence. High frequency patterns of co-occurrence can be used to some extent as identifiers of an author's style, and have been demonstrated to operate similarly across certain languages - without requiring deeper linguistic knowledge. However, making best use of such information remains unresolved. We compare results from applying three approaches overs such patterns: a frequency-mean-variance framework; a positional-frequency cosine comparison approach, and a cosine distance-based approach. A clearly advantageous approach across all languages and genres is yet to emerge.
A Brief Contrast to Previous PANs
PAN2014 focuses Authorship Attribution to a question of whether the author of a set of documents is also the author of a given document outside this set. This task covers six text collections across four different languages: English, Greek, Spanish and Dutch, and covering four genres: Essays, Reviews, Novels and Articles.
For PAN2012 [3] , given a set of documents from different known authors and a set of documents with unknown authors; the task was to allocate the documents to one author (or none). The PAN2013 [4] approach required a Boolean response as to whether an unknown document was likely written by the same author as a set of (from 1 to 10) “known” documents from that (single) author. PAN2014 now allows for three responses – introducing a “non-committal” value (0.5); better performing systems may be hedging responses rather than committing to a wrong answer. The size of the training corpus has changed substantially from 35 across three categories for PAN2013 to 696 across 6 categories for PAN2014.
Table 1, below, shows details of the six training subcorpora for PAN2014, covering numbers of cases, and averages numbers of known and unknown documents as well as statistics for the known documents. Effects of document length and number of comparisons may be useful in subsequent analysis.
For PAN2012, we approached attribution using a mean-variance framework on patterns of stopwords using a specified maximum window size for pairs of the 10 most common English stopwords to identify positional frequencies, and allocated an author based on nearest frequency-mean-variance match. We achieved F1 of 0.42, and saw post-submission that it might have been possible to achieve F1 of 0.48 using paired sets of 5 stopwords (i.e. patterns combined from the first 5 stopwords with the second 5, hence a smaller feature space) [1]. For PAN2013, the core approach remained the same with output adapted to the boolean output required. The task introduced Greek and Spanish texts, of which the authors have no real knowledge, and so lists of 10 frequent stopwords were sought for each.
For PAN2014, we reuse these stoplists and have now added Dutch stopwords to address Dutch subcorpora – with Dutch as yet another language of which the authors have no real knowledge.
We follow the approach detailed at length in
This approach is based on FMV, above, but omits step 4 and so acts as a cosine comparison on positional frequencies for each pattern. This would tend to require comparable frequencies for each feature to ensure a good match.
We modify the Positioning approach to consider the frequency information for all patterns as a single vector, then apply cosine distances between resulting vectors. Here we also consider how to determine a match: a single cosine distance between one known and one unknown; a difference in distance within a threshold when two known texts can be compared; and distances between the unknown and many known texts to be at a suitable point on the distribution of distances amongst knowns. Acceptability, according to thresholds, and cosine distance can then be used together to determine match confidence. 4
Submissions, Results and Evaluations
In PAN2013, we determined a set of parameters, values for which would embody language-specific treatment. In PAN2014, introduction of different text genres required additional considerations relating to the likelihood of pattern occurrence in much shorter documents. To account for such differences, we conducted a parameter sweep (over 10000 tests) based on values shown in Table below: S1, S2 and S represent first 5, second 5 and all ten most frequent stopwords respectively.
Dutch
Spanish
Essay Review Essay Novel Article Article 5 10 5 5 5 20 Essay Review Essay Novel Article Article 5 5 5 5 5 5 0 0 0 0 0 0
A similar strategy leads us to the values shown in Table 5 for the Positioning and Cosine approaches.
Table 6 shows results from the 3 approaches on 3 datasets: training, corpus-1 and corpus-2 – for all 6 categories. The best overall result is still obtained for FMV, although comparison between the values showed that the Cosine approach achieves much higher results for English Novels where the unknown documents was only being compared to 1 known document, while FMV approach had higher score for categories in which there were more known documents; e.g. Spanish with 5 known documents per test. Training Corpus-1 Corpus-2
FMV - competition Positioning Cosine FMV - competition Positioning Cosine FMV - competition Positioning
Cosine 5
In this paper, we attempted to reuse and adapt a fairly simple approach from PAN2013 for Authorship Attribution. Our frequency-mean-variance framework demonstrates reasonable performance (0.63) on training data, and similar (0.57) on test data. Our positioning approach is less performative (0.59 and 0.53), and cosine approach sits between these two (0.60 and 0.55). These results suggest that a broader grain in comparison achieves a marginally better result than a positional focus might offer, which indicates that the direction of future exploration needs to account for this broader grain.
The authors gratefully acknowledge prior from EPSRC/JISC (EP/I034408/1), the UK’s Technology Strategy Board (TSB, 169201), and also the efforts of the PAN organizers in crafting and managing the tasks.