Previous PAN workshops have offered us the opportunity to explore three different approaches using basic statistics of stopword pairs for author verification. In this PAN, we were able to select our 'best' approach and explore the question of how authors writing about different subjects would necessarily adapt to term lengths specific to the subject. The adaptation required is, essentially, a redistribution of frequency: where longer terms occur. We introduce the notion of a 'topic cost' which increases the propensity for matching. Results show AUC and C1 scores of 0.51, 0.46 and 0.59 for Dutch, Greek and Spanish respectively. The English results are not yet available, as the evaluation system was unable to run the approach due to as yet unknown reasons.
In the 6th International Workshop on Uncovering Plagiarism, Authorship,
In this paper, we suggest extension to our
As discussed in [1], for P
For PAN2013, the core approach remained the same with output adapted to the Boolean output required. The task introduced Greek and Spanish texts, of which the authors have no real knowledge, and so lists of 10 frequent stopwords were sought for each.
For PAN2014, we reused these stoplists along with a stoplist for Dutch – with Dutch as yet another language of which the authors have no real knowledge. We also evaluated 3 approaches based on:
Frequency-Mean-Variance: We follow the approach detailed at length in Vartapetiance and Gillam 2013, generating frequency information for stopword pairs, determining mean and variance for separation, then applying cosine distance to compare the resulting feature vectors.
Positioning: This approach is based on FMV, above, but omits step 4 and so acts as a cosine comparison on positional frequencies for each pattern. This would tend to require comparable frequencies for each feature to ensure a good match.
Cosine: We modify the Positioning approach to consider the frequency information for all patterns as a single vector, then apply cosine distances between resulting vectors. Here we also consider how to determine a match: a single cosine distance between one known and one unknown; a difference in distance within a threshold when two known texts can be compared; and distances between the unknown and many known texts to be at a suitable point on the distribution of distances amongst knowns. Acceptability, according to thresholds, and cosine distance can then be used together to determine match confidence. 3
PAN 2015
For this year’s task, we wanted to explore the ability to match where the same author may necessarily vary their writing according to the topic. This would account for, say, simple temporal modification– discussing for example ‘the former Prime Minister of’ rather than ‘the Prime Minister of’ – but is principally geared to account for differences in term lengths as relate to topics. In the ‘Prime Minister’ example given, the same stopword pair of the-of is present, but with a positional mismatch. Since position, and variability in position, is core to our approaches, we require a simple way to address the pattern-specific positional mis-alignment that occurs.
To approach this, we introduce the notion of a ‘topic cost’ and distribute positional frequencies according to this topic cost. To determine topic cost, we simply count the number of terms and the length of these terms, and use the difference between these values for redistribution. The only additional resource employed is a languagespecific stoplist as exposes the terms.
As an example, consider the following passage of text:
UK interest rates have been kept unchanged again by the Bank of England, meaning they have now been at their record low of 0.5% for six years. Rates were first cut to 0.5% in March 2009 as the Bank sought to lift economic growth amid the credit crunch.
Take stopword pairs as formed from [the, of, in, for, to]. If we ignore the sentence break, the first pair of interest offers us: “for six years. Rates were first cut to”. The distance covered by the pair is 6 (the number of words between “for” and “to”). Collecting all multi-word terms, using all stopwords (not just those listed) as delimiters (and, here, the full-stop also), results in 3 terms comprising 5 words – six years, rates, first cut. The topic cost, then, is 2. Instead of counting once at position 6, we uniformly distribute – other weightings possible but unexplored - across position 6 and the two preceding positions and so positions 4, 5 and 6 each receive 0.333. This example, and further from the above passage, are shown in the table below.
Results for each of the PAN 2015 collections are shown in the table below based on 4 language categories.
Due to yet unknown problem with English run, the system was unable to calculate the outcomes of the test. Also, unfortunately, the results from the runs using last year’s systems will not be available until after this paper is submitted, so the authors are not able to provide a comparison between systems to see whether or not this approach improves the outcome of detection. However, the results on runs on training datasets using FMV, Positioning and Topic Cost systems (Table 3) show some improvements in detection using the new system.
In this paper, we suggested an extension to our
The authors gratefully acknowledge prior funding from the UK’s Technology Strategy Board (TSB, 169201), and also the efforts of the PAN organizers in crafting and managing the tasks.