The authorship verification task requires deciding whether a given test document was written by the same author as a training set. For my attempt I tested a simple voting ensemble of local (character) n-gram methods, using a grid search to choose parameters. This results in a method that requires little preconfiguration and can be applied to any language with a concept of characters. The method itself is quite fast, however training is slow with the large number of attempted parameter combinations. The approach results in accuracies of around 60% depending on the corpus and application.
Local n-gram (LNG) methods employ a profile based character n-gram approach to authorship analysis. An author’s profile consists of the top L n-grams for a given author, different to many feature selection methods which usually use a globally relevant set of features. The author profile is then compared to a similarly created document profile using a given distance metric. For authorship attribution, a classification task, each candidate author is first profiled. A test document of unknown authorship is then profiled and the distance to each candidate author profile is computed. The author with the lowest distance is predicted as the author.
Applying LNG to authorship verification makes use of the concept of distance, but applies in a different way. First we formally define the problem.
We are given a training set of documents D (usually such that 1 <= jDj <= 5) all authored by the same person A. Next, we are given a test document dt. The task is to determine whether the author of dt is A, i.e. the author of the documents in D. We refer to the above task as a single trial, with the authorship verification task composing of a large number of trials in different languages and different contexts.
We employed a straight-forward translation of the use of LNG for classification purposes to authorship verification purposes. We calculated the distance between the test document dt and each of the documents in the training set D, called the inter-distance, and compared that to the internal distance between documents in D to themselves, the intra-distance. The assumption was that if the inter-distance was approximately equal to the intra-distance, then the document was likely to be from the same author. If they were not approximately equal, then it is more likely a different author wrote dt. 1.1 Our results were quite poor for this year’s competition. The focus on automation of the algorithm may have hurt performance, due to a lack of tuning. It would be recommended to use a more fine-tuned approach, rather than a generic ‘catch-all’ approach. 2
There were six datasets in the training corpus, and a total of 596 individual trials. There
were four languages represented in the released datasets; Dutch, English, Greek and
Spanish. Compositions of the datasets are provided in table 1, and all datasets had
approximately equal number of positive and negative trials.
In recent years, the authorship analysis field has used machine learning techniques for
a majority of research [
Advances in local algorithms have shown great success in this alternate form of
model. A document, or set of documents, is represented as a profile P such that P (x)
is the value of feature x for the document, or set of documents. The features chosen
are usually character n-grams, subsequences of continuous characters or length n. The
value of P (x) is then given as the frequency of n-gram x in the document, or set of
documents, being profiled. When local models are used with character n-grams, the
approach is called Local n-grams (LNG). What makes a local model particularly different
from a global model is that there is no global set of features. By profiling the set of all
documents known to be from one author, we can profile that author’s writings. While
most applications of LNG have been supervised, LNG methods have also been used for
unsupervised methodologies, outperforming feature based models [
Instead, each document (or set) is profiled using the set of features most distinctive to that document (or set). This means each profile has a separate set of features associated with it. The phrase most distinctive usually means ‘most frequent’, however the RLP algorithm introduced later has a different definition.
As notation for the following, we state that a feature x is ‘in’ a profile P if P (x) 6= 0. The intersection of two profiles is the set of features that are in both, and the union is the set of features in either (ignoring values).
The first model of this type was the Common n-grams (CNG) method [
P2(x)) !2
P1(x) + P2(x) K(P1; P2) =
x2XP1 [XP2
A variant of this form, the Source Code Author Profile (SCAP) algorithm, was
introduced by [
Stamatatos’ d1 and d2 measures are improvements designed to work with
unbalanced datasets [
The Recentred Local Profiles (RLP) algorithm was developed by [
R(P1; P2) = 1
P1 P2 jjP1jj2jjP2jj2
LNG methods have shown a high accuracy in difficult domains [
For this application, we used the CNG, SCAP and RLP methods as base level classifications. These methods were chosen as a representative sample of LNG methods, not specifically due to the relative efficacy, as other LNG methods achieve similar accuracies. The n values chosen for the task were 3 to 5 inclusive, while L values of 1000, 2000, 5000 and 10000. This set of parameters was chosen to be small enough to compute in a reasonable time, while still being relatively representative of feature values proven effective in other studies.
For each of the based methods (CNG, SCAP and RLP), a grid search of parameters was conducted to find the most accurate. The grid search compared all combinations of n and L values 5
We used a distance based threshold for determining whether the given document belonged to a specific author. This threshold was relative to the documents themselves, not a global value.
We calculated the distance between the test document dt and each of the documents in the training set D, called the inter-distance, and compared that to the internal distance between documents in D to themselves, the intra-distance. The assumption was that if the inter-distance was approximately equal to the intra-distance, then the document was likely to be from the same author. If they were not approximately equal, then it is more likely a different author wrote dt.
Because there was only one test document, we could not apply standard statistical distribution comparisons, and instead opted for simpler approach. The inter-distance was considered to be approximately equal to the intra distance if it was less than the average intra-distance, plus two standard deviations (of the intra-distance for a given dataset D).
The obtained results were less than expected, and less than expected for the individual based models. This suggests that this threshold method may require extensive work, and perhaps an alternate strategy. 6
While there was some variability to the results and rank (with a top rank of 4th on one corpus), the results were typically quite poor. The baseline for the results is approximately 0.5, which was beaten in all datasets, but only barely in most. The performance was also less than expected, based on cross-validation results within the dataset. The reason for this is likely the small amount of data was not properly accounted for in the cross-validation model, meaning that the final model overfit the data. The early investigations into these errors suggests that this is caused by the threshold based method, and not the baseline LNG methods. There were some errors that can be traced to the baseline LNG methods though, suggesting that further testing is necessary for this form of application.