We use ensembles of proximity based one-class classifiers for authorship verification task. The one-class classifiers compare, for each document of the known authorship, the dissimilarity between this document and the most dissimilar other document of this authorship to the dissimilarity between this document and the questioned document. As the dissimilarity measure between documents we use Common N-Gram dissimilarity based on character or word n-grams.
We describe our submission to the task of Author Identification of the PAN 2014
competition [
The required output in the competition task is a real number in the range from 0 to 1, encoding the probability of the positive answer to this question. A probability score that is less than 0.5 is interpreted as a negative answer; a probability score that is greater than 0.5 is interpreted as a positive answer; the score of 0.5 is interpreted as the "I don’t know" answer.
The submissions are evaluated using the measure of area under the ROC curve
(AUC) based on the probability scores, and the c@1 measure [
The Author Identification at PAN 2014 is similar to the Author Identification task
at PAN 2013, described in [
Let A = d1, ..., dk , k 2, be a set of "known" documents written by a given author. Let u b{e the questione≥ddocument which authorship we are to verify.
}
Our algorithm is a proximity-based one-class classification algorithm, based on
an idea resembling the idea of the k-centers algorithm [
D(di,u) r(di, u, A) = Dmax(di,A) . We apply a threshold θ on the value of M (u, A) that is the average of the r(di, u, A) over all known documents di, i = 1, ..., k. We classify u as written by the same author as known documents iff M (u, A) <= θ. Specifically, we linearly scale the average dissimilarity ratio M (u, A) using the threshold θ, so that the value of M equal to θ corresponds to the score 0.5, values greater than θ correspond to the scores between 0 and 0.5, and values less than θ correspond to the scores between 0.5 and 1 (a cutoff is applied, i.e. all values of M (u, A) < θ cutoff are mapped to the score 1, and all values of M (u, A) > θ + cutoff are mapped −to the score 0).
For the dissimilarity measure between documents we use the Common N-Gram
(CNG) dissimilarity, proposed by Kešelj et al. [
D(P1, P2) =
X x∈(P1∪P2) fP1 (x) fP2 (x) ! fP1 (x)−+fP2 (x) 2 2 (1) where x is an n-gram from the union of two profiles, and fPi (x) is the normalized frequency of the n-gram x in the the profile Pi, i = 1, 2 (fPi (x) = 0 whenever x does not appear in the profile Pi).
If there is only one known document, we cut it in half to obtain two known documents. We also truncate all documents in a given problem to the length of the shortest one. We also make sure that each profile for a given problem has exactly the same length in cases when the number of distinct n-grams in any of the documents within given problem is less than the requested length of the profiles.
Ensembles comprise of such classifiers that differ between themselves in at least
one of the following parameters: type of the tokens in n-grams (characters or words),
the length of n-grams, the length of profiles. We used ensembles with weighted
voting [
We select classifiers for the ensembles separately for each corpus, based on their
performance on the training datasets. We investigate performance of classifiers, varying their
parameters. The tokens were utf8-encoded characters or turned to uppercase words. For
classifiers based on characters the length of n-grams varied from 3 to 10. For classifiers
based on word n-grams the length of n-grams varied from 1 to 6. The length of profiles
was in 200, 500, 1000, 1500, 2000, 2500, 3000 for both kinds of tokens. This space of
{ }
parameters results in 98 single classifiers: 56 character-based ones and 42 word-based
ones. Package Text::Ngrams [
We select for each training corpora separately a fixed odd number of 31 classifiers that yield the best AUC. Subsequently for each of those classifiers the optimal threshold is found (i.e., the threshold for which the maximum accuracy is achieved). In an ensemble for a given corpus, the threshold for all classifiers is set to one value: the average of the optimal thresholds on the training data for the selected single classifiers.
The ranges of AUC and of maximum accuracy (accuracy at the optimum threshold) for the sets of 31 classifiers are presented in Table 1.
We observe that our method performs best for the training corpus for Dutch essays and Spanish articles. It performs worse on the Greek articles set. For the sets of English novels and Dutch reviews the performance is low. Most likely the reason behind that lies in the fact that in these two sets all but one problem have exactly one known document. We observed that such problems are especially challenging for our method. This is most likely because the two halves of a single known document, that we compare the questioned document with, are much more similar to each other than two different documents written by the same person. The reasons behind the low results on the English essays set are not clear to us and require further investigation. 4
The results of our submission on the PAN 2014 evaluation set for the Author Identification tasks are presented in Table 2. Similarly as in our experiments on the training data, the sets of English novels, English essays and Dutch reviews were most challenging for our method, as discussed in section 3. On the English novel set the ensemble achieved a noticeable lower AUC value then the separate classifiers on the training data. Our method performed better on the sets of Dutch essays, Spanish articles, and Greek articles.
AUC 0.86892 0.6376 0.5179 0.49125 0.7308 0.8026 0.84201 0.54837 0.45727 0.68 0.73 final score 0.73165 0.35706 0.284 0.22464 0.49694 0.5859