In the PAN 2013 Author Identification task, the problem was to verify whether a document was written by the same author as a small set of given reference documents. We approached the problem as a classification task for which the reference documents are from the target class. We further collected a set of documents for the non-target or outlier class. For this classification problem we prepared three submissions for the English Authorship Verification task. The first submission applies the nearest neighbor rule using compression distances from the "questioned" document to the reference and outlier documents. The second and third submission utilize a document representation with compression distances to random prototype documents. In the resulting prototype space the Lowest Error in Sparse Subspace (LESS) classifier is applied. The third submission additionally uses document resampling or bootstrapping to mitigate the small sample problem in case the number of reference documents is low. The evaluation result of our submission achieved the best performance of 16 teams with precision = 0:80 , recall = 0:80 and F1 = 0:80.
In the PAN 2013 Author Identification task, the problem was to verify whether a unknown document was written by the same author as a small set of given reference documents. The given reference documents were parts of books with approximately 1000 words per document. The number of reference documents varied between 1 and 10. We followed a statistical pattern recognition approach. There are two possible learning paradigms to apply. The unsupervised way is to establish whether or not the unknown document could be from the same distribution as the reference documents by using the reference documents only. In a supervised approach, a learned classifier establishes a boundary between the reference documents and documents written by other authors using labeled examples of both groups.
In either approach a suitable vector representation of the documents is required. Representations of text documents are typically high dimensional, while the number of reference documents is low. Especially is such small sample cases, an unsupervised approach will expectedly lead to mediocre performing recognition models. On the other hand, to have a representative sampling of documents written by all possible authors, as is required for the supervised approach, is nearly impossible. Fortunately, the given reference documents all had similar genre, theme, and date of writing and the same would hold for the final test cases according to the contest descriptions. As a result, we would only need a representative sample of documents for the given language, genre, theme, and date of writing. Therefore, we followed the supervised learning paradigm. Below we elaborate on the different aspects of this learning task, being: selection of the labeled dataset, the representation of the documents, and the model learning.
Because of time limitations, we only participated in the English Authorship Verification tasks. 2
We considered the verification problem as a two-class pattern recognition problem. The reference author is the first target class and all other authors are in the second outlier class. From the reference author, we have a limited number of given documents: 1-10. From the outlier class, there are no examples given. Therefore, we needed to collect similar documents as given in the training set to obtain information about the writing style of other authors. It was stated in the task description that in the final test, the documents would have the same language and similar genre, theme, and date of writing as the given training samples. Accordingly, we collected documents with these properties.
In order to obtain similar English documents as the ones provided by the PAN 2013 organization, we searched the Internet for substrings of the provided documents. We found several of the training documents in he bookboon.com repository, which provides open access for students to text books. From this repository we selected 66 textbooks from the Engineering (Chemical Engineering, Construction Engineering, Electrical Engineering, Energy Engineering, Environmental Engineering, Mechanical Engineering, Nanotechnology and Petroleum Engineering) and IT & Programming sections. The books were authored by 46 different authors.
We prepared the documents similarly as the training/reference material. That is, we converted the text sections to UTF8 format and split the books into documents of 6,000 to 8,000 characters each. This resulted in 2 to 75 documents per text book with roughly 1,000 words each. 3
As document representation, we chose the compression distance to other documents. Several compression distance measures have been proposed in the past [6]. We used the Compression Dissimilarity Measure (CDM) [3]:
CDM (x; y) =
C(xy)
C(x) + C(y)
;
(
We submitted three approaches to the English Authorship Verification task, with an increasing modeling complexity. All approaches used the collected documents to represent the outlier class and compression distances as document representation. 4.1
Nearest Neighbor with Compression Distances
The simplest approach we prepared is purely based on compression distances between
the unknown document on the one hand and the reference documents and collected
documents on the other hand, see Figure 1. The decision rule is as follows: if the closest
document in the dataset is from the reference author, we conclude that the unknown
document is from that author too. Otherwise, if a document from any author from our
collected corpus is closer, we conclude that the unknown document is not written by
the reference author. Such a 1-nearest neighbor approach is known to be sensitive for
overfitting or outliers in the data. On the other hand, the length of documents probably
mitigates this problem to a certain extend. That is, for longer documents it is less
probable that accidentally a document of another author is more similar than that of the true
author. On the provided cases, this approach resulted in 1 error out of 10 (10%).
For the second approach, we learned a two-class classifier in prototype space as in
[4], see Figure 2. The first (target) class is the class of the documents of the reference
author and the second (outlier) class is that with all collected documents (by other
authors). All documents are represented by their compression distance toward a number
of selected documents from the training set, i.e. the prototypes. To be able to compute
as much of the compression distances beforehand, the prototypes are selected from
the collected documents, i.e. the outlier class. Since this class is diverse, the distances
to these documents will give a rich representation. Because of the limited amount of
available documents, the prototypes themselves were not removed from the dataset for
model learning. Per test case, we learned a classifier with the reference and collected
documents and applied the resulting classifier to the unknown document. The classifier
decides if the document is from the reference author or not. As classifier, we used the
Lowest Error in a Sparse Subspace (LESS) classifier [8] that is able to deal with small
sample size problems, see Eq. 2.
(
i=1 subject to: where and ( x 2 Xt; x 2 Xo; (x; j) = xj wj 0; i 0: Pp
j=1 wj (x; j) Pp j=1 wj (x; j) <
2 tj xj 1 oj
ti 1 + oi 2
Here, w is the model vector with weights wj per dimension, Xt and Xo contain the target and outlier samples, nt and no are the sizes of Xt and Xo, t and o are the mean vectors of Xt and Xo, p is the number of dimensions (in this case prototypes), and ti and oi are the slack variables for documents from the respective classes to enable the modeling of inseparable classes. Further, C is a tunable parameter that balances model sparseness against model accuracy.
Our collected dataset is a lot bigger than the provided reference documents. Without precautions, the scarce target class would be neglected by the model, just because it is scarce. From the training cases it turned out that roughly in 50% of the cases the unknown document is from the reference author (target). Therefore, we made both classes equally important in the LESS model in order to deal with these strongly unbalanced classes, see Eq. 3.
p min X wj + C j=1 1 Xnt ti + nt i=1 1 Xno oi ; no i=1 subject to: ( x 2 Xt; x 2 Xo; where and
(x; j) = xj wj 0; i 0: Pp
j=1 wj (x; j) Pp j=1 wj (x; j) <
2 tj xj 1 oj
ti 1 + oi 2
The number of prototypes and the C trade-off parameter for LESS, we optimized
on the collected corpus. In turn, we selected one of the 46 authors as reference author
and took up to 10 of his documents as reference documents. We put a selection of
documents from the that author and from other authors aside for testing. In this way, we
established p = 200 and C = 10; 000.
In the last approach, we attempt to make the method more robust for the low number of
document samples in the reference class. Ultimately, there could be only one reference
document. Therefore, we resampled the available reference documents as we did in
[2], see Figure 3. To make the method generic, we first merged the available reference
documents (
We ran several experiments with different bootstrapped document samples and different prototypes on the provided cases. This resulted in 16% average error on the 10 cases. 5
We prepared three submissions for the PAN 2013 Authorship Verification task. An important part of our approach, was careful selection of documents not authored by the reference author, the outlier class. All our submissions used compression distances as document representation. Two of our submissions additionally used a representation with compression distances to prototypes, that were selected as a subset of the outlier class. The last submission used document resampling to increase the size of the target class. On the provided 10 test cases the submissions had similar performance: 1-2 errors. With only 10 test cases significance of differences in performance could not be established. With these submissions we obtained the best (ex aequo) score F1 = 0:80 out of the 16 teams, that submitted for the English Authorship Verification task.