We describe the submission of the So a University team for the Author Identi cation Task, part of the PAN 2015 Challenge. Given a small set of documents by a single person and a \questioned" document, possibly of a di erent genre and/or topic, the task is to determine whether the questioned document was written by the same person who wrote the known document set. This is a hard but realistic formulation of the task, also known as author veri cation. We experimented with an SVM classi er using variety of features extracted from publicly available resources. Our solution was among the fastest, and running time was an o cial evaluation metric; however, our results were not so strong on AUC and C1.
We describe the submission of the So a University team, registered as kiprov15, to the Author Identi cation task of the PAN 2015, the 13th evaluation lab on uncovering plagiarism, authorship, and social software misuse.
Given a small set of documents (no more than ve, possibly as few as one) by a single person and a \questioned" document, the task is to determine whether the questioned document was written by the same person who wrote the known document set. In practical terms, the output of the participating systems was expected to be a real number between 0 and 1, which corresponds to the probability of a positive answer. This is a hard but realistic formulation, known as author veri cation, of the more general author identi cation task. Note that unlike previous editions of the task, this year the questioned document di ered from the documents in the known set in terms of genre and/or topic.
The organizers provided training data in four languages (English, Greek, Spanish, and Dutch), and we submitted a system for all four. We experimented with an SVM classi er using variety of features extracted from publicly available resources. Our system was among the fastest-running ones: each testset was fully analysed and scored under two minutes, and time was an o cial evaluation metric; however, we were not so strong on AUC and C1.
We used GATE [
Our GATE processing pipeline includes the following components:
1. Reset PR
2. ANNIE Tokenizer [
Once the annotations were done, we extracted some features, which we then
used in the SVM classi er. While the challenge contained documents in di erent
languages (as well as genres and topics), we aimed to use as few language-speci c
markers as possible; thus, most of our features are token-based. Here is the list
of the features we used:
1. Average sentence length to character count ratio
2. Average sentence length to word count ratio
3. Average word length
4. Average paragraph length to word count ratio [
Previous research has shown that machine learning can be used successfully
to tackle the task of author identi cation; see [
We extracted the above-described features, and as a result, for each document we obtained a feature vector. Given a problem, i.e., a set of known documents and a questioned document, we aggregated these feature vectors for all known documents. Similarly, we built a feature vector for the questioned document (but this time there was nothing to aggregate as it is only one). Finally, we produced a 10-dimensional vector for the (known set, questioned document) pair as follows: for the rst seven features (i.e., excluding the n-grams), we just subtracted them, and for the n-gram features, we calculated separately the cosine similarity for the 1-grams, the 2-grams and the 3-grams, and we used the values as eighth, ninth and tenth features. Then, we scaled the real values to the [0;1] range, and we saved the scaling factors. We further added a class label: same or di erent (author). On testing, we produced the 10-dimensional vectors in the same way, except that we reused the scaling factors from training. 3 3.1
We used the following datasets for training (we used them one at a time, as we trained and tested separate models for each of the four languages): { pan15-authorship-veri cation-training-dataset-dutch-2015-04-19 { pan15-authorship-veri cation-training-dataset-english-2015-04-19 { pan15-authorship-veri cation-training-dataset-greek-2015-04-19 { pan15-authorship-veri cation-training-dataset-spanish-2015-04-19 3.2
As we mentioned above, we used LibSVM. We chose a C-SVM type of classi er with radial basis function (RBF) kernel. We further set the following parameter values: C = 1, = 0:5, " = 0:001. We used di erent cache sizes depending on the purpose of the executions: testing on our machines or testing on TIRA.
We selected the above SVM parameter values experimentally, following the recipe in the Practical Guide to Support Vector Classi cation 3, which can be found on the LibSVM website. After setting the described kernel and SVM type, we manually tried di erent values for the C and the parameters as prescribed, and we ended up with the above values. Given that the search process was not automated, we might have missed some better parameter values. 3.3
O
cial Results Our o cial results are summarized in Table 1. More detailed results can be found at the corresponding TIRA page:4 look for kiprov15, which is our team's name.
We can see in Table 1 that our highest score is for Greek, where our team was ranked 9th out of 15. A notable characteristic of our solution is its runtime. In all instances, our system was among the fastest-running ones: each testset was fully analysed and scored under two minutes.
However, our results for English, Spanish and Dutch are suprisingly low, which could indicate a bug in the execution of the above-described pipeline. We are planning a detailed investigation in future work. 3 http://www.csie.ntu.edu.tw/ cjlin/papers/guide/guide.pdf 4 http://www.tira.io/task/authorship-veri cation/
A possible explanation for our relatively good results for Greek could be the number of the known documents for each problem. It turns out that for Greek, there are generally less known documents per problem in the training data. This could help as having more known documents might be confusing for the classi er. When these documents are from di erent genres and have di erent text structure, the classi er might pick the wrong characteristics to focus on, i.e., try to model genre/structure instead of author. 4
We have described the submission of the So a University team for the PAN'2015 Author Identi cation Task. We experimented with an SVM classi er using variety of features extracted from publicly available resources. Our solution was among the fastest, but it did not perform very well in terms of AUC and C1.
Note that our solution is con gurable and can be easily expanded and tweaked,
which we plan to explore in future work. For example, it is very easy to generate
new language-speci c features, e.g., by adding new processing resources to the
GATE pipeline. A wide range of these are readily available, but a careful
selection and evaluation might be further required. Such features can include lists
of stopwords, language-speci c resources, character n-grams [
Another idea is to try to arti cially expand the training data by using some of the examples in the known set as questioned examples; with such a tweak, we will provide more training examples to the SVM, which is likely to improve its predictions at testing time. 5
The project source code can be found on BitBucket: https://bitbucket.org/StanimirNikolov/pan-author-identi cation 6
We would like to thank the anonymous reviewers for their constructive comments, which have helped us improve the paper.