This paper presents our approach for the Author Identification task in the PAN CLEF Challenge 2015. We identified the challenges of this year's are the limited amount of training data and the problems in the sub-corpora are independent in terms of topic and genre. We adopted a machine learning based intrinsic method to verify whether a pair of documents have been written by same or different authors. Several content-independent features, such as function words and stylometric features, were used to capture the difference between documents. Evaluation results on the test corpora show our approach works best on the Spanish data set with 0.7238 and 0.67 for the AUC and C@1 scores respectively.
Given a pair of documents (X,Y), the task of author verification is to identify whether
the documents have been written by same or different authors. Compared to authorship
attribution, the authorship verification task is significantly more difficult. Verification
does not learn about the specific character of each author, but rather about the
differences between a pair of documents. The problem is complicated by the fact that an
author may consciously or unconsciously vary his/her writing style from text to text [
This year’s PAN lab Author Identification task focuses on cross-genre and crosstopic authorship verification. In this case, the genre and/or topic may differ significantly between the known and unknown documents. This task is more representative of real world applications where we could not control the genre/topic of the documents.
The PAN Author idenfitication task is defined as follows: “Given a small set (no
more than 5, possibly as few as one) of known documents by a single person and a
questioned document, the task is to determine whether the questioned document was
written by the same person who wrote the known document set. The genre and/or topic
may differ significantly between the known and unknown docu-ments” [
The data set consists of author verification problems in four different languages. In each
problem, there are some known documents written by single person and only one
unknown document. The genre and/or topic between documents may differ significantly.
The document length varies from a few hundred to a few thousand words. Table 1 shows
the sub-corpora including their language and type (cross-genre or cross-topic)
The author verification systems are tested on a set of problems. The system must provide
a probability score for each unknown document. The performance of the system will
be evaluated using area under the ROC curve (AUC). In addition, the output will also
be measured based on c@1 score [
The overall performance will be evaluated on the product of AUC and c@1 2
We adopted a machine learning-based intrinsic method to address this verification
problem. Intrinsic methods use only the provided documents (in this case known and
unknown documents) to determine whether they are written by same author or not. A
machine learning algorithm then will be trained on the labeled document pairs to construct
a model which can be used to classify the unlabeled pairs. Note that in the
verification problems, the machine learning does not learn about the specific character of each
author, but rather about the differences between a pair of documents [
As the genre and/or topic may differ significantly between the known and unknown documents, we can not rely on the content based features to capture the differences between documents. We therefore focused more on content-independent features such as function words and stylometric features. In addition, those features can be applied to any of the language used in the task. We used six types of features in total including: stylometric features (10), function words, character 8-grams, character 3-grams, word bigrams, and word unigrams.
Given collection of problems P = fPi : 8i 2 Ig where I = f1; 2; 3; ::; ng is
the index of P . Pi contains exactly one unknown document U and a set of known
documents K = fKj : 8j 2 J g where J is the index of K and 1 J 5. Our
approach represented each problem Pi as vector Pi = fR1; R2; ::; Rng where n is the
maximum number of feature types (in our case are six). Ri is the distance of two similar
feature vector representation of a set of known documents K and unknown document U .
If K contains more than one document, then the generated feature vector is an average
vector of J documents. Table 2 shows details of the features vector representation and
comparison measures used.
Stylometric Features There are ten sytlometric features used in our experiment. Some
features were adapted from Guthrie’s work [
We also implemented three readibility measures:
ReadingEase = 206:835
GradeLevel = 11:8
3. Gunning-Fog Index [
total_words total_sentences 84:6 total_syllables
total_words total_syllables total_words We experimented with several different comparison measures for computing similarity between a pair of vectors. We noticed particular comparison metric performs better in certain type of features, thus we applied different measure for each features type. Three different distance measures were used:
d(x; y) = x:y jjxjjjjyjj = p P xiyi i=1 s p
P xi2 i=1 s p
P yi2 i=1
City block distance (also called Manhattan distance or L1 distance) p
P min(xi; yi) minmax(x; y) = i=1 p P max(xi; yi) i=1 p d(x; y) = X jxi yij (2)
100 (4) (5) (6) (7) 2.3
Our authorship identification software was written in Python. We applied feature selection using Extratreeclassifier and the SVM classifier. The classifier hyperparameters were optimized using the GridSearchCV. Scikit-learn library2 was used for both feature selection and classification. 3 3.1
We evaluated the approach on the training data using 10-fold cross validation. Since there are some incompatibility issues, we did not perform the verification task on Greek data. Table 3 shows the result of our approach on three of the sub-language corpora. The best result was achieved on the Spanish data set with 0.846 and 0.807 for the AUC and C@1 scores respectively. Compared to other sub-language corpora, the Spanish data set contains more known documents; which may explain why the results on this data set are better than the results on the other sub-languages data. We also observed that the use of some NLP libraries which are mainly trained on English data did not perform well on the non-English language data sets. Thus, feature selection was applied to remove unhelpful features.
4
This year’s author verification problem is considerably harder than last year’s since the number of known documents is very limited and the genre/topic between known and unknown documents differ significantly. In addition, for English, the data set was derived from Project Gutenberg’s opera play scripts which are an unusual type of text.
We identified that the most challenging part of this task was to find suitable features which could capture the differences between documents. In addition, for certain data set, not all features were helpful. Thus applying feature selection were beneficial and greatly improved the accuracy of the classifier.