This article describes our proposal for the Author Identification task in the PAN CLEF Challenge 2014. We have adopted a machine learning approach based on several representations of the texts and on optimized decision trees which have as entry various attributes and which are learned for every training corpus separately for this classification task. Our method ranked us at the 2nd place with an overall AUC of 70.7%, and C@1 of 68.4% and, between the 1st and the 6th place on the six corpora.
The task Author Identification (AI) in the CLEF-PAN Challenge is to solve a large set of problems like : given a set A of sample texts, all texts in A are written by a single author and an unidentified document u, determine if u was written by the author of A. The difficulties of this task are various : the limited data : sometimes, A has only one text, some languages that we do not know or we are not able to understand. We adopt a machine learning approach based on several representations of the texts and on optimized decision trees which are based on various attributes and which are learned for every training corpus separately. We decided to represent the documents in different vector spaces and by various types of features : – length of the sentences, – variety of vocabulary, – n-characters grams, n-words gram, – punctuation marks.
For each feature, we considered two numerical values : a mean and a counter. Another global counter was also used. Because we are not able to indicate or to justify the features which are the most important for a given type of document, we used decision trees based on an adapted version of CART, to learn a decision model suited for a kind of document. Thus, each corpus defined by a language and a genre, has its own learned tree.
So, our proposal is based on: – the proposition of vector space models and attributes that represent the documents in a way as optimal as possible. – the formulation of the Author Verification problem as a supervised classification problem. – the evaluation of this approach on different groups of problems in the challenge context.
Section 2 describes the vector spaces that we choose to represent the documents. The Section 3 is dedicated to the methodological approach. Finally, Section 4 presents the experiments and the results obtained on the training set and for the challenge. We will finish with some conclusions and future perspectives. 2
A problem inside a corpus consists in a given set A of documents written by the same author and another document u whose author is unknown. The aim is to decide whether u has the same author as all documents di in A. 2.1
In order to represent the textual documents as vectors we use different vector space
models [
jfd 2 A : tj 2 dgj
This representation can be defined for terms corresponding either to words or characters. In order to take into account the variety of the style and vocabulary, we consider representations based on the punctuation, length of phrases and diversity of the vocabulary as detailed in Table 1. R1 Character 8-grams tf-idf cosine similarity R2 Character 3-grams tf-idf correlation coefficient R3 Word 2-grams tf-idf correlation coefficient R4 Word 1-gram tf-idf without the 30% most frequent words correlation coefficient R5 Word 1-gram tf-idf without stop words correlation coefficient R6 Phrases word per sentence mean and standard deviation correlation coefficient R7 Vocabulary diversity total number of different terms divided by the euclidean distance total number of occurrences of words R8 Punctuation average of punctuation marks per sentence cosine similarity characters: "," ";" ":" "(" ")" "!" "?"
We note that this table contains usual representation spaces and some original ones as well as different comparison methods that, based on an empirical search, appeared to be relevant for this task. 2.2
Our approach requires to compare all documents inside a corpus using the cosine similarity, euclidean distance or the correlation coefficient. These measures are normalized, between 0 and 1 for the euclidean distance and cosine similarity and, between -1 and 1 for the correlation coefficient. For two documents represented as vectors di and dj , the cosine similarity cos(di; dj ) is defined as follows:
The cosine similarity equals to 1 when the documents have the same representation. Conversely, if two documents are highly different, cosine similarity will tend to be 0.
The correlation coefficient corrcoef (di; dj )[
Cij pCii
Cjj (2) (3) where Cij denotes the covariance between the documents di and dj . Given a corpus P containing all the documents having the same language and the same type, we have p 2 P problems to solve and, for each problem there are one or several documents written by the same author and one document (u) whose author is unknown. Thus, the dataset of the supervised learning problem contains all the unknown documents of one corpus, described by 17 attributes but also by the class which has two modalities (SameAuthor or DifferentAuthor). Note that the known documents are not directly taken into account in this dataset however they are used to compute the representation of unknown documents. In supervised learning, models are learned by splitting the dataset into two subsets. The first one, called learning set, is used to learn the model, in our case, a decision tree. The second subset, called test set, is used to evaluate the model. The decision tree learned during the learning step is use to define the class of each unknown document corresponding to a problem. The evaluation of the quality of the decision rules is done by computing the well classification rate or the area under the ROC curve (AUC) obtained by comparing the predicted class and the true class for the unknown documents belonging to the test set. The accuracy of the models depends largely on the attributes predictive power. That leads us to define two attributes per representation space and a global attribute. 3.1
We use a dissimilarity counter method that we designed while experimenting on the
PAN2013 corpora in Author identification and which yielded very good results [
Given P, the set of problems provided for one corpus defined by Ap the set of documents written by one author and up the unknown document for a problem p; p = 1; :::; jPj, such as:
P = f(Ap; up); p 2 1; :::; jPjg
For each document up, corresponding to a given problem, and for each representation space Rv, v 2 f1; ::; 8g, we calculate two attributes countv(up) and meanv(up) as follows: countv(up) = jfdi 2 Ap=minfs(di; dj ); dj 2 A dig > s(di; up)gj (4) (5) (6) (7) meanv(up) =
1 jApj
X di2Ap s(di; up) countv(up) gives the number of documents di 2 Ap for which the similarity between di and up is lower than the minimum of the similarities of di with the other documents dj 2 Ap di. It comes intuitively and indicates how many times up is the most dissimilar to every document in Ap.
meanv(up) represents the average of the similarities between the documents in Ap and up.
These two attributes are computed for each representation space. Consequently, since v 2 f1; ::; 8g we have 16 attributes. A last attribute, T OTcount(up) is built to have a more global representation:
T OTcount(up) =
8 X countv(up) v=1
Finally we have 17 attributes describing each unknown document belonging to a problem provided for one corpus comprised by the documents with the same language and genre. 3.2
For the task of Author Verification, we used the Classification and Regression Trees
(CART) algorithm which constructs binary trees using the features and thresholds that
yield the largest information gain at each node [
For each problem of the corpus, the decision tree has the following informations for the unknown document: – countv(up); 8v 2 f1; ::; 8g – meanv(up); 8v 2 f1; ::; 8g – T OTcount(up) – class(up), the true label of a problem
The previous data allow us to build rules where we classify 100% of problems correctly. In order to handle overfitting we remove all leaves with less than 5% of the total number of problems so we can keep more general rules. Moreover, we choose not to answer problems that have a low probability to belong to one class. The rule we set is that when the probability for a text to be written by the same author is between 0.4 and 0.6, we change the probability to 0.5 so that we choose to not answer this problem. So finally there are 3 modalities for the class: sameAuthor, differentAuthor or undefined. 4
For the learning step, the implementation has been done in Python. We used scikit-learn library 3 for the n-grams representation and for CART. 4.1
The experimentation has been made on the training corpus which contains 696 problems labelled as DE, DR, GR, EN, EE or SP where D stands for Dutch (DE, DR), GR for Greek, SP for Spanish and E for English (EE, EN). We have essays and review for Dutch (DE, DR) and essays and novels for English (EE, EN). For experimentation, we have made a 10-fold cross validation for each group of problems in order to evaluate the quality of the decision trees on the training set.
The table 2 shows for each corpus: the number of problems and the result calculated with the area under the ROC curve (AUC) on the training dataset. 3 http://scikit-learn.org
The following tree is the one used over the English Essays (EE) corpus where "samples" is the number of problems remaining at a node. There are, in total, 200 problems to be classified.
X[
With an overall score of 0.707 for AUC and 0.684 for C@1 we obtained a final score of 0.484 (AU C C@1) which is the second best submission. As shown in Table 3, we obtained the 1st rank for the English Essays corpus, 2nd for the Dutch Essays corpus and 4th for the Spanish corpus. For the evaluation corpora of PAN2014, the results we obtained were consistent with the ones we had while training our decision tree. However we lost significant accuracy for the English novels corpus (near 30% of loss). We would need to study the evaluation corpus to understand why we had such a loss of accuracy. Moreover our approach is not time-consuming as shown in Table 3.
During this challenge we saw that the most difficult task was to gather features that complement each other. CART enable us to identify good predictive features. However, we did not try all possibilities for text representations. Lastly we found that building efficient attributes, like with the counter method, greatly improved the accuracy of CART for some corpora.