In this paper we present the evaluation of three different classifiers (Rocchio, Naïve Bayes and Greedy) with the aim of obtaining a baseline in the task of authorship identification. We decided to employ as features the original words contained in each document of the test set, with a minimum of preprocessing which included elimination of stopwords, punctuation symbols and XML tags. As may be seen in this paper, the obtained results are adequate, reflecting the aim of the experiments. In average, Rocchio slightly outperformed the Naïve Bayes and the Greedy classifier. However, we recommend using both, Rocchio and Naïve Bayes in future evaluations of the PAN competition as baselines from which other teams may compare their own approach.
Authorship identification is the task of determining the real author of a given text. Nowadays, there exist many texts that have been written anonymously or under false names which lead to confusion on the identification of their authorship. The main challenge of authorship identification is to automatically assign a text to one of a set of known candidate authors. The experiments reported in this paper were carried out in the framework of the 5th International Workshop on Uncovering Plagiarism, Authorship, and Social Software Misuse (PAN’11).
Given texts of uncertain authorship and texts from a set of candidate authors, the task of “Authorship identification” of PAN’11 consists of mapping the uncertain texts onto their true authors among the candidates. For this purpose, the organizers of the aforementioned task have developed a new authorship evaluation corpus which is better described in [2].
We have tackled this problem through the use of supervised learning methods. The aim of this paper is to show the performance of three different methods in order to determine whether or not they may be used as baselines in future evaluations.
The rest of this document is structured as follows. Section 2 present the pre-processing techniques applied to the given corpus and a description of the classifiers used. Section 3 presents a discussion of the obtained results. Finally, in Section 4 the conclusions are given.
The aim of our first participation in PAN’11 was to obtain baselines for the task of authorship identification. Therefore, we decided to employ as features the original words contained in each document of the test set, with a minimum of preprocessing which included elimination of stopwords, punctuation symbols and XML tags. Future evaluations will consider appropriate features which may improve the classification task. The description of the datasets used and the classifiers evaluated follows. 2.1
Five training collections consisting of real-world texts (i.e., often short and messy) were given to the task participants. The first one with 26 different authors, the second one with 72 different authors, and the remaining three each with a single author (for author verification). In the first two cases, there were two testing scenarios, represented by a different test set: one with only authors from the training (not necessarily all), and one with authors from the training and from outside the training. For the verification task, each test set included documents from the author to be verified and some documents from other authors as well. A complete description of the rationale on the construction of these corpora is given in [2]. 2.2
Supervised learning methods are used to calculate the most likely class given a set of features. We are given a training set D of labeled documents < d, a >, where d ∈ D is set of documents, and a ∈ A is the set of authors, in order to obtain the classification model.
In this paper we evaluated three supervised learning methods (Rocchio, Naïve Bayes and Greedy) which are described as follows.
Rocchio Rocchio classification is a form of Rocchio relevance feedback. The average of the relevant documents, corresponding to the most important component of the Rocchio vector in relevance feedback, is the centroid of the “class” of relevant documents (the authorship centroid in our case). We omit the query component of the Rocchio formula in Rocchio classification since there is no query in text classification. Rocchio classification can be applied to classes whereas Rocchio relevance feedback is designed to distinguish only two classes, relevant and nonrelevant.
The centroid calculation formulae is shown in Eq.(
−−→ μ(a) = 1
X −→v(d),
|Da| d∈Da
(
In this approach we have used a vectorial document representation with a TF-IDF weighting schema[3]. The classification criterion presented in algorithm 2 uses the Euclidean distance.
Input: A : Set of authors; D : Set of documents Output: Centroids of each author document set
2 Dj ← {d :< d, aj >∈ D}; 3 −→μj ← |D1a| Pd∈Da −→v(d); 4 end 5 return {−→μ1, · · · , −→μJ }
Input: {−→μ 1, · · · , −→μJ } : Centroids; d : Document of being classifed
Output: The best class (author) for the input document 1 return arg minj| μ j − −→v(d)|
−→ Naïve Bayes We have used a probabilistic supervised learning method named multinomial Naïve Bayes in order to determine the authorship attribution (as described in [1]). The probability of a document d being written by author a is computed as shown in Eq.(2).
P (a|d) ≈ P (a) Y
P (tk|a) where P (tk|a) is the conditional probability of the k-th term (tk) occurring in a document written by author a. Actually, P (tk|a) measures the contribution of term tk so that the document d belongs to class a. nd is the number of terms in document d. P (a) is the prior probability of a document written by author a. Since we are really interested in finding the best class (author) for the document, we may calculate the maximum a posteriori (MAP) as shown in Eq.(3).
amap = arg max P ∗(a|d) = arg max P ∗(a) a∈A a∈A
Y 1≤k≤nd
P ∗(tk|a)
P ∗(tk|a) is estimated by using Laplace smoothing, which simply adds one to each count (See Eq. (4)).
P ∗(tk|a) =
Tatk + 1 Pt′∈V (Tat′ + 1) (2) (3) where Tatk is the number of occurrences of tk in training documents from class a, including multiple occurrences of a term in a document and V is the corpus vocabulary. A greedy approach was also employed in the task of authorship attribution. We calculated a similarity matrix among all the documents in the training set. Thereafter, we selected the 100 most similar documents with respect to each document of the test set. The final class (author) is obtained by counting the most frequent class (author) from those 100 documents. 3
In Tables 1, 2, 3 and 4 we may see the obtained results for the three different approaches. Although we have used the tags Rocchio, Naïve Bayes and Greedy for these approaches, in the task description paper [2] they may be reported as authorship-vilarino-2011-0531-1456/, authorship-vilarino-2011-05-31-1455/ and authorship-vilarino-2011-05-311454/, respectively.
We have also included include the best run, the worst run and the arithmetic mean of all the runs submitted at the competition, so that the reader may compare the three approaches reported in this paper with respect to the rest runs.
In general, Rocchio obtained a more stable behavior than Naïve Bayes and the Greedy approach. However, it is worth noticed that Naïve Bayes outperformed the Rocchio classifier in terms of macro average precision but lousy macro average recall, which is likely because it will tend to classify most documents into the largest document classes, and so will not do well for the less frequent authors.
By comparing the arithmetic mean (obtained with all the runs submitted to the competition) with respect to the presented approaches, we may suggest that both, Rocchio and Naïve Bayes would be used as baselines in future competitions of authorship attribution. Further analysis of variance is needed in order to confirm this claim. It is important to notice that these classifiers were executed with raw data, i.e., none feature selection was performed and, therefore, we expect that any features analysis would improve significantly the performance of these classifiers.
In Tables 5, 6 and 7 it is presented the results obtained when the classifiers were executed using corpora attempting to detect only one author. In other words, the task
Having seen that Rocchio performs well, we are in conditions of analyzing and exploring different feature selection techniques for improving the authorship identification task. 4. Yang, Y.: Noise reduction in a statistical approach to text categorization. In: Proc. of the 18th annual international ACM SIGIR conference on Research and development in information retrieval - SIGIR-ACM. pp. 256–263 (1995)