This paper gives an overview of the evaluation methodology applied to authorship identi cation solutions as part of PAN 2011. The two variations of authorship identi cation that were explored were authorship attribution, determining which of a known set of authors wrote a text, and authorship veri cation, determining if a speci c authors did or did not write a text. We summarize the methods used by the various participants, which were quite varied, and present the overall results of the evaluation.
There has been much interest in recent years in research on automatic
methods for determining the authorship of anonymous documents based on internal
evidence [
A total of 13 di erent research groups submitted results for 7 di erent tasks within this evaluation framework, eight of which submitted papers describing their systems. In this paper we describe the evaluation framework, and report on evaluation of the di erent authorship analysis methods. 1.1
In the basic form of the authorship attribution problem, we are given examples of the writing of a number of candidate authors and are asked to determine which of them authored a given anonymous text. In this straightforward form, the authorship attribution problem ts the standard modern paradigm of a text categorization problem [15]. The components of text categorization systems are by now fairly well-understood: documents are represented as numerical vectors that capture statistics of potentially relevant features of the text and machine learning methods are used to nd classi ers that separate documents that belong to di erent classes.
However, real-life authorship identi cation problems are rarely as elegant as straightforward \research-type" text categorization problems, in which we have a small closed set of candidate authors and essentially unlimited training text for each. One important issue that arises in the real world is the existence of an open candidate set, that is, the actual author might be an author we don't know about at all. In this case, the problem is to assign the document either to one of the authors we know of, or to \Someone Else".
The most reduced version of this open-candidate case is that where there is
no candidate set at all, but just a single suspect. In this case, the challenge is to
determine if the suspect is or is not the author. This is called the authorship
veri cation problem. As a categorization problem, veri cation is signi cantly more
di cult than basic attribution and less work has been done on it, but see, e.g.,
[
A corpus was developed, based on the Enron email corpus3, to account for several di erent common attribution and veri cation scenarios. The corpus contains ve separate training collections, and seven test collections, as follows. Two training sets are provided for authorship attribution, a \Large" set containing 9337 documents by 72 di erent authors and a \Small" set containing 3001 documents by 26 di erent authors (the author sets are disjoint). For each attribution problem, two test sets are provided, one containing texts only written by the authors in the training set, and one also containing texts written by around 20 other authors each.
The other three training sets are for veri cation, and so contain only emails from a single author (di erent from those in other training sets). The veri cation training sets contain 42, 55, and 47 documents, respectively. Each has an associated test set comprising a mixture of documents written by the training 3 http://www.cs.cmu.edu/ enron/ author and written by others (some of these are from the Enron corpus, and some are not).
As the tasks are intended to re ect a natural task environment, there are some texts, both in training and in testing sets, that are not in English, or that are automatically generated.
Personal names and email addresses in the corpus have been automatically redacted, and replaced (on a token-by-token basis) by <NAME/> and <EMAIL/> tags, respectively. This redaction is admittedly imperfect, but random spotchecking was applied to reduce the likelihood of missing occurrences. Other than this redaction, each text is typographically identical to the original electronic text, so systems could, in principle, rely on line length, punctuation, and the like.
Finally, authorship was determined based on From: email headers; this necessitated determining, in some cases, that multiple email addresses corresponded to the same individual. Manual spot-checking was applied here as well to ensure quality, though some errors were let through and discovered during the evaluation. 2.2
For evaluating authorship identi cation, we used the standard information retrieval metrics of precision, recall, and F1. Precision, for a particular author A, is de ned as the fraction of attributions that a system makes to A that are correct: Recall, for a particular author A, is de ned as the fraction of test documents written by A that are (correctly) attributed to A: F1 is de ned as the harmonic mean of recall and precision:
For the authorship attribution tasks, we need to aggregate these measures over all the di erent test authors. We applied two methods with di erent properties, macro-averaging and micro-averaging. For a given metric M , set of n authors fAig, with a total of k test documents, these are de ned as: macro-avgM (fAig) = n1 Pi MAi micro-avgM (fAig) = k1 Pi kiMAi where ki is the number of test documents written by author Ai. Micro-averaging will give more credit to accuracy on authors with more test documents, while PA =
correct(A) attributions(A) RA =
correct(A) documents-by(A) F1 = 2PARA
PA + RA macro-averaging gives the same credit to all authors, even if they wrote just one test document.
For authorship veri cation, the author set contains just one author, so averaging is not necessary.
Finally, to achieve an overall ranking for each task, we ranked system performances for each of the measures{six for attribution (macro- and micro-averaged P , R, and F1) and three for veri cation, and summed the ranks for each entry in each task. The lower the rank sum, the better (overall) the performance. 3
All documented submissions, within their diversity, followed classic methodology for authorship identi cation, and (a) identi ed a set of features that were calculated from the texts, whose values then (b) served as input to some classi cation algorithm. In this section, we summarize the submissions in terms of what features they used and what algorithms they used. We note that, as in any such summary of varied systems, we inevitably must oversimplify, so please see the full papers describing each system for more information. 3.1
There were a number of di erent kinds of features used by participants, some traditional, some quite novel.
The rst type of feature are those derived from the word usage in the texts, which we term lexical features. Simplest, are the frequencies of the various words and word n-grams that appear in the text. Also relevant, based on previous studies, are the relative frequencies of function words (or stopwords) and of speci c classes of words: pronouns, modal verbs, discourse linking words/phrases (such as \however", \on the other hand"), slang terms, contractions, and emoticons (or smileys). Also considered were frequencies of US vs. UK variants (a dialect indicator), various types of spelling errors, di erent types of named entities (people, organizations, dates, etc.), and semantic features of words (polysemy, speci city of meaning, etc.).
The second type of feature are those at the character level, and include character n-grams (usually for n = 3), frequent su xes, and punctuation usage.
The third type of feature considered relate to the format of the text, including various length-related features (lengths of lines, words, sentences), overall formatting of the text (e.g., fraction of empty lines), orthographic features (e.g., capitalization, frequency of non-alphanumeric characters), and a novel feature, Intro/Outro that looked for common beginnings and endings of texts, and noted their presence/absence as cues to authorship.
The fourth type of feature were syntax related features, both part-of-speech n-grams and phrase types (or dependency link types).
Additionally, two of the submissions used forms of complexity measures over sentences and words, by measuring such things as perplexity and morphological complexity.
The last, perhaps most novel, kind of feature used was one used by Solorio et al., based on clustering the training data and measuring the distance of various texts from the cluster centroids, using those distances as features for learning.
The types of features and the submissions using each are listed in Figure 1. 3.2
A wide variety of algorithmic approaches were taken by the participants. Several
used di erent forms of linear classi ers. Support vector machines [
Other machine learning approaches were also applied. Tanguy et al. applied
maximum entropy learning [12] for attribution, and decision trees [13] and rule
learning [
Authorship attribution results, for the four attribution tasks, are given in Tables 2 through 5. As the tables show, the authorship attribution approach of Tanguy et al. was very highly ranked across all the attribution tasks. It was beaten only once signi cantly by the approach of Kourtis and Stamatatos on the Large task. The approach of Kern et al. achieved very high precision on the Small attribution tasks, but paid for it in reduced recall. 4.2
Veri cation results are given in Table 6. Snider et al. achieved the best precision performance overall, though not the highest recall. It should be mentioned that
Macro-averaged Micro-averaged Rank Run Prec Recall F1 Prec Recall F1 Sum tanguy-2011-06-07-1700 0.688 0.267 0.321 0.779 0.471 0.587 9 snider-2011-06-08-1548 0.654 0.227 0.258 0.627 0.405 0.492 20 kern-2011-06-08-1500 0.673 0.179 0.226 0.802 0.383 0.518 21 tanguy-2011-06-07-1600 0.806 0.148 0.208 0.924 0.299 0.451 26 escalante-2011-06-07-0934 0.53 0.203 0.191 0.446 0.446 0.446 29 vilarino-2011-05-31-1456 0.347 0.245 0.263 0.368 0.368 0.368 32 mikros-2011-06-08-2245 0.398 0.183 0.209 0.499 0.292 0.369 36 vilarino-2011-05-31-1455 0.488 0.084 0.088 0.222 0.222 0.222 50 ryan-2011-06-08-2331 0.19 0.154 0.132 0.216 0.216 0.216 53 eriksson-2011-06-13-0920 0.432 0.064 0.062 0.201 0.201 0.201 59 vilarino-2011-05-31-1454 0.153 0.092 0.089 0.175 0.175 0.175 63 noecker-2011-06-08-2356 0.227 0.054 0.06 0.037 0.037 0.037 70 noecker-2011-06-08-2337 0.001 0.011 0 0.001 0.001 0.001 78 authorship veri cation is considerably more di cult than authorship attribution. High precision evidently is easier to achieve than high recall. 5
With the great variety of feature sets and classi cation methods applied, it is di cult to form any overall conclusions from the basic results; more nuanced understanding will have to emerge from discussion among researchers and follow-on studies. One thing that is clear, however, is the need to decouple, to the extent possible, feature choice from classi cation method, so that the separate advantages and de ciencies of di erent feature types and algorithms can be understood, as well as their interactions. As well, one characteristic of all the better methods seems to be a preference for precision over recall (which is probably preferred in real-world applications), as in the more di cult open tasks, precision generally stayed high, while recall declined.
Regarding the di erent methods, the best method overall for attribution was that of Tanguy et al., who applied the largest and most diverse feature set to the problem, which may indicate the usefulness to nd ways of pro tably learning classi ers from very large numbers of features with diverse characters.
We would like to thank the organizers of CLEF 2011 and PAN 2011 for their support which enabled this competition to take place. Thanks are also due all the participants who made this e ort such a success.
Development of this competition was funded in part by National Science Foundation grant CRI-CRD-0751198. 8. M. Koppel and J. Schler. Authorship veri cation as a one-class classi cation problem. In Proceedings of the twenty- rst international conference on Machine learning, page 62. ACM, 2004. 9. M. Koppel, J. Schler, and S. Argamon. Computational methods in authorship attribution. Journal of the American Society for information Science and Technology, 60(1):9{26, 2009. 10. D. Lewis. Naive (bayes) at forty: The independence assumption in information retrieval. Machine Learning: ECML-98, pages 4{15, 1998. 11. K. Luyckx and W. Daelemans. Authorship attribution and veri cation with many authors and limited data. In Proceedings of the 22nd International Conference on Computational Linguistics-Volume 1, pages 513{520. Association for Computational Linguistics, 2008. 12. K. Nigam, J. La erty, and A. McCallum. Using maximum entropy for text classi cation. In IJCAI-99 workshop on machine learning for information ltering, volume 1, pages 61{67. Citeseer, 1999. 13. J.R. Quinlan. C4. 5: programs for machine learning. Morgan Kaufmann, 1993. 14. J.J. Rocchio. Relevance feedback in information retrieval. Prentice-Hall, Englewood
Cli s NJ, 1971. 15. F. Sebastiani. Machine learning in automated text categorization. ACM computing surveys (CSUR), 34(1):1{47, 2002. 16. E. Stamatatos. A survey of modern authorship attribution methods. Journal of the
American Society for information Science and Technology, 60(3):538{556, 2009. 17. I. Tsochantaridis, T. Hofmann, T. Joachims, and Y. Altun. Support vector machine learning for interdependent and structured output spaces. In Proceedings of the twenty- rst international conference on Machine learning, page 104. ACM, 2004. 18. H. van Halteren. Linguistic pro ling for author recognition and veri cation. In Proceedings of the 42nd Annual Meeting on Association for Computational Linguistics, pages 199{es. Association for Computational Linguistics, 2004.