In this paper, we present our solution to the PAN workshop challenge of authorship attribution. In multiple sub-problems, the original authors from given documents have to be chosen from a fixed training set. The core of our approach lies in traditional character n-gram analysis in combination with a linear SVM as a classifier. To find optimal values for our model parameters, we combined predefined parameters determined by a preliminary run on a training set run with dynamically determined parameters from an ad-hoc grid search approach.
The 2018 PAN workshop [12] features an authorship attribution task [
First, the training documents for each author cover a different domain than the testing documents. All texts are chosen from different fan fiction domains, in which fans of a specific author, novel, movie, TV-show, etc. produce new content while adhering to the original work’s environment and atmosphere. Typical examples of such domains, which will be referred to as fandom in the remainder of this paper, are Harry Potter or Star Wars. The testing part of each dataset contains at least one text from each author, but only covers one fandom. The training part covers multiple fandoms that are distinct from the evaluation fandom, whereby the same number of documents is provided for each author. Table 1 shows an exemplary schematic overview of the task. It can be seen that every sub-problem can have a varying amount of authors, and each author can have different fandoms (displayed as capital letters) available for training.
Secondly, the sub-problems cover a variety of different languages and sizes. Documents within a sub-problem are all in the same language, and the sub-problems themselves can be English, French, Italian, Polish or Spanish.
For developing a classification model, the organizers of the challenge have provided
a development dataset. The characteristics of this dataset can be seen in Table 2. Each
contestant is given a virtual machine on the tira web service [
For evaluation, the macro-F1-score is computed for each sub-problem, and the arithmetic mean of these scores represents the final score of a contestant’s model.
Problem 1: English author train test m e l b o r
P
In 2011 and 2012, similar authorship attribution tasks have been held an PAN [
Sapkota et al. [8] show that the cross-topic nature of datasets increases the difficulty for classification tasks for many traditional models. Furthermore, they demonstrate that by increasing the number of topics that are used for training (while still not having the testing topic(s) available), the accuracy of their model can be increased significantly.
The use of character n-grams has been proven to be an efficient feature for
authorship attribution in multiple works [
Along the line of these works, the main focus of this work is to build a generalpurpose model based on character n-grams that is able to perform well on different languages and topics. 3
To make the start of the challenge easier for the contestants, a baseline script was provided along with the training dataset. It uses the scikit-learn python machine learning library1 and character 3-gram frequencies in combination with a linear SVM. Based on this approach, we implemented several improvements.
Originally, we wanted to implement a dynamic search for optimal parameters for each sub-problem individually. Therefore, we made use of the grid search tool shipped with scikit-learn, which tries all combinations of given parameter ranges in a bruteforce fashion. The detailed values of the tested parameters are listed in Table 3. All runs were performed with 5-fold cross validation, while optimizing the f1_macro target.
This way, an important limitation of the model is its computational complexity. Even with a small set of possible parameter values, the number of possible combinations increases quickly. For example, given the parameter combinations in Table 3 and the 5-fold cross validation, each sub-problem is trained 10,800 times before one parameter combination is selected for predicting the testing data. This made this approach difficult for the evaluation run, where no information regarding the corpus size was given. Therefore, we limited the complete grid-search procedure to the development dataset and fixed the parameters that showed the most similar values for multiple problems. For example, the parameter strip accents, which if set, removes accents from letters, was chosen to be true for most of the sub-problems of the development dataset by the grid search. For the sub-problems with the parameter set to false, we were not able to determine which characteristic of the sub-problem caused the change of the parameter. Therefore, a majority vote was performed instead and the parameter was set to true for the entire evaluation set.
Two remaining parameters (i.e., the minimal document frequency of character ngrams and their size), that did not show a clear majority among the sub-problems, were left in the grid search to be determined dynamically for each part of the evaluation set, yielding 6 3 = 18 possible parameter combinations for each sub-problem. The fixed values are displayed in bold in Table 3, whereas the per-problem trained parameters are printed in italics. 4
The results of the development dataset can be seen in Table 4a. As expected, it can be seen that the (even numbered) shorter sub-problems (containing less authors) are generally easier to classify than the bigger ones. No significant differences between the different languages can be detected. For overall evaluation and comparison between the 1 http://scikit-learn.org contestants, the arithmetic mean of all macro-F1-scores is used, which is displayed at the bottom of the table.
While the details of the evaluation dataset are not available at the time of writing this paper, the evaluation results are visible for the author. In Table 4b, the F1-scores for the evaluation dataset are displayed. The sparse information available suggests that problems 13,15 and 16 seem to be especially hard for our model and have lessened our total score notably. However, no implications can be made on the reasons for this performance at this point.
Table 5 shows the final ranking of the contestants. It can be seen that our solution reached the second rank, clearly beating the baseline provided by the task organizers. In this paper, a combination of precomputed parameters and a dynamic grid search is used for the task of cross-fandom authorship attribution. Our method relies on traditional character n-gram analysis, which uses specific parameters for each sub-problem, which are found by a standard grid-search approach. Although the methodology is simple in its approach, we were able to reach the second place on the PAN task leader board. Given more time and resources, more parameters could be optimized at runtime rather than pre-calculating a sensible default value. 8. Sapkota, U., Solorio, T., y Gómez, M.M., Bethard, S., Rosso, P.: Cross-Topic Authorship Attribution: Will Out-Of-Topic Data Help? In: Proceedings of the 25th International Conference on Computational Linguistics (COLING’2014). pp. 1228–1237 (Aug 2014) 9. Stamatatos, E.: A Survey of Modern Authorship Attribution Methods. Journal of the
American Society for Information Science and Technology 60(3), 538–556 (March 2009) 10. Stamatatos, E.: On the Robustness of Authorship Attribution Based on Character N-Gram
Features. Journal of Law & Policy pp. 421–439 (2013) 11. Stamatatos, E.: Authorship Attribution Using Text Distortion. In: Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics (EACL’2017). pp. 1138–1149. Association for Computational Linguistics (Apr 2017) 12. Stamatatos, E., Rangel, F., Tschuggnall, M., Kestemont, M., Rosso, P., Stein, B., Potthast, M.: Overview of PAN-2018: Author Identification, Author Profiling, and Author Obfuscation. In: Bellot, P., Trabelsi, C., Mothe, J., Murtagh, F., Nie, J., Soulier, L., Sanjuan, E., Cappellato, L., Ferro, N. (eds.) Experimental IR Meets Multilinguality, Multimodality, and Interaction. 9th International Conference of the CLEF Initiative (CLEF 18). Springer, Berlin Heidelberg New York (Sep 2018)