In this paper, we present our solution to the PAN workshop challenge of authorship attribution. In multiple sub-problems, the original authors from given documents have to be chosen from a fixed training set. The core of our approach lies in traditional character n-gram analysis in combination with a linear SVM as a classifier. To find optimal values for our model parameters, we combined predefined parameters determined by a preliminary run on a training set run with dynamically determined parameters from an ad-hoc grid search approach.
The 2018 PAN workshop [12] features an authorship attribution task [3] that consists of multiple sub-problems. The main challenge of the task lies in two characteristics:
First, the training documents for each author cover a different domain than the testing documents. All texts are chosen from different fan fiction domains, in which fans of a specific author, novel, movie, TV-show, etc. produce new content while adhering to the original work's environment and atmosphere. Typical examples of such domains, which will be referred to as fandom in the remainder of this paper, are Harry Potter or Star Wars. The testing part of each dataset contains at least one text from each author, but only covers one fandom. The training part covers multiple fandoms that are distinct from the evaluation fandom, whereby the same number of documents is provided for each author. Table 1 shows an exemplary schematic overview of the task. It can be seen that every sub-problem can have a varying amount of authors, and each author can have different fandoms (displayed as capital letters) available for training.
Secondly, the sub-problems cover a variety of different languages and sizes. Documents within a sub-problem are all in the same language, and the sub-problems themselves can be English, French, Italian, Polish or Spanish.
For developing a classification model, the organizers of the challenge have provided a development dataset. The characteristics of this dataset can be seen in Table 2. Each contestant is given a virtual machine on the tira web service [6], on which the model can run. To prevent cheating, evaluation runs must be initiated with a web-application, and any external network connections are cut off from the virtual machine once started.
For evaluation, the macro-F1-score is computed for each sub-problem, and the arithmetic mean of these scores represents the final score of a contestant's model.
In 2011 and 2012, similar authorship attribution tasks have been held an PAN [1,2]. One of the main differences was that the dataset used consisted of single-topic (or domain) documents. Sapkota et al. [8] show that the cross-topic nature of datasets increases the difficulty for classification tasks for many traditional models. Furthermore, they demonstrate that by increasing the number of topics that are used for training (while still not having the testing topic(s) available), the accuracy of their model can be increased significantly.
The use of character n-grams has been proven to be an efficient feature for authorship attribution in multiple works [4,10,9]. Variations to this feature (e.g., by distorting the text [11] or distinguishing the relative position of each n-gram inside each word [7]) can improve a model even further. Markov et al. [5] showed that character-n-gram-based models can be used efficiently for cross-topic authorship attribution, and preprocessing the corpora can improve the performance of cross-topic models.
Along the line of these works, the main focus of this work is to build a generalpurpose model based on character n-grams that is able to perform well on different languages and topics.
To make the start of the challenge easier for the contestants, a baseline script was provided along with the training dataset. It uses the scikit-learn python machine learning library 1 and character 3-gram frequencies in combination with a linear SVM. Based on this approach, we implemented several improvements.
Originally, we wanted to implement a dynamic search for optimal parameters for each sub-problem individually. Therefore, we made use of the grid search tool shipped with scikit-learn, which tries all combinations of given parameter ranges in a bruteforce fashion. The detailed values of the tested parameters are listed in Table 3. All runs were performed with 5-fold cross validation, while optimizing the f1_macro target.
This way, an important limitation of the model is its computational complexity. Even with a small set of possible parameter values, the number of possible combinations increases quickly. For example, given the parameter combinations in Table 3 and the 5-fold cross validation, each sub-problem is trained 10,800 times before one parameter combination is selected for predicting the testing data. This made this approach difficult for the evaluation run, where no information regarding the corpus size was given. Therefore, we limited the complete grid-search procedure to the development dataset and fixed the parameters that showed the most similar values for multiple problems. For example, the parameter strip accents, which if set, removes accents from letters, was chosen to be true for most of the sub-problems of the development dataset by the grid search. For the sub-problems with the parameter set to false, we were not able to determine which characteristic of the sub-problem caused the change of the parameter. Therefore, a majority vote was performed instead and the parameter was set to true for the entire evaluation set.
Two remaining parameters (i.e., the minimal document frequency of character ngrams and their size), that did not show a clear majority among the sub-problems, were left in the grid search to be determined dynamically for each part of the evaluation set, yielding 6 × 3 = 18 possible parameter combinations for each sub-problem. The fixed values are displayed in bold in Table 3, whereas the per-problem trained parameters are printed in italics.
The results of the development dataset can be seen in Table 4a. As expected, it can be seen that the (even numbered) shorter sub-problems (containing less authors) are generally easier to classify than the bigger ones. No significant differences between the different languages can be detected. For overall evaluation and comparison between the contestants, the arithmetic mean of all macro-F1-scores is used, which is displayed at the bottom of the table.
While the details of the evaluation dataset are not available at the time of writing this paper, the evaluation results are visible for the author. In Table 4b, the F1-scores for the evaluation dataset are displayed. The sparse information available suggests that problems 13,15 and 16 seem to be especially hard for our model and have lessened our total score notably. However, no implications can be made on the reasons for this performance at this point.
Table 5 shows the final ranking of the contestants. It can be seen that our solution reached the second rank, clearly beating the baseline provided by the task organizers.
In this paper, a combination of precomputed parameters and a dynamic grid search is used for the task of cross-fandom authorship attribution. Our method relies on traditional character n-gram analysis, which uses specific parameters for each sub-problem, which are found by a standard grid-search approach. Although the methodology is simple in its approach, we were able to reach the second place on the PAN task leader board. Given more time and resources, more parameters could be optimized at runtime rather than pre-calculating a sensible default value.