The task of authorship attribution (AA) requires text features to be represented according to rigorous experiments. In the current study, we aimed to develop three different n-gram models to identify authors of various fanfictional texts. Each of the three models was developed as a variable-length ngram model. We implemented both a standard character n-gram model (2 ¡ 5 gram), a distorted character n-gram model (1 ¡ 3 gram) and a word n-gram model (1¡3 gram) to not only capture the syntactic features, but also the lexical features and content of a given text. Token weighting was performed through term-frequency inverse-document frequency (tf-idf ) computation. For each of the three models, we implemented a linear Support Vector Machine (SVM) classifier, and in the end applied a soft voting procedure to take the average of the classifiers' results. Results showed, that among the three individual models, the standard character n-gram model performed best. However, the combination of all three classifier's predictions yielded the best results overall. To enhance computational efficiency, we computed dimensionality reduction using Singular Value Decomposition (SVD) before fitting the SVMs with training data. With a run time of approximately 180 seconds for all 20 problems, we achieved a macro F1-score of 70.5% for the development corpus and a F1-score of 69% for the competition's test corpus, which significantly outperformed the PAN 2019 baseline classifier. Thus, we have shown that it is not a single feature representation that will yield accurate classifications, but rather the combination of various text representations that will depict an author's writing style most thoroughly.
Authorship Attribution (AA) is the task of determining the author of a text from a set
of candidates. It can be a daunting exercise for both humans and machines, if one
does not know which parts of a document represent an author’s writing style.
However, if features are represented according to rigorous experiments (e.g., through the
use of regular expressions and hyper parameter optimization) and adequately
capture the syntactic use of language, it may well support automated systems which aim
to recognize a text’s author. In the context of machine learning, AA can be regarded
as a multi-class, single-label text classification problem [
In this working notes paper, we describe our approach to the cross-domain AA
task of the PAN 2019 competition, which comprised of the identification of the
authors of fan fiction [
Similarly to last year’s best performing team [
The first model we implemented, which was a standard character n-gram model,
consisted of bigram, trigram, four-gram and five-gram token representations (i.e.,
2 ¡ 5 grams). Hyperparameter tuning experiments revealed that, 2 ¡ 5 gram
models yield the best results and represent an author’s text more thoroughly than any
other additive lower or higher n-gram text representations. However, for this
standard character model we also computed an additional vector of unigram
punctuation marks, which we then concatenated with all other character n-gram
representations after pre-processing computation. We kept punctuation marks as we
considered them important features for an author’s use of language. The use, and in
particular the frequency, of punctuation marks in a given text, reveals crucial information
about an author’s writing style [
Lastly, we computed a variable-length word n-gram model, which consisted of unigram, bigram and trigram token representations (i.e., 1 ¡ 3 grams). As a word ngram model, contrary to a character n-gram model, rather captures the vocabulary and register than morhological and syntactic features and thus draws attention to the content, we removed punctuation marks and special characters as such tokens do not reveal decisive information about an author’s choice of words. We only kept alphanumerical characters.
To summarize, for all three n-gram models, we computed the following text preprocessing steps: – Each non-zero digit was represented by a 0 digit place-holder. This step was computed, since different digits neither reveal crucial information about the vocabulary and register nor about the syntax of a given text. What’s important is whether a digit or number appeared in a text and how frequently it was used. Furthermore, to gather information about syntactic features, it is indispensable to know how numerical characters were depicted by an author (see section on standard character n-gram model). – Each hyperlink, that appeared in a given text, was replaced by an @ sign. We computed the latter step, as we did not deem hyperlinks crucial features for an author’s use of language. What’s decisive is whether and how frequently hyperlinks were used within a given text but not the actual hyperlink that was cited. – For each string, new lines denoted as either "\n" or "\t" were replaced by the empty string. – We did not perform text lower-casing as we suppose, that the usage (and in particular its corresponding frequency) of upper-case letters (e.g., nouns, proper nouns, entities, named entities) reveal important information about the writing style of an author. – We did not remove stop words as we believe, that function words (e.g., "the", "a", "have") - which, in the vast majority of documents, are the most frequently used words - reveal decisive insights into an author’s use of language. However, we still assign lower weights to terms that appear highly frequently across the entire corpus. – Weighting for both character and word n-gram models was performed through term-frequency inverse-document-frequency (tf-idf) computation to assign lower weights to words, which appear more frequently across the entire corpus, and higher weights to words which are found only in specific documents and thus reveal crucial information about a text. To prevent a division by zero, we computed smooth inverse document frequency instead of normal idf. Smoothing idf means adding a 1 to the denominator within the logarithmic fraction: log ³ 1Ånt . N ´ – To decrease dimensionality, enhance computational efficiency and reduce time complexity, we computed Singular Value Decomposition (SVD) for each of the three variable length n-gram models. The final number of dimensions chosen (d Æ 63) was dependent on the number of samples per problem. As each problem consisted of 63 texts and d cannot be higher than the number of samples in SVD, we deemed this number as an appropriate size for d to capture the most variance. 2.3
For each of the three models (standard character, distorted character and word
ngrams) we developed linear, multi-class Support Vector Machines (SVM). In so doing,
we made use of Python’s Sklearn library [
Results of the three individual SVMs were combined in a soft-voting manner: We simply averaged the probabilities for each candidate author across the three models. Initial experiments revealed that a pure averaging yields better results than feeding the probabilities into a new classifier. Hence, we did not deploy a fourth model (see Figure 1).
The soft voting computation was performed as follows: arg max
Pk iÆ1 pi k (1) where pi is the probabilities vector for each individual model and k Æ 3 – Firstly, the predicted probabilities obtained by each individual classifier were concatenated into a n £ d matrix, where n is the number of classifiers (n Æ 3) and d denotes the number of classes / candidates. – Secondly, for each text, we computed the average among the three 1£d probability vectors, which, according to soft voting, yields a more accurate probabilities depiction than individual probability distributions. – Lastly, for each text, a "new" average probability distribution was computed, which served as the probabilities vector for the final prediction.
Our classifiers were required to consider that a test text may have been written by an
unknown author. To enhance this algorithmic decision-making process, our models
classified a text’s author as unknown, if the difference between the model’s highest
and second highest probabilities was below 0.1, or if the highest probability was
below 0.25. The latter served as an additional feature (compared to the PAN 2019
baseline model), we regarded as crucial to include in our algorithm. The hyper
parameters of the final analysis pipeline are summarized in Table 2. Our final model for the
PAN 2019 shared task was deployed on a virtual machine using the TIRA architecture
[
No changes were made to the provided baseline classifier; as such it utilized only character trigrams, a minimum document frequency of 5 (character trigrams that appeared less frequently than five times across the corpus were not included in the vocabulary), no text lower casing and no weighting of the bag of words (only using a count-based approach), a one-vs-rest multi-class strategy. It classified documents as unknown, if and only if, the highest and second highest predictions were less than 0.1 apart.
Results showed that the standard character n-gram model performed generally better than the variable length word n-gram or distorted character n-gram model. However, in the vast majority of runs, the soft voting classifier resulted in a higher score than any of the individual models (see Table 2). According to the mean scores obtained for the three individual models, the word n-gram model showed the worst performance. This is in line with the assumption, that authorship manifests itself in style and thus a text’s syntax and morphology rather than in a text’s vocabulary. Word n-grams primarily encode lexical information about a document, whereas standard character n-grams and diacritic characters n-grams rather capture the syntactic and morphological characteristics of the text, which further reveal information about an author’s writing style.
SVD implementation reduced our algorithm’s computational time from approx. 30 minutes to just 180 seconds. In the final evaluation, we, unfortunately, were unable to deploy SVD due to time constraints, which is why the run time for our PAN 2019 model is 30 minutes and not 180 seconds. The macro F1-score (approx. 70%), however, did not change as a result of dimensionality reduction. 4
Results displayed that the soft voting procedure notably outperformed the individual classifiers. Intuitively, one might assume that averaging over classifiers with lower macro F1-scores might yield a worse and not a better performance. However, we did not perform averaging across the F1-scores, but across the predicted probabilities yˆi obtained for each classifier. This served as an additive factor as all three feature representations were combined into one thorough representation.
One limitation to our approach might have been that we did not apply a weighted voting procedure. This could and shall be addressed in future research. Moreover, it might be interesting to consider a hard instead of a soft voting procedure. We further encourage others to deploy different machine learning classifiers, such as Multinomial Logistic Regression, Multinomial Naive Bayes or Neural Networks. Additionally, further experiments might compute Principal Component Analysis (PCA) instead of Singular Value Decomposition (SVD) to reduce dimensionality and enhance computational efficiency. 5
This paper presented our soft voting ensemble classifier for cross-domain authorship attribution. We combined a standard character n-gram model, a distorted character n-gram model and a word n-gram model to achieve more accurate predictions than the individual models themselves. All three models represented the texts as variable length n-grams, which were weighted by term frequency-inverse document frequency (tf-idf ). Our algorithm can be perceived as an enhancement of the PAN 2019 baseline system. One may infer from the results we obtained, that authorship attribution models generally benefit from the inclusion of different representations of text. It is not a single feature representation that will yield accurate classifications, but rather the combination of various document representations that will depict an author’s writing style.