In this work, we perform authorship attribution on a new dataset of German news articles. We seek to classify over 3,700 articles to their five corresponding authors, using four conventional machine learning approaches (na¨ıve Bayes, logistic regression, SVM and kNN) and a convolutional neural network. We analyze the effect of character and word n-grams on the prediction accuracy, as well as the influence of stop words, punctuation, numbers, and lowercasing when preprocessing raw text. The experiments show that higher order character n-grams (n = 5,6) perform better than lower orders and word n-grams slightly outperform those with characters. Combining both in fusion models further improves results up to 92% for SVM. A multilayer convolutional structure allows the CNN to achieve 90.5% accuracy. We found stop words and punctuation to be important features for author identification; removing them leads to a measurable decrease in performance. Finally, we evaluate the topic dependency of the algorithms by gradually replacing named entities, nouns, verbs and eventually all tokens in the dataset according to their POStags.
When the author of a text is the subject of
particular interest, there exist three main approaches
in the field of natural language processing (NLP):
author profiling, authorship verification, and
authorship attribution. They mean respectively,
aiming at detecting details of the author such as age
or gender, measuring the similarity between an
author’s work and a text in question, and trying
to identify the author of a text given a group of
potential authors. All approaches are based on
the assumption that individuals have unique
writing styles and habits
Working on a new dataset of 3,700 German
news articles written by five authors, we carry out
a multiclass classification using different machine
learning (ML) models and linguistic features. As
ML models, we test multinomial na¨ıve Bayes
(NB), logistic regression (LR), support vector
machine (SVM) and k-nearest-neighbors (kNN)
using scikit-learn implementations, and a
convolutional neural network (CNN) for text
classification using PyTorch. We experiment with word
n-grams, character n-grams and fusions of both,
as well as punctuation, numbers, stop words, and
lowercasing. We further seek to evaluate the
topicdependency of the algorithms by gradually
replacing named entities, nouns, verbs and all tokens in
the dataset with the help of their part-of-speech
(POS) tags. This study aims to quantify the effect
of linguistic features on AA using a new source of
German news articles.
Previous approaches to AA greatly vary regarding
applied linguistic features, implemented ML
models and investigated languages. The combination
of character n-grams and traditional ML models
such as na¨ıve Bayes has been deployed for many
publications on AA, such as in Amasyalı and Diri
(2006) on Turkish news articles, Markov et al.
(2017) on Portuguese news articles, or Oppliger
(2016) on instant messages in Swiss German.
Punctuation n-grams have been used as
stylometric features for English, French, Italian and
Spanish
Originally developed for computer vision tasks, CNNs have been successfully applied in various text classification tasks lately, including AA. Ruder et al. (2016) presented state-of-the-art results in large-scale AA on various social media datasets. The authors of the paper implemented different multi-channel word and character CNNs and create hybrid models of both. On average, the char-CNN outperformed not only previous models, but also word CNNs and hybrids. In the study of Shrestha et al. (2017), a CNN with three convolutional layers was trained on character n-grams and outperformed traditional models as well as recurrent neural networks (RNN) on recognizing the authors of tweets. 3
We collect a new dataset of newspaper articles published by Main-Post, a German newspaper. In cooperation with the newspaper, we decide to choose the articles by five journalists from the regional department in Schweinfurt. In their daily work, the journalists cover similar topics, mostly local news in and around the city. With this choice, we hope to alleviate the likelihood of classifying authors by topic specific vocabulary. The newspaper reaches a circulation of around 40,000 readings per day in Schweinfurt. All articles (mostly behind a paywall) can be accessed via the company’s online presence.1 In a first step, we collect the weblinks to all articles for each author. Then, a second script opens each link and extracts the corresponding text into a csv-file. We clean the collected articles by removing:
Author names in the text where indicative of the writer (e.g. comments);
Articles listed more than once (e.g. regional and trans-regional versions); Non-text elements such as image or video boxes that were downloaded due to variations in the webpages’ html-structures.
The final dataset consists of 3,717 articles by five different journalists, written between May 2013 and October 2019. The number of articles per author is imbalanced and varies from 331 to 972. The average length of an article is 455 words and it comprises 24 sentences and 7.5 paragraphs. The shortest article measures 26 words, the longest 2299. The overall corpus size is 1.6 Million words. Compared to other author attribution datasets in literature, our dataset contains fewer authors, but larger available data per author. This facilitates the prediction task but allows to draw more meaningful conclusions about the effect of analyzed linguistic features. The dataset is not publicly available but can be obtained from the authors of this paper on reasonable request. 4 4.1
As a first step, we split off a stratified test set containing 20% of each author’s available articles. All final models are evaluated by their prediction accuracy on this test set. Since this is the first work on a new dataset, we start by establishing two baseline models. Due to their decent performance in many applications, we chose na¨ıve Bayes and logistic regression, a generative and a discriminative approach, respectively. We process the raw text using word unigrams obtained through splitting by whitespaces and test for the following linguistic features:
Stop words (keep/remove), using NLTK’s list of 232 German stop words; Lemmatization, using Spacy’s implementation for German; Stemming, using NLTK’s German SnowballStemmer; 1https://www.mainpost.de/regional/schweinfurt/
We further define a minimum document frequency of 5 and tune the model-specific hyperparameters via grid search. For NB, LR, SVM and kNN, we vectorize counting raw token frequencies using scikit-learn’s CountVectorizer. As a result of the baseline experiments, the best parameter combinations achieved 81.85% on the test set for na¨ıve Bayes and 90.59% for logistic regression. During our work on the baselines, we observed an average performance decrease of 2.8 percentage points for lemmatization and 3.9 percentage points for stemming, along with a drastically increasing runtime. Thus, both techniques were excluded from the following experiments. Instead, we focus on the effect of punctuation, numbers, stop words, and lowercasing on the prediction accuracy.
We expand our work with SVM and kNN models, and add bigrams and trigrams, as well as character n-grams of different lengths (n = 3 – 6). Finally, for each model, we combine the best word n-gram vectorizer with the best character n-gram vectorizer in a fusion model. For every combination of model and word/character n-gram, a random search with 100 iterations tests the beforementioned linguistic features and the model specific hyperparameters by averaging the results of a 5-fold cross-validation. Then, each model’s best configuration is trained on the whole training set and its performance on the test set is reported.
The implementation of CNN is based on the
work of
Additionally, replace all verbs (including auxiliary verbs) by ‘VB’; Finally, replace all tokens by their corresponding Treebank POS-tag.
Then, we run the best performing version for each machine learning model on these variations of the dataset and report the accuracy on the hold out test set. We utilize Spacy’s pretrained German tagger with a reported accuracy of 96.3% for POStagging. 5 5.1
The left side of Table 1 shows the best results obtained for the five implemented ML models. Overall, we note high performances above 80% for all models except the kNN approach. Logistic regression and SVM performed the best and achieved almost same results in all tested variations. Except for kNN, word n-grams slightly outperformed character n-grams. On word level, adding bigrams and trigrams outperformed unigrams, with only marginal differences between the models. On character level, higher order n-grams improved the results. Combining the best character and word n-grams in fusion models led to higher accuracies for LR and SVM. Here, SVM delivered the best predictions with 91.9% (macro-averaged F1score = 0.898). In all variations, SVMs with linear kernels were more accurate than those with polynomial kernels. On average, the performance of the CNN was two percentage points below the SVM. Both, character and word n-gram features achieved accuracies in the high 80s. The best performance of the CNN was 90.5% (macro-averaged F1-score = 0.855), applying two convolutional layers of filter sizes 2 and 3, with 500 filters each, after word level tokenization. Using smaller vocabulary sizes of 5k and 10k improved results, as well as high dropout values of 0.5.
Besides word and character n-grams, the conducted experiments allow us to quantify the effect of stop-words, punctuation, numbers and lowercasing on the performance for this dataset. The corresponding values are displayed in Table 2 and obtained by taking each model’s best implementation, retraining it either with or without the feature in question, and finally averaging the performance difference over all models for each feature. For all 28 random searches and 8 CNN-configurations, removing stop words or punctuation decreased the prediction accuracies. For the most accurate implementations, this resulted in an average decrease of 1.23 and 1.06 percentage points, respectively. Lowercasing and removing numbers on the other hand barely influenced the results. 5.2
The results of the experiments with reduced topicdependency are presented in Table 1. Replacing named entities by ’NE’ did not affect the performance negatively as expected. Instead, kNN,
on performance 1.23 1.06 0.09 + 0.09
SVM and na¨ıve Bayes slightly improved their performances and the SVM reached 92.2% (macroaveraged F1-score = 0.901), the highest accuracy in the project. Replacing nouns and then verbs decreases the performance for all models yet still allows accuracies in high 80s (except kNN). Finally, replacing the whole text with corresponding Treebank POS-tags led to poorer yet reasonable results above 76%, again excluding kNN. 6
With logistic regression, SVM and CNN, three models reached prediction accuracies over 90%, while kNN was found less applicable on this task. The CNN did not outperform traditional approaches in this work. However, the innumerable network structures and tunable (hyper)parameters of this model leave room for improvement. We are in line with Sanchez-Perez et al. (2017) showing that higher orders of character n-grams outperform lower orders. Combining word and character n-grams also improved results. Therefore, exploring longer character n-grams (n = 7, 8, . . . ) could extend this study.
Regarding text preprocessing, we conclude that
most changes in the raw text, despite being
useful in other NLP domains, decrease the
performance on this task. Removing stop words
reduces the accuracy measurably, this confirms the
consent in literature. In the study of A
Overall, this work demonstrated the importance of stop words, punctuation, and fusions of word and character n-grams for AA on German news articles. It further revealed the potential of POStags as meaningful features for this task.
We would like to thank Main-Post GmbH for their cooperation in finding adequate authors and free access to all articles.
This research work is supported by National Sciences and Engineering Research Council of Canada Discovery Accelerator Supplements RGPAS-2018-522708.
The fourth author is supported in part by the Canada CIFAR AI Chair program.