This paper describes the approach we took to create a machine learning model for the PAN 2020 Authorship Veri cation Task. For each document pair, we extracted stylometric features from the documents and used the absolute di erence between the feature vectors as input to our classi er. We created two models: a Logistic Regression Model trained on a small dataset, and a Neural Network based model trained on the large dataset. These models achieved AUCs of 0:939 and 0:953 on the small and large datasets, making them the second-best models on both datasets submitted to the shared task.
This paper presents our approach for the Authorship Veri cation Shared Task
at PAN 2020 [6]. The objective of this task was to create an approach that would
be able to predict if two given documents were written by the same person. The
dataset provided for this task was compiled by Bischo et al. [
This paper is structures as follows: in Section 2 we will describe our approach and in Section 3 we will present our results of the shared task. Section 4 discusses our conclusions and future work.
This section describes the approach we took to build two models (trained on
the smaller and larger datasets) for authorship veri cation. The pre-processing
and feature extraction processes were identical for both models. We used a
Linear Regression classi er for the smaller dataset and a Neural Network for
the larger dataset. Our approach was implemented on Python with NLTK [
The PAN 2020 shared task was to predict if two given documents (Di and Dj) were written by the same person. We modeled this as a binary classi cation problem, in which the input to our classi er is a feature vector encoding the two documents (X) and the target variable (Y ) indicating whether or not the two documents were written by the same author.
Preprocessing Each document in the dataset was run through a series of preprocessing steps prior to feature extraction. We will use the following sentence as a running example in this section:
\The Soviets had already been merciless, ruthless as the next army." Tokenizer: We used the NLTK Treebank Word Tokenizer with the default parameters. The tokenized version of the document is stored to be used in the next pre-processing steps and to be used in feature extraction steps.
Part of Speech (POS) Tagging We trained a Brill Tagger1 using the script provided by Jacob Perkins2. A Brill Tagger uses a combination of simpler taggers provided by NLTK to assign initial tags to a text and then applies a set of transformational rules to x incorrect tags. We opted to use this method, which is slightly less accurate than NLTK's default Perceptron based POS-tagger, due to the Brill Tagger's signi cant performance gain. In our preliminary analysis, we realized that a signi cant amount of time was spent on the POS-tagging phase of our pipeline. The following would be the output of our POS-tagger for the example sentence above: [('The', 'DT'), ('Soviets', 'NNPS'), ('had', 'VBD'), ('already', 'RB'), ('been', 'VBN'), ('merciless', 'NN'), (',', ','), ('ruthless', 'NN'), ('as', 'IN'), ('the', 'DT'), ('next', 'JJ'), ('army', 'NNP'), ('.', '.')] 1 https://www.nltk.org/_modules/nltk/tag/brill.html 2 https://github.com/japerk/nltk-trainer Generating Parse Tree (POS Tag Chunking) We used NLTK's Regex Parser to parse POS-tags and generate a parse tree from documents. We designed regular expression rules that would identify Noun Phrases and Verb Phrases given a sequence of POS-tags. While we could have used a machine-learning-based parser, which would have been slightly more accurate, we opted to use the simpler regular-expression-based parser due to performance concerns. The following would be the output of our parser for the example sentence above: (S (NP The/DT Soviets/NNPS) (VP had/VBD already/RB been/VBN) (NP merciless/NN) ,/, (NP ruthless/NN) as/IN (NP the/DT next/JJ army/NNP)) 2.2
This section lists the features that we extract from the preprocessed data. These
features are commonly used in most previous stylometry work [13]. We used some features
that are described in Writeprints feature set [
Several of our features described below are computed in terms of TFIDF values. We used NLTK's TFIDFVectorizer to compute the TF-IDF vectors for the documents. We set the min df parameter to be 0:1 in order to ignore tokens that have a document frequency less than 10%.
{ Character n-grams: TF-IDF values for character n-grams, where 1 n 6 { POS-Tag n-grams: TF-IDF value of POS-Tag tri grams. { Special Characters: TF-IDF values for 31 pre-de ned special characters. { Frequency of Function Words: Frequencies of 179 stopwords de ned in the
NLTK corpus package. { Number of characters: The total number of characters in the document. { Number of words: The total number of tokens in the document. { Average number of characters per word: The average number of characters per document. { Distribution of word-lengths (1-10): The fraction of tokens tokens of length l, wehre 1 l 10 { Vocab Richness: The ratio of hapax-legomenon and dis-legomenon. (Divided by the number of tokens in the document to account for documents of varying lengths). Here, hapax-legomenon is the number of words that only occur once in the document and dis-legomenon is the number of words that occur twice in the document. { POS-Tag Chunks: TF-IDF values for Tri-grams of POS-Tag chunks. Here, we consider the tokens at second level of our parse tree. For example, for the sentence above, the input to our vectorizer would be ['NP', 'VP', 'NP', ',', 'NP', 'IN', 'NP', '.']. { NP and VP construction: TF-IDF values of each noun phrase of verb phrase expansion. For the sentence above, these expansions are ['NP[DT NNPS]', 'VP[VBD RB VBN]', 'NP[NN]', 'NP[NN]', 'NP[DT JJ NNP]']
The features for each document were scaled. We used the absolute di erence of feature vectors of each document as input to our classi er. Speci cally, we took the feature vector for the documents Di and Dj to be Xi and Xj. The feature vector used by our classi er was X = jXi Xjj 2.3
We computed the features for each document pair in the two datasets (smaller and larger) as described in the previous section. Each dataset was randomly divided into three sets: train (70%), validation (15%), and test(15%). The training set was used to train the feature vectorizers and the classi ers, the validation set was used for model selection and parameter tuning and the test set was used to measure performance before submitting our model to PAN 2020 organizers for nal evaluation.
We trained a Logistic Regression classi er using the features from the smaller dataset. The validation dataset is used to tune model parameters. We used a Neural Network with hidden layer of size 100 for the larger dataset and the model that achieved the highest AUC on the validation set, over 100 epochs was selected as the nal model. 3
Once the nal models were trained, we deployed these models to the TIRA evaluation
system [11] provided by the PAN 2020 organizers where the models were evaluated
on an unseen dataset. They were evaluated on 5 measures: area under the ROC curve
(AUC), F1-score, c@1 (a variant of the F1-score, which rewards systems that leave
di cult problems unanswered [10]), and F 0.5u (a measure that puts more emphasis
on deciding same-author cases correctly [
In this paper we presented the approach we took in designing machine learning models for authorship veri cation. Our approach involves extracting stylometric features from a given document pair, taking the absolute di erence (or the L1 distance) of the feature 3 https://pan.webis.de/clef20/pan20-web/author-identification.html\ #results vectors of the document pair and using the resulting vector as input to a machine learning model. This approach allows us to use features that were used in authorshipattribution problems and use them in an authorship-veri cation setting. Most machine learning models that solve authorship attribution problems are author-speci c, i.e., the models are trained on a known set of authors. Authorship veri cation problems{ and our proposed solution{create a machine learning model that is generic, and thus applicable to any two given documents, even when a speci c author is not known. While this particular problem set is closed-world, based on our preliminary results from other open-world datasets, we believe that our approach would generalize well for open-world scenarios.
As future work, we would like to optimize our model for the rest or the evaluation metrics. We believe it is possible to optimize for the c@1 score by taking classi er con dence into account. We would also like to perform a feature analysis of our models to see which features become important in determining if two documents are written by the same person. 5
We thank PAN2020 organizers for organizing the shared task and helping us through the submission process. We also thank the reviewers for their helpful comments and feedback. Our work was supported by the National Science Foundation under grant 1931005.