Authorship verification, the task of identifying if two text excerpts are from the same author, is an important part of evaluating the veracity and authenticity of writings and is one of the challenges for this year's PAN @ CLEF 2020 event. In this paper, we describe our PAN authorship verification submission system, a neural network that learns useful features for authorship verification from fanfiction texts and their corresponding fandoms. Our system uses the Longformer, a variant of state-of-the-art transformer models, that is pre-trained on large amounts of text. This model combines global self-attention and local self-attention to enable efficient processing of long text inputs (like the fanfiction data used for PAN @ CLEF 2020), and we augment the pre-trained Longformer model with additional fully-connected layers and fine-tune it to learn features that are useful for author verification. Finally, our model incorporates fandom information via the use of a multi-task loss function that optimizes for both authorship verification and topic correspondence, allowing it to learn useful fandom features for author verification indirectly. On a held-out subset of the PAN-provided “large training” set, our Longformer-based system attained a 0.963 overall verification score, outperforming the PAN text compression baseline by 32.8% relative. However, on the official PAN test set, our system attained a 0.685 overall score, underperforming the PAN text compression baseline by 7.6% relative.
As more of us rely on online sources for our news and information, it becomes
increasingly important to vet their veracity and authenticity. One key component of this
vetting process is identifying the authorship of the information. Knowing the author of
the information can help us better ascertain its trustworthiness which is critical in light
of the growing amount of online misinformation propagated by so-called trolls, bots,
and other online agitators. There has been a lot of prior work on computational and
statistical methods for determining the authorship of text writings based on writing style:
word choice, punctuation usage, idiosyncratic grammatical errors, and in more recent
digital texts the use of emoticons. One vibrant community in which these
computational approaches to authorship identification are developed and evaluated is PAN [
PAN hosts scientific evaluations for digital text forensics and stylometry, and one of
this year’s tasks in PAN @ CLEF 2020 is that of authorship verification [
PAN @ CLEF 2020 builds off of earlier evaluations in Authorship Identification
tasks [
Our approach is to use neural networks to learn text and fandom embeddings that
are useful for authorship verification. Our final submission system is built using a
variant of state-of-the-art transformer models [26] that have recently been setting the pace
on a wide variety of natural language processing tasks such as translation,
questionanswering, cloze tasks, and language modeling [
The key contributions that we will cover in our PAN Notebook paper are the
following:
1. We train our neural network models to incorporate auxiliary fandom information
using a multi-task loss function that combines both authorship and fandom
correspondence classification losses.
2. We investigate the effectiveness of large-scale text pre-training for authorship
verification by building a system using the state-of-the-art Longformer model [
Authorship verification and authorship attribution are related subfields within the larger field of Forensic Authorship Analysis. Authorship verification seeks to determine if two different writings are from the same author, while authorship attribution’s goal is to identify who wrote a given writing. Both of these fields rely on the extraction of useful text features for discriminating between different authors. Traditionally, researchers have relied on linguistic style or stylometric features, such as the counts and frequency of function words, average length of sentences, part-of-speech, characters, punctuation, whitespace usage, and other low-level features [25].
More recently, with the advent of many successful end-to-end deep learning systems
in computer vision, speech recognition, and natural language processing, researchers
have begun exploring the idea of learning what features are most useful for both
verification and attribution. Many researchers have explored using convolutional neural
networks (CNNs) and have successfully used them to extract text features [
Computational and neural network-based approaches for Natural Language
Processing (NLP) have seen a spike in the growth of their popularity mostly due to the
effectiveness of their usefulness across a wide-range of NLP tasks. Simple word-embedding
techniques like Word2Vec [
Many authorship verification systems have explored architectures other than CNNs
and Transformers. Some have sought to learn features using autoencoder-inspired
architectures from a variety of word and character n-grams and POS [
Finally, this year’s PAN baseline systems are two simple, yet effective approaches.
The “TFIDF” baseline computes the term frequency-inverse document frequency
normalized counts of character tetragrams of the pair of text excerpts, and then uses the
cosine similarities between them as an authorship verification score. The “Compress”
baseline is an adaptation of [
Our work explores neural networks as a means for extracting discriminative features directly from the text without any explicit feature engineering. Towards this end, we explore two models: Character Convolutional Neural Network (CN)2, and Longformer. More detail about each model will be given in Section 3.3 and Section 3.4.
At a high level, there are three components to our approach: text feature extraction, topic embedding, and final classification. Of these three, only the text feature extraction changes depending on which of the two models is being used. To embed the fandoms1, we apply an embedding layer Etopic that maps each fandom identifier to a 512 dimensional vector. Then, the topic embeddings are combined with the text features and passed to two multi-layer perceptrons Mauthor and Mtopic for authorship verification and topic correspondence respectively. The whole process is outlined in Figure 2 for the (CN)2 model. 3.1
Depending on which model is being used for text feature extraction, our tokenization differs: 1. (CN)2 - This model looks at the text from the standpoint of characters, including all punctuation and white-space as tokens. This model can thus focus more on the syntactic quirks of each author rather than the semantic meaning. The size of the 1 In this paper we refer to fandoms as “topics” interchangeably since the fandom of a piece of writing can roughly be considered to be its topic.
character vocabulary is 2; 500 and 4; 800 for the small and large version
respectively. In the case where a character in the validation set in unseen, we use an
“unknown” token.
2. Longformer - This model looks at the text from the standpoint of words and can
be thought of as placing more emphasis on the semantic meaning and particular
word combinations. Here, we tokenize our document using RoBERTa’s tokenizer
which has a vocabulary of size 50,265 tokens. More detail about this tokenizer can
be found in [
Because the average length of the fanfiction excerpts is about 20,000 characters long making it difficult for our models and their intermediate feature representations fit in GPU memory, we chose to use multiple random windows from each excerpt as input. The number and length of random windows varies depending on the model being used: 1. (CN)2 - Uses 10 random windows, 5 from each pair, each composed of 1000 characters. 2. Longformer - Uses 511 words from each pair, these are then separated using the special <SEP> token, and the classification token <CLS> is prepended to learn the document features that are most beneficial for authorship verification. See Figure 3 and Section 3.4 for more information.
Figure 1 shows an example of how we prepare the fanfiction documents for input to our authorship verification systems. 3.3
As mentioned previously, the (CN)2 model looks at the document from the standpoint of characters. For input, we take in a set of 10 random windows, 5 from each document. When each random window is embedded, we stack them on top of each other along the channel dimension. Then, 1-D convolutions are applied over n-grams of size one through five after which we maxpool over the length dimension. For each n-gram, there are 512 different convolutional filters, and these filters essentially learn the character n-grams that are most useful for authorship verification. Letting the output of each 1-D convolution and maxpool be called C1; C2; ; C5, where C1 is the output of the 1-D convolution and maxpool that looks at unigrams and C2; ; C5 are the ones from higher order n-grams, we define an operation we call Recursive Self Attention: out = SA(C5
SA(C4
SA(C3
SA(C2
C1)))) (1)
Where is the concatenation operation, and SA is the Self-Attention operation. This allows the network to learn features in a hierarchical manner starting from the smallest n-grams to the highest. Finally, the output of this process is passed through a fully connected layer to generate the final documents’ feature vector of size 512. The documents’ features are then concatenated with the fandom embeddings forming a composite feature vector F , which is then sent to two separate two-layer multilayer perceptrons: one for learning the probability that the two documents are from the same author and the other for the probability that they come from the same topic (i.e., the fandom of the fanfic excerpt). The system architecture is shown on Figure 2. 3.4
The Longformer [
We continue training from the weights of the RoBERTa model released by Beltagy,
et al. [
L(F; a; t) = 1 N
X BCE( (Mauthor(Fi)); ai) + BCE( (Mtopic(Fi)); ti) N i=1 (2)
Where N is the number of samples, F 2 RNx3D are the features extracted from the model, a 2 f0; 1g are the author labels, and t 2 f0; 1g are the topic labels. The label ai is 1 if the text pair is written by the same author and 0 otherwise; t is analogous to a but for topic correspondence. Given the features F , we map them to authorship verification scores and topic correspondence scores using separate two-layer multilayer perceptrons Mauthor and Mtopic respectively. These scores are then bounded between zero and one through the use of the sigmoid function .
BCE is the binary cross-entropy loss function here defined for one sample: BCE(x; y) = y log(x) (1 y) log(1 x) (3)
Where x is a probability, and y 2 f0; 1g is the label.
After training, the output of the neural networks Mauthor and Mtopic approximate the posterior probabilities of the two excerpts being from the same author and being from the same topic respectively. We use the same-author posterior probability, the output of Mauthor, as our authorship verification score and can be thresholded at 0:5 to decide whether the pair of fanfic excerpts are from the same author or not.
To evaluate our model, we used the script provided by the PAN organizers. This script
implements four metrics: AUC, F1-Score, C@1, and F_0.5u. C@1 [
F_0.5u places more importance on documents that come from the same author (i.e., true positives). This is in contrast to C@1 which treats both positive and negative examples with equal importance. Because of this, in F_0.5u the unanswered samples are treated as false negatives resulting in worse performance for models that leave a lot of samples unanswered. Finally, F_0.5u weights true positives higher compared to false positives and false negatives. Although both C@1 and F_0.5u interact with unanswered samples differently, we didn’t map any scores to 0:5 and simply used the raw sameauthor posterior probability as our verification score. 4.2
The PAN Authorship Verification 2020 datasets come in two flavors: small and large. Each dataset was built by scraping fanfiction writings from fanfiction.net. Pairs of fanfictions along with their respective topic (fandom) were provided for training with the classification of positive (same author) or negative (different authors) as ground truth. For training our models, we used both the small and large versions of the dataset with our final submission being based upon a model that was calibrated on the large dataset.
The differences between both datasets is highlighted in Table 1. There are 52; 655 samples in the small dataset and 278; 169 in the large dataset, thus the large dataset contains about 5.28 times more samples. In terms of document length, the average number of characters is approximately 21,400 and the split between negative and positive samples is close to 50% on both datasets. Finally, both the small and the large dataset share the same 1; 600 fandoms.
To validate our approaches, we split both the small and large dataset into training and validation splits. To construct these splits, we performed the following steps: 1. Separate the positive and negative samples from each set. 2. Randomly choose 70% of the negative samples for the training set, and the remaining 30% for the validation set. 3. We pick 15% of the authors randomly and use their positive samples in our validation set. If two or more positive pairs existed, we used half for our training set and the rest for the validation set. 4. The positive samples of the remaining 85% of authors are used in the training set. 4.3
Tables 2 and 3 show our evaluation results of the baseline systems, (CN)2 and Longformer, on the validation set for the small and large datasets respectively. Comparing (CN)2 and Longformer against the Compress Baseline on the small dataset (Table 2), both models see an improvement on every metric except (CN)2 on AUC. Overall, (CN)2 outperforms the baseline by 5% absolute, while the Longformer model outperforms it by 14.5%. On the large dataset (Table 3), the improvement is even more pronounced. (CN)2 achieves an improvement of 12.7% while Longformer achieves an improvement of 23.8%. Both (CN)2 and Longformer benefit from having more data available which highlights the possibility of further improving results by the addition of more data.
Given these results, we chose the Longformer system trained on the large dataset as
our submission for the TIRA evaluation system [
From the results we see that Longformer clearly outperforms both (CN)2 and the PAN baselines on our own held-out validation set. We surmise that there are three reasons why the Longformer architecture won out over (CN)2 on this held-out validation set: 1. It was pre-trained on about 6.5 billion tokens from multiple online sources. 2. Its deep architecture allows the model to learn more distant relationships of elements in the text than just n-grams. In contrast, (CN)2 is limited to exploiting relationships of n-grams of size 1 through 5. 3. Because the grammar of fanfictions are relatively clean, the word-based Longformer is more suited to the task as it exploits semantic relationships between the words and the structure of the sentences. On the other hand, (CN)2, focuses more on the syntactic features, thus we hypothesize that character-level features would be better in a dataset with shorter, more informal excerpts where words aren’t necessarily written correctly.
As of the writing of this paper, we’re still unsure as to why the Longformer model didn’t generalize on the test set as can be seen in Table 4. We conjectured that our validation set may not have sufficiently exhibited “the significant shift in the relation between authors and fandoms” in the test set from those seen during training. To test this hypothesis, we created a new dataset split such that the (author, fandom) pairs seen in the validation set are unseen in the training set. For example, if an author wrote in 5 fandoms, we used his/her writings from 4 fandoms in the train set and reserved the last for the validation set. This new validation set allowed us to test the most extreme form of (author, fandom) shifts where there is no overlap between (author, fandom) pairs between training and validation sets. Using this new train/validation data split, the Longformer model achieved an overall score of 93.6% on the validation set. While this score is less than that achieved in our previous validation set, it is still high, and thus fandom shift does not fully explain our Longformer model’s failure to generalize on the official test set. 6
In this paper, we described our approach to the authorship verification task in the PAN @ CLEF 2020 challenge based on using neural networks for learning discriminative features from the text as well as the fandom from which the text derives. We compared two different neural network architectures. The first architecture, (CN)2, uses a convolutional stack followed by a recursive self-attention stack to simultaneously learn useful character n-grams and their combinations that are most useful for determining whether the excerpt pair is from the same author. The second architecture leverages the state-ofthe-art text features learned by the Longformer model, pre-trained on text with about 6.5 billion words, and we fine-tune the last two encoder stacks to learn useful features for authorship verification. By leveraging a powerful text modeling architecture like the Longformer, we sought to investigate the effectiveness of a transfer learning approach that has shown great results for other NLP tasks. Both systems learn a separate set of embeddings for the fandoms of both fanfic excerpts. The text based features and the fandom features are concatenated and used to predict authorship and fandom correspondences. We used a multi-task loss function to simultaneously optimize for both authorship verification and topic correspondence, allowing it to learn useful fandom features for author verification indirectly.
For our validation testing, we partitioned each of the PAN provided “small” and “large” training sets into our own training and validation sets. We evaluated our (CN)2 and Longformer systems along with the two simple PAN baseline systems. Both of our systems outperformed each baseline, but our Longformer system was the winner by a wide margin, attaining a 0.963 overall verification score which is a 32.8% relative improvement over the best baseline system. Thus, we chose to submit our Longformer model trained on the “large” training set as our submission system for PAN. Unfortunately, this system failed to generalize on the official PAN test set, only attaining a 0.685 overall score and failing to outperform the baseline systems. Without access to the PAN test set, we have started to diagnose the source of the generalization failure. An initial experiment on a new train/validation split where (author, fandom) pairs do not overlap between train and validation sets has shown that an extreme shift in (author, fandom) pairs may not be the most significant cause of performance degradation. Our future work will further investigate this issue and seek to improve the Longformer system’s generalization performance. 7
This work was performed under the auspices of the U.S. Department of Energy by Lawrence Livermore National Laboratory under Contract DE-AC52-07NA27344. This document may contain research results that are experimental in nature, and neither the United States Government, any agency thereof, Lawrence Livermore National Security, LLC, nor any of their respective employees makes any warranty, express or implied, or assumes any legal liability or responsibility for the accuracy, completeness, or usefulness of any information, apparatus, product, or process disclosed, or represents that its use would not infringe privately owned rights. Reference to any specific commercial product, process, or service by trade name, trademark, manufacturer, or otherwise does not constitute or imply an endorsement or recommendation by the U.S. Government or Lawrence Livermore National Security, LLC. The views and opinions of authors expressed herein do not necessarily reflect those of the U.S. Government or Lawrence Livermore National Security, LLC and will not be used for advertising or product endorsement purposes. 23. Ruder, S., Ghaffari, P., Breslin, J.G.: Character-level and multi-channel convolutional neural networks for large-scale authorship attribution. ArXiv abs/1609.06686 (2016) 24. Shrestha, P., Sierra, S., González, F.A., y Gómez, M.M., Rosso, P., Solorio, T.:
Convolutional neural networks for authorship attribution of short texts. In: EACL (2017) 25. Stamatatos, E.: A survey of modern authorship attribution methods. Journal of the
American Society for information Science and Technology 60(3), 538–556 (2009) 26. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L.,
Polosukhin, I.: Attention is all you need. ArXiv abs/1706.03762 (2017) 27. Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., Brew, J.: Huggingface’s transformers: State-of-the-art natural language processing. ArXiv abs/1910.03771 (2019)