The PAN 2021 authorship verification (AV) challenge is part of a three-year strategy, moving from a cross-topic/closed-set AV task to a cross-topic/open-set AV task over a collection of fanfiction texts. In this work, we present a novel hybrid neural-probabilistic framework that is designed to tackle the challenges of the 2021 task. Our system is based on our 2020 winning submission, with updates to significantly reduce sensitivities to topical variations and to further improve the system's calibration by means of an uncertainty adaptation layer. Our framework additionally includes an out-of-distribution detector (O2D2) for defining non-responses. Our proposed system outperformed all other systems that participated in the PAN 2021 AV task.
Deep Metric
Learning Pair of documents
Linguistic Embedding Vectors (LEVs)
Bayes Factor
Scoring Uncertainty Modeling
Posterior 0.3 probabilities 0.7 Confusion matrix 0.8 0.1 0.2 0.9 Out-of-Distribution Detector (O2D2) 0.1 0.9
Non-response? Uncertainty Adaptation
Calibrated posterior probabilities 0.31 0.69
Final Decision 0.69 or 0.5 the fact that the test dataset, which is not publicly available, only contains trials from a subset of the authors and fandoms provided in the training data.
To increase the level of dificulty, the current PAN AV challenge moved from a closed-set
task to an open-set task in 2021, while the training dataset is identical to that of the previous
year [
The overall structure of our revised system1 is shown in Fig. 1. It expands our winning system
from 2020 as follows: Suppose we have a pair of documents 1 and 2 with an associated
ground-truth hypothesis ℋ for ∈ {0, 1}. The value of indicates, whether the two documents
were written by the same author ( = 1) or by diferent authors ( = 0). Our task can formally
be expressed as a mapping :{1, 2} →− ∈ [
In [
For the decision whether to accept ℋ0/ℋ1, or to return a non-response, i.e. ℋ2, it is desirable
that the value of the posterior reliably reflects the uncertainty of the decision-making process.
We may roughly distinguish two diferent types of uncertainty [
Additionally, epistemic or model uncertainty characterizes uncertainty in the model
parameters. Examples are unseen authors or topics. Epistemic uncertainty can be reduced through a
substantial increase in the amount of training data, i.e. an increase in the number of training
pairs. We capture epistemic uncertainty in our work through the proposed O2D2 approach
and also by extending our model to an ensemble. We expect all models to behave similarly for
known authors or topics, but the output predictions may be widely dispersed for pairs under
covariate shift [
The training procedure consists of two stages: In the first stage, we simultaneously train the DML, BFS and UAL components. In the second stage, we learn the parameters of the O2D2 model.
The text preprocessing strategies, including tokenization and pair re-sampling, are
comprehensively described in [
Each document pair is characterized by a tuple (, ), where ∈ {0, 1} denotes the authorship similarity label and ∈ {0, 1} describes the equivalent for the fandom. We assign each document pair to one of the following author-fandom subsets2 SA_SF, SA_DF, DA_SF, and DA_DF given its label tuple (, ).
As shown in [
49,654 docs 39,547 authors 200 fandoms
46,373 docs
37,883 authors
200 fandoms
• The training set is identical to the one used in [
It thus does not represent a union of the calibration and validation sets. • Finally, the PAN 2021 evaluation set, which is not publicly available, has been used to test our submission and to compare it with the proposed frameworks of all other participants.
Note that both, the validation and development set in Table 1 only contain SA_DF and DA_SF pairs, for reasons discussed in Section 5. The pairs of these sets are sampled once and then kept ifxed.
In this section, we briefly describe all components of our neural-probabilistic model. Sections 4.1
through 4.4 repeat information that is already provided in [
Feature extraction and deep metric learning are realized in the form of a Siamese network, feeding both input documents through exactly the same function.
The system passes token and character embeddings into a two-tiered bidirectional LSTM network with attentions,
= NeuralFeatureExtraction (︀ , )︀ ,
where contains all trainable parameters, represents word embeddings and represents
character embeddings. A comprehensive description is given in [
We feed the document embeddings in Eq. (1) into a metric learning layer, = tanh (︀ DML+
DML)︀ , which yields the two LEVs 1 and 2 via the trainable parameters = { DML, DML}.
We then compute the Euclidean distance between both LEVs, (1, 2) = ‖1 − 2‖22 . In [
DML(ℋ1|1, 2) = exp (︀ − (1, 2) )︀ , where and can be seen as both, hyper-parameters or trainable variables. The loss then is given by
DML = · max {︀ − DML(ℋ1|1, 2), 0}︀ 2 + (1 − ) · max {DML(ℋ1|1, 2) − , 0}2 , ℒ , (3) and incorporate Eq. (5) into the binary cross entropy,
BFS = · log {BFS(ℋ1|1, 2)} + (1 − ) · log {1 − BFS(ℋ1|1, 2)} , ℒ where all trainable parameters are denoted with = {︀ BFS, BFS, , , }︀ . (1) (2) (5) (6) where we set = 0.91 and = 0.09.
We assume that the LEVs stem from a Gaussian generative model that can be decomposed as
= + , where characterizes a noise term. We assume that the writing characteristics
of the author lie in a latent stylistic variable . The probability density functions for and
are modeled as Gaussian distributions. We outlined in [
(1, 2|ℋ1)
(1, 2|ℋ1) + (1, 2|ℋ0)
= Sigmoid(︀ score(1, 2))︀
(4)
We reduce the dimension of the LEVs via BFS = tanh(︀ BFS + BFS)︀ to ensure numerically
stable inversions of the matrices [
BFS(ℋ1|1, 2) = Sigmoid(︀ score(1BFS, 2BFS))︀
as follows Now, we treat the posteriors of the BFS component as noisy outcomes and rewrite Eq. (5) as BFS( ℋ̂︀1|1, 2) to emphasize that this represents an estimated posterior. We firstly have to ifnd a single representation for both LEVs, which is done by UAL = tanh (︀ UAL(︀ 1 − 2)︀ ∘ 2 + UAL)︀ , where (· )∘ 2 denotes the element-wise square. Next, we compute a 2 × 2 confusion matrix (ℋ | ℋ̂︀, 1, 2) = for , ∈ {0, 1}.
∑︀ ′∈{0,1} exp (︀ BFS +
︀) exp (︀ ′ BFS + ′ ︀) The term (ℋ | ℋ̂︀, 1, 2) defines the conditional probability of the true hypothesis ℋ given the hypothesis ℋ̂︀ assigned by the BFS. We can then define the final output predictions as: UAL(ℋ |1, 2) =
∑︁ (ℋ | ℋ̂︀, 1, 2) · BFS( ℋ̂︀|1, 2).
The loss consists of two terms, the negative log-likelihood of the ground-truth hypothesis and a regularization term, ℒ
UAL = − log UAL(ℋ |1, 2) + ∑︁
∑︁ (ℋ | ℋ̂︀, 1, 2) · log (ℋ | ℋ̂︀, 1, 2), ∈{0,1}
∈{0,1} ∈{0,1}
with trainable parameters denoted by = {︀ UAL, UAL, , |, ∈ {0, 1}}︀ . The
regularization term, controlled by , follows the maximum entropy principle to penalize the confusion
matrix for returning over-confident posteriors [
All components are optimized independently w.r.t. the following combined loss: ℒ , ,, = ℒ ,
DML + ℒ
BFS + ℒ
UAL.
(7) (8) (9) (10) (11) (12)
Following [
̂︀ = arg max [︀ UAL(ℋ0|1, 2), UAL(ℋ1|1, 2)]︀ .
Now, we can define the binary O2D2 labels as follows:
0, otherwise. O2D2 =
{︃1, if ̸= ̂︀ or 0.5 − ≤ UAL(ℋ1|1, 2) ≤ 0.5 + , The model-dependent hyper-parameter ∈ [0.05, 0.15] is optimized on the validation set w.r.t the PAN 2021 metrics. The input of O2D2, noted as O2D2, is a concatenated vector of the LEVs, i.e. (︀ 1 − 2)︀ ∘ 2 and (︀ 1 + 2)︀ ∘ 2, and the confusion matrix. This vector is fed into a three-layer
All trainable parameters are summarized in Γ = {︀ O2D2, O2D2| ∈ {1, 2, 3}}︀ . The obtained prediction for hypothesis ℋ2 is inserted into the cross-entropy loss,
O2D2 = O2D2 · log {O2D2(ℋ2|1, 2)} + (1 − O2D2) · log {1 − O2D2(ℋ2|1, 2)} . (14) ℒΓ
As a last step, an ensemble is constructed from trained models, ℳ1, . . . , ℳ , with being an odd number. Since all models are randomly initialized and trained on diferent re-sampled pairs in each epoch, we expect to obtain a slightly diferent set of weights/biases, which in turn produces diferent posteriors, especially for pairs under covariate shift. We propose a majority voting for the non-responses. More precisely, the ensemble returns a non-response, if
Authorship Veri cation Fandom Veri cation (13) (15) (16) ∑︁ 1[︀ O2D2(ℋ2|1, 2, ℳ) ≥ 0.5]︀ > =1 ︂⌊ ⌋︂ 2 where 1[· ] denotes the indicator function. Otherwise, we define a subset of confident models, ℳ = {ℳ| O2D2(ℋ2|1, 2, ℳ) < 0.5}, and return the averaged posteriors of its elements, 1 E︀[ UAL(ℋ1|1, 2)]︀ =
∑︁ UAL(ℋ1|1, 2, ℳ).
|ℳ| ℳ∈ℳ Our submitted system consisted of an ensemble with = 21 trained models.
The PAN evaluation metrics and procedure are described in [
Inspired by the promising results in domain-adversarial training of neural networks in [
We first evaluated the UAL component on the calibration set (without non-responses) and calculated the respective PAN metrics for diferent combinations of the author-fandom subsets. Results are shown in Table 2. To guarantee that the calculated metrics are not biased by an imbalanced dataset, we reduced the number of pairs to the smallest number of pairs of all subsets. Thus, all results in Table 2 were computed from 2 × 2, 100 pairs. Unsurprisingly, best performance was obtained for the least challenging SA_SF + DA_DF pairs and the worst performance was seen for the most challenging SA_DF + DA_SF pairs. We continued to optimize our system w.r.t this most challenging subset combination in particular, even though we specifically expect to see SA_DF + DA_DF pairs in the PAN 2021 evaluation set.
Table 3 additionally provides the corresponding calibration metrics. Analogously to the PAN metrics, the ECE consistently increases from the least to the most challenging data scenarios. Interestingly, our system is under-confident for SA_SF pairs, i.e. conf < acc. The predictions then change to be over-confident (conf > acc) for SA_DF pairs.
Next, we separately provide experimental results for all system components on the validation dataset, since O2D2 has been trained on the calibration dataset. The first four rows in Tables 4 and 5 summarize the PAN metrics and the corresponding calibration measures averaged over all ensembles models.
The overall score of the UAL component in the third row of Table 4 is on par with the DML and BFS components and slightly lower compared to the corresponding UAL score measured on the calibration dataset in Table 2. Nevertheless, we do not observe significant diferences in the metrics for both datasets, which shows the robustness and generalization of our system.
Going from the third to the fourth row in Table 4, it can be observed that the overall score, boosted by c@1 and F1, significantly increases from 92.5 to 93.2. Hence, the model performs better if we take undecidable trials into account. However, the f_05_u score decreases, since it treats non-responses as false negatives. The percentage of undecidable trials generally ranges 1.00
0.4 0.6 Output predictions 0.8 from 8% to 11%.
In Table 5, we see that both, the BFS and UAL components notably improve the ECE and MCE metrics. However, an insertion of non-responses via O2D2 significantly increases the MCE. This can be explained by the posterior histograms in Fig. 4. The plots (a) and (b) show the histograms for SA_DF and DA_SF pairs without applying O2D2 to define non-responses. In contrast, plots (c) and (d) present the corresponding histograms including the 0.5-values of non-responses. The efect of O2D2 is that most of the trials, whose posteriors fall within the interval [0.3, 0.7], are eventually declared as undecidable. Hence, the system correctly predicts nearly all of the remaining as confidently assigned trials around 0.7/0.8 for same-author pairs or 0.2/0.3 for diferent-author pairs. As a result, we see a large gap (i.e. conf << acc) between the confidence score and the averaged accuracy in these bins.
The last two rows in Tables 4 and 5 show the performance of the ensemble, first without and then with non-responses, to show the efect of O2D2. On the validation set, our ensemble with O2D2 returns non-responses in 9% of the test cases. Comparing the last two rows, we obtain the highest overall score with our proposed framework, which ultimately presents our final submission.
To conclude this section, we present our results on the oficial PAN 2021 evaluation set. The performance for both, the early-bird and the final submission, can be found in Table 6. We also provide the reported result on the PAN 2020 evaluation set for the predecessor model.
Unsurprisingly, the early-bird overall score (single model) on the PAN 2021 evaluation set is slightly higher, since it contains DA_DF pairs instead of DA_SF pairs. The main diference is, unexpectedly, given by the f_05_u score, which increases from 89.3% to 94.6%. In our opinion, this is caused by returning a lower number of non-responses, which would also explain the lower values for c@1 and F1.
Comparing the early-bird (2 row) with the final submission ( 4ℎ row), we can further significantly increase the overall score by 1.5%. We assume that the ensemble now returns a higher number of non-responses, which results in a slightly lower f_05_u score. Conversely, we can observe improved values for the c@1, F1 and brier scores.
The last row displays the achieved PAN 2020 results. As can be seen, our final submission ends up with a higher overall score (plus 2%) by significantly improving all single metrics, although the PAN competition moved from a closed-set to open-set shared task, illustrating the eficiency of the proposed extensions.
In this work, we presented O2D2, which captures undecidable trials and supports our hybrid neural-probabilistic end-to-end framework for authorship verification. We made use of the early-bird submission to receive a preliminary assessment of how the framework behaves on the novel open-set evaluation. Finally, based on the presented results, we submitted an O2D2-supported ensemble to the shared task, which clearly outperformed our own system from 2020 as well as the new submissions to the PAN 2021 AV task.
These results support our hypothesis that modeling aleatoric and epistemic uncertainty and using them for decision support is a beneficial strategy—not just for responsible ML, which needs to be aware of the reliability of its proposed decisions, but also, importantly, for achieving optimal performance in real-life settings, where distributional shift is almost always hard to avoid.
This work was in significant parts performed on an HPC cluster at Bucknell University through the support of the National Science Foundation, Grant Number 1659397. Project funding was provided by the state of North Rhine-Westphalia within the Research Training Group "SecHuman - Security for Humans in Cyberspace" and by the Deutsche Forschungsgemeinschaft (DFG) under Germany’s Excellence Strategy - EXC2092CaSa- 390781972.