In our participation in the authorship verification task (small corpus), our main objective is to be able to discriminate between pairs of texts that were written by the same author (denoted “same-author”) and pairs of snippets written by different ones (“different-authors”). The paper describes a simple model that performs this task based on a Labbé similarity. As features, we employed the most frequent tokens (words, and punctuation symbols) from each author after including the most frequent ones of a given language. Such a representation strategy is based on word used frequently by a given author but not belonging to the most frequent in the English language. Evaluation based of authorship verification task with a rather small set of features shows an overall performance with the small dataset of F1= 0.705 and AUC = 0.840.
Authorship verification at CLEF PAN 2020 is the task of determining whether two
texts (or excerpts) have been written by the same author [
For the CLEF PAN 2020 (small corpus), the pairs of texts have been extracted from
the website www.fanfiction.net storing texts about numerous well-known
novels, movies or TV series (e.g., Harry Potter, Twilight, Bible)[
Thus for the proposed task, the question is to identify whether or not a pair of text excerpts have been authored by the same person. We view this author verification as a similarity detection problem, or to detect when a similarity between two texts is too high to reflect two distinct authors.
Just like in authorship attribution, the author of a given text had to be revealed by
identifying some of his/her stylistic idiosyncrasies and to measure the similarity
between two author’s profile. [
As possible application of author verification, one can mention analysis of
anonymous emails for forensic investigations [
The rest of this paper is organized as follows. Section 2 describes the text datasets while Section 3 describes the features used for the classification. Section 4 explains the similarity measure and Section 5 depicts some of our evaluation results. A conclusion draws the main findings of our experiments.
The corpus contains a set of pairs composed of two short texts (or snippets) describing proposed variants or continuation of a series achieving a popular success. In this context, given two snippets, the task is to determine whether the text pair has been written by a single author or by two distinct writers. This question is a stylistic similarity detection problem assuming that two snippets could cover distinct topics with very distinctive characters and temporal differences but still being written by the same person.
These pairs of texts have been extracted from different domains (fandoms) and Table 1 reports some examples of such fandoms with the number of texts extracted from them.
('G-Gundam', 56), ('Vampire Knight', 175), ('Free! - Iwatobi Swim Club', 121), ('DC Superheroes', 98), ('Friends', 111), ('CSI: Miami', 133), ('Grimm', 108), ('Danny Phantom', 200), ('Primeval', 117), ('Kingdom Hearts', 219), ('Jurassic Park', 107), ('Tarzan', 40), ('Dungeons and Dragons', 73), ('Final Fantasy X', 152), ('Fast and the Furious', 112), ('OZ', 33), ('Sons of Anarchy', 115), ('Avatar: Last Airbender', 223), ('Attack on Titan', 194), ('Madam Secretary', 47)
This corpus contains 52,590 text pairs (denoted problems) from which 27,823 pairs corresponding to the same author and 24,767 are pairs written by two distinct persons. Each text excerpt contains, in mean, 2,200 word-tokens. An example of a pair is provided in Table 2 with their respective length and their vocabulary size.
Guardians of Ga'Hoole Tokens = 2,235 | Voc |= 1,353 I shift a bit, warily letting my eyes dart from one owl to the other -- but my eyes are trained on the Barn Owl the most. Like Hoole...so like Hoole... He turns a bit, and our eyes meet directly. I can"t describe it...in this next moment, I don"t look away, how awkward it seems. I stare into his eyes. They"re like Hoole"s... They are Barn Owl eyes, but Hoole"s eyes. They"re his eyes...Hoole"s eyes... They hold that light of valor, … Hetalia - Axis Powers Tokens = 2,032 | Voc |= 1,422 "All will become one with Russia," he said, almost simply, his cheer eerie. Fists were already clenched; now they groped about, for a pan, a rifle, a sword-there was nothing. In some way, this brought her but a sigh of relief-Gilbert and Roderich, she was reminded, were not here to suffer as well. If Ivan put his giant hands on Roderich... Click, went an object, and Elizaveta was snapped into the world when her own instincts …
To determine the authorship of the two text chunks, we need to specify a text representation that can characterize the stylistic idiosyncrasies of each possible author. As a simple first solution, and knowing that only a small text size is available, we will focus on the most frequent word-types.
To generate a text representation, a tokenization must be defined. In this study, a
token is a sequence of letters delimited by space or punctuation symbols. We also
consider as token the punctuation marks (or sequence of them) such as comma, full
stop, or question marks. Words appearing the nltk stopword list are included in this
representation (179 entries composed of pronouns, determiners, auxiliary verb forms,
some conjunctions and prepositions). Thus our strategy is based on word-types used
recurrently by one author but not frequent in the underlying language (English in this
case). One can compare this solution to the Zeta model [
Then to determine the most frequent ones, the occurrence frequency (or term frequency denoted tf) of each word-type inside a chunk of text is computed. However, as each text chunk has a different size and we opt for a relative term frequency (rtf) computed as the term frequency divided by the text size. One can interpret this rtf value as an estimation of the probability of occurrence of the term in the underlying snippet. Finally, each pair is represented by the rtf of the k most frequent word-types, with k varying from 100 to 400.
Based on a vector of k elements reflecting the rtf of each selected word-type, the
similarity (or distance) between the two text excerpts can be computed. In this study,
we opt for the Labbé similarity [
Denoting by d1 and d2 two document vectors, the Labbé distance corresponds to the ratio of the absolute difference of all n terms to the to the maximal distance between the two text representations as shown in equation 1.
Dist Labbé ( 1, 2) = ∑ =1 ,1− ,2 2∗ 2 ℎ ,1 = ,1 ∗ 2 1 (with δ = 0.5):
The decision rule is based on the value of Labbé similarity, which is (1 - Dist Labbé)
if Sim Labbé ( 1, 2) > 0.5
The implementation considers the absolute difference of all n terms in the text representations. For each term in the document, the difference between the absolute frequencies in Text 1
and 2 is computed. This requires both documents to have equal length. To ensure that both text have similar length, assuming Text 1 ( 1) is larger than Text 2 ( 2 ), we multiply the relative term frequency of Text 1( ,1) with the ratio of the two lengths 1
and 2 as shown in Equation 1.
During the PAN CLEF 2020 author verification task, the system must return a value between 0.0 and 1.0 for each problem. In our case, the Labbé similarity score provide this value. In addition, we must specify "same-author", "different-authors" or provide a blank answer (meaning " I don't know") that will be considered as an unanswered question during the evaluation. Specify δ = 0.5 (see Equation 2), we ignore this last possible answer and we provide an answer to all problems. 5
As a performance measure, four evaluation indicators have been used. First, the AUC (area under the curve) is computed. This value corresponds to area under the curve generated according to the percentage of false positives (or false alarms) in the xaxis and the percentage of true positive cases in the y-axis over the entire test set. A model whose predictions are 100%
wrong obtains an AUC of 0.0; one whose predictions are 100% correct has an AUC of 1.0.
F1 combines the
precision and the recall into a unique value. In this computation, the non-answers are
ignored. Third, c@1 is a variant of the conventional F1 score which rewards systems
leaving difficult problems unanswered. It takes into account both the number of correct
(1)
(2)
answers and the number of problems left unsolved [
The entire system was based on only the training set, so the training and the evaluation was done directly on the same corpus. With 52,590 problems in the ground truth, the results of the similarity verification are shown in Table 3.
k 100 150 200 250 300 350 400 450 500
AUC 0.847 0.851 0.854 0.855 0.857 0.858 0.859 0.530 0.535 0.530 0.530 0.530
Increasing the number of features from 100 to 500 does not have a significant impact on the overall results as shown in Table 3. On the other hand, the run time is clearly increasing.
Figure 1 : Distribution of the similarity values for the two classes,
same author or distinct authors (k=100)
To have a better view of the results, Figure 1 shows the distribution of the Labbé similarity values for the two classes, namely "same -author" and "different authors" (k=100). As one can see, the "same-author" distribution (in blue) presents a higher (mean: 0.723, sd: 0.041) and the distribution is more on the right (higher value) compared to the "different authors" distribution (mean: 0.660, sd: 0.048) and shown in red. However, the intersection between the two distribution is relatively large.
In a last experiment, instead of building the text representation with all possible word-types, we remove the 179 most frequent word appearing in the nltk stopword list. Table 4 reports the overall performance of both approaches with k=500. Depending on the evaluation measure, one representation strategy tends to propose the best effectiveness. The results are thus inconclusive.
Without AUC 0.860 0.531 0.591 F_0.5_u 0.585 F_0.5_u 0.599
F1 0.705 overall 0.672
Due to time constraint, this report proposes a simple text similarity technique to solve the authorship verification problem when facing with pairs of snippets. We proposed to select features by ranking them according to their frequency of occurrence in each text and taking only the most frequent ones (from 100 to 500) but in including the most frequent ones in the underlying language. With this proposed strategy, we want to identify terms occurring frequently by an author and frequent in the current language (English in this study).
The similarity computation is based on the Labbé between two vectors. The next
step for us is to explore the reverse text representation, taking account only of the most
frequent terms of a given language [