This paper describes and evaluates an unsupervised authorship verification model called SPATIUM-L1. The suggested strategy can be adapted without any problem to different languages (such as Dutch, English, Greek, and Spanish) with their genre and topic differ significantly. As features, we suggest using the k most frequent terms of the disputed text (isolated words and punctuation symbols with k may vary from 200 to 300). Applying a simple distance measure and a set of impostors, we determine whether or not the disputed text was written by the proposed author. Moreover, based on a simple rule, we can define when there is enough evidence to propose an answer with a high degree of confidence or when the attribution scheme is given without certainty. The evaluations are based on four test collections (PAN AUTHOR IDENTIFICATION task at CLEF 2015).
Automatic authorship identification aims to determine, as accurately as possible, if
the proposed author of a document or a text excerpt is the real one [
This authorship verification question seems simpler than the classical authorship
attribution problem, but it is not. For example, if we want to know if a newly
discovered poem was really written by Shakespeare [
This paper is organized as follows. The next section presents the test collections and the evaluation methodology used in the experiments. The third section explains our proposed algorithm called SPATIUM-L1. In the last section, we evaluate the proposed scheme and compare it to the best performing schemes using four different test collections. A conclusion draws the main findings of this study.
The experiments supporting previous studies were usually limited to one language,
one author, and one or a few texts. For real cases, this limitation makes sense; for
example we have only one newly discovered poem that might be attributed
to Shakespeare [
During the PAN CLEF 2015 evaluation campaign, four test collections were built, each containing at least 200 problems (training + testing). In each collection, all the texts matched the same language but can be cross-topic or cross-genre and may differ significantly. In this context, a problem is defined as:
Given a small set of “known” documents (no more than seven, possibly as few as one) written by a single person, is the new “unknown” document also written by that author?
The four benchmarks are composed of a Dutch and Spanish cross-genre collection and an English and Greek cross-topic corpus. An overview of these collections is depicted in Table 1. The training set will be used to evaluate our approach and the test set will be used in order to be able to compare our results with those of the PAN CLEF 2015 campaign.
Type PrNoboleomfs doMcuemanent wMoeradns cross-genre 100 1.8 449 cross-topic 100 1.0 341 cross-topic 100 2.9 688 cross-genre 100 4.0 976 Table 1. PAN CLEF 2015 corpora statistics
No of Problems 165 500 100 100
The number of problems is given under the label “No of Problems”. The mean number of known documents for each problem is indicated in the column “Mean document”, and the mean number of words per known document under the label “Mean words”. The mean number of words in the unknown documents is close to the latter. The last two metrics are not available for the test corpora because the datasets remained undisclosed thanks to the TIRA system.
When inspecting the English training collection, the number of words available is rather small (in mean 341 words for each document, and exactly one document per problem). Similarly the Dutch collection only provides 808 words in mean per problem (in mean 449 words for each document, and 1.8 documents in each problem). This collection has mostly one or two document per problem but also some with 5, 6 or even 7 known documents. Therefore, we can expect the mean performance for these languages to be lower than for the other languages under the assumption that all languages present the same level of complexity to solve this problem. For the Spanish corpus we have always four documents and rather long ones to learn the stylistic features of the proposed author. A relatively higher performance can be assumed with this benchmark. A similar conclusion can be expected with the Greek collection consisting of longer documents (in mean, 1,995 words).
When considering the four benchmarks as a whole, we have 865 problems to solve and 400 to train (pre-evaluate) our system. When inspecting the distribution of the correct answers, we can find the same number (432 in test and 200 in training) as positive or negative answers. In each of the individual test collections, we can also find a balanced number of positive and negative answers.
During the PAN CLEF 2015 campaign, a system must return a value between 0.0 and 1.0 for each problem with a precision down to a thousandth. A value strictly larger than 0.5 indicates that the query text was written by the proposed author and a value strictly lower than 0.5 the opposite. Returning the value 0.5 indicates that the system is unable to take a decision based on the given information. Of course, a value closer to 1.0 (or to 0.0) is a stronger evidence in favor of (or against) a positive answer.
As performance measure, two evaluation measures were used during the PAN CLEF
campaign. The first performance measure is the AUC (Area Under the Curve) of the
ROC (Receiver Operating Characteristic) curve [
As another measure, the PAN CLEF campaign adopts the c@1 measure [
To solve the verification problem, we suggest an unsupervised approach based on a
simple feature extraction and distance metric called SPATIUM-L1 (Latin word meaning
distance). The selected stylistic features correspond to the top k most frequent terms
(isolated words without stemming but with the punctuation symbols). For determining
the value of k, previous studies have shown that a value between 200 and 300 tends to
provide the best performance [
short and we further excluded the words only appearing once in the text.
filtering decision was taken to prevent overfitting to single occurrences. The effective number of terms k was set to at most 200 terms but was in most cases well below. With this reduced number the justification of the decision will be simpler to understand because it will be based on words instead of letters, bigrams of letters or combinations of several representation schemes or distance measures.
In the current study, a verification problem is defined as a query text, denoted Q, and a set of texts (between 1 and 7) written by the same proposed author. The concatenation of these texts forms the author profile A. To measure the distance between Q and A, SPATIUM-L1 uses the L1-norm as follows: ∆(, ) = ∆0= ∑ =1 | [ ] − [ ]|
where k indicates the number of terms (words or punctuation symbols), and PQ[ti] and PA[ti] represent the estimated occurrence probability of the term ti in the query text Q and in the author profile A respectively. To estimate these probabilities, we divide the term occurrence frequency (tfi) by the length in tokens of the corresponding text (n), Prob[ti] = tfi / n, without smoothing and therefore accepting a 0.0 probability.
To verify whether the resulting ∆0 value is small or rather large, we need to select a set of impostors. To achieve this, three profiles from other problems in the test set were chosen randomly with preference to candidates that show the same number of known documents. This value of three is arbitrary and will be denoted by the variable m. After computing the distance between Q and each of these m profiles, we retain only the smallest distance.
Instead of limiting the number of possible impostors to m, we iterate this last stage r times, and we suggest to fix the value r = 5. After this last step, we have r values denoted ∆1 , …, ∆
, each of them corresponding to the minimum value of a set of m impostors. Instead of working with r values, we compute the arithmetic mean, denoted ∆ , of the sample ∆1 , …, ∆ .
Finally, the decision rule is based on the value of the ratio ∆0 / ∆ as follows: ∆ ∆ ∆0 < 0.975 ∆0 > 1.025
0 value is similar to ∆ (in the range ±2.5%), the system specifies that the solution of this problem cannot be determined with good certainty and provides the answer don’t know. On the other hand, when ∆ 0 is small compared to ∆ , the evidence is in favor of assuming that the author of the profile A is the real author. Finally, when ∆ is small compared to ∆0, we conclude that Q and A are written by different authors. The limit of two times 2.5% was chosen arbitrarily but corresponds to a well-known limit value in statistical tests.
Since our system is based on an unsupervised approach we were able to directly evaluate it using the training set. In Table 2, we have reported the same performance measure applied during the PAN CLEF campaign, namely the final score, which is the product of the AUC and the c@1.
The algorithm returns the best results for the Spanish collection with a final score of 0.5694 closely followed by the Greek corpus possibly due to the fact of the longer and numerous documents in these two languages. The worst result is achieved with the Dutch collection that shows a highly diverging number of known documents per problem. Usually the AUC values should be consistent and comparable with the c@1 values but in some cases the AUC values are a lot higher than the c@1 values (mainly in the Spanish collection but also observable in the English and Greek corpus). As possible reason we saw a few misclassifications that have very high probability scores. The AUC of the ROC is biased in a way that the ROC gives more emphasis on the first position and therefore increases the total AUC. A misclassification with a lower probability is less penalized in this performance measure.
Due to the fact that our algorithm is based on a probabilistic approach (random selection of candidates) the results in Table 2 may vary between runs. To verify the impact of this selection in the reported performance measures, Table 3 shows the standard deviation and the estimated confidence interval covering 95% of the cases for the c@1 value based on 200 restarts with random impostor selection.
c@1 Language Mean Standard deviation Interval (95%) Dutch 0.4769 0.0210 [0.4356 – 0.5181] English 0.5776 0.0237 [0.5312 – 0.6241] Greek 0.6932 0.0258 [0.6426 – 0.7437] Spanish 0.6997 0.0308 [0.6393 – 0.7601] Table 3. Variation around the c@1 performance for the SPATIUM-L1 system We can see the possible variation around the reported performance is noticeable but relatively small. Similarly when changing the values of the two numbers m (number of impostors) and r (number of iterations) then the difference of the best possible combination of the two parameters to the performance reported in Table 2 is not significant.
The test set is then used to rank the performance of all 18 participants in this task. Based on the same evaluation methodology, we achieve the results depicted in Table 4 corresponding to the 865 problems present in the four test corpora.
As we can see the final score with the Greek corpus is as high as expected from the training set. The results we achieved in the Dutch collection is as low as in the training set. On the other hand the English results are better than anticipated and the Spanish score is worse than the estimation from the training set. It seems like the system performs better on the two cross-topic corpora (English and Greek) than on the two cross-genre corpora (Dutch and Spanish).
Language Dutch English Greek Spanish
To put those values in perspective we can see in Table 5 our result in comparison with the other 17 participants using macro-averaging. We have also added a baseline corresponding to a system that always produces the answer yes (trivial acceptor). The bad performance in the Dutch collection clearly worsens our overall results.
Another pertinent observation is the fast runtime of our system in comparison with other solutions1. The median execution time of the other systems is almost one hour. Also the runtime only shows the actual time spent to classify the test set. On TIRA there was the possibility to first train the system using the training set which had no influence on the final runtime. Since we have an unsupervised system it did not need to train any parameters, but this possibility might have been used by other participants.
In text categorization studies, we are convinced that a deeper analysis of the evaluation results is important to obtain a better understanding of the advantages and drawbacks of a suggested scheme. By just focusing on overall performance measures, we only observe a general behavior or trend without being able to acquire a better
explanation of the proposed assignment. To achieve this deeper understanding, we could analyze some problems extracted from the English corpus. Usually, the relative frequency (or probability) differences with very frequent words such as when, is, in, that, to, or it can explain the decision.
This paper proposes a simple unsupervised technique to solve the authorship
verification problem. As features to discriminate between the proposed author and
different impostors, we propose using the top 200 most frequent terms (words and
punctuations). This choice was found effective for other related tasks such
as authorship attribution [
The proposed approach tends to perform very well in two different languages
(English and Greek) on cross-topic collections and well in a Spanish cross-genre
corpus. Such a classifier strategy can be described as having a high bias but a
low variance [
It is common to fix some parameters (such as time period, size, genre, or length of the data) to minimize the possible source of variation in the corpus. However, our goal was to present a simple and unsupervised approach without many predefined arguments. This turned out to not work well on the Dutch cross-genre corpus. We suspect this to be mostly related to the genre variation than the language itself.
SPATIUM-L1 returns a numerical value (between 0 and 1) that can be used
to determine a degree of certainty [
To improve the current classifier, we will investigate the consequence of some smoothing techniques, the effect of other distance measures, and different feature selection strategies. In the latter case, we want to maintain a reduced number of terms. In a better feature selection scheme, we can take account of the underlying text genre, as for example, the most frequent use of personal pronouns in narrative texts. As another possible improvement, we can ignore specific topical terms or character names appearing frequently in an author profile, terms that can be selected in the feature set without being useful in discriminating between authors. We might also try to exploit PAN specific properties such as the requirement for equally distributed positive and negative problems. In case our system decides in over half the cases for (or against) a verification we could assign for the least certain part that it is a don’t know answer.
The author wants to thank the task coordinators for their valuable effort to promote test collections in authorship attribution. This research was supported, in part, by the NSF under Grant #200021_149665/1. 6