This article describes our approach for the Author Identification task introduced in PAN 2015. Given a set of documents written by the same author and a questioned document with an unknown author, the task is to decide whether the questioned document was written by the same author as the other documents or not. Our approach uses Random Forest and a feature-encoding scheme based on the Universal Background Model strategy, building different feature vectors that describe: 1) the complete population of authors in a dataset, 2) the known author, 3) the questioned document and combines the three of them in a single representation.
Authorship Attribution is the process of attempting to identify the likely authorship of
a given document [
The PAN 2015 Author Identification task focuses on the authorship verification problem. In this article we describe our approach for this task, using a feature-encoding scheme inspired on the Universal Background Model and applying Random Forest for prediction.
In section 2 we define the problem of authorship verification formally and introduce relevant notations. Section 3 lists and explains in detail the full set of features considered. Two baseline methods: a simple model based on distances between feature vectors and an implementation of a Gaussian Mixture Model - Universal Background Model are introduced in section 4. Section 5 describes our approach, the feature-encoding scheme and the post-processing done to the Random Forest for probabilistic classification. In Section 6 we present our results and evaluations, both on the train and test corpus. Finally in Section 7 conclusions and future research directions are exposed. 2
In this section we describe the problem of authorship verification as introduced by the
PAN 2015 Author Identification task [
Let P = (D; q) be a problem, where D is a small set of documents written by a known author and q is a questioned document whose author we do not know. The Author Identification task consists of determining whether the question document q was written by the same author who wrote the documents on set D or not. In our approach we model this as a probabilistic classification problem, where rather than only outputting the most likely class that the sample should belong to, we obtain a probabilistic output that indicates a degree of certainty between 0.0 and 1.0, corresponding to the probability of a positive answer.
The classification function f is defined as: f (D; q) = pr. Where pr is the probability that the questioned document was written by the same author. The size of D will range from 1 to 5 documents.
In this task, we have problems for four different languages: Dutch, English, Greek
and Spanish. Evaluations will be measured according to the area under the ROC curve
(AU C) of pr and the c@1 measure [
Where n refers to the number of problems being evaluated, nc refers to the number of correct answers and nu refers to the number of unanswered problems. For measuring the correctness of an answer, a binary evaluation is performed, where pr > 0:5 corresponds to a positive answer, pr < 0:5 corresponds to a negative answer and pr = 0:5 will be considered as an unanswered problem. 3
This section is devoted to the enumeration and description of the features extracted for this task. We extracted a heterogeneous set of features describing properties related to the style of the author from low level features (e.g. vocabulary diversity, document length, etc) to high level features (e.g. part-of-speech, LDA topics, etc). All of our features are expressed at the author level, considering the total set of documents D for each sample. 3.1
Structure and Extension Features – Number of tokens: minimum, average and maximum number of tokens per document, paragraph and sentence. Also, we extracted the same statistics considering a single occurrence per word, hereafter referred as unique tokens. – Number of stop words: minimum, average and maximum number of stop words (and unique stop words) per document, paragraph and sentence. We considered the stop words dictionaries provided by the Python library many-stop-words. – Number of sentences: minimum, average and maximum number of sentences per paragraph and document. – Number of paragraphs: minimum, average and maximum number of paragraphs per document. – Spacing: minimum, average and maximum number of consecutive spaces, number of consecutive spaces in the beginning/end of the line and number of empty lines. – Punctuation: minimum, average and maximum number of punctuation characters (.,;?¿!¡"’) per document, paragraph and sentence. 3.2
Distributional Features – Word distribution: Frequencies of the words contained in the documents written by the author, divided by the total number of words in them. – Character distribution: Frequencies of the alphanumeric characters in the documents written by the author divided by the total number of characters in his documents. Also, we extracted the minimum, average and maximum number of lowercase characters, uppercase characters and digits per document. – Punctuation Bigrams: Frequency of the punctuation characters bigrams observed in the author documents. 3.3
For each author, the following features are extracted independently for each document and then aggregated taking their max, min and average values.
– Lexical density: measure of how “dense” is the content, i.e, the ratio between each lexical category (nouns, adjectives, verbs and adverbs) divided by the total number of words. – Word diversity: ratio between the number of lemmas found divided by the total number of words. – Lemmas BoW: frequency of the lemmas. – Lemmas diversity: for each lemma, the number different words mapped to it. – Uniqueness: number of words that appear only one time. – Hapax: number of words that appear only one time and are only used by the current author.
The POS tags and lemmas used were extracted using the Tree Tagger provided by
Helmut Schimd [
4
– Word Topics: Closeness of a document to the K-th LDA topic. – Stop word Topics: Closeness of a document to the K-th LDA topic. Topics are built using only stop words.
As a way to test our proposal, which will be thoroughly described on section 5, we
proposed two baseline models: a simple approach based on distances between feature
vectors and an implementation of a Gaussian Mixture Model - Universal Background
Model (GMM-UBM) [
In Section 2 we described the Author Identification task as a problem P = (D; q), where D is the set of known documents written by author A, and q is a questioned document.
Let T = fPi(Di; qi); :::; Pm(Dm; qm)g be our complete set of samples in the training set and Fmxn the matrix of the complete features extracted for each set of known documents Di, where m corresponds to the number of problems and and n to the number of features extracted for each Di. For each row fj in F , corresponding to the values of feature j 2 f0; :::; ng for all samples in the training set, we adjust a Gaussian distribution.
We want to determine how unique is each author described by Fi;j with respect to the total population of samples. For measuring uniqueness, we do: 1:0 p(author).
The lower the probability, the more unique the author described by Fi;j and thus the importance of the feature is higher for said author. This is done to derive weights for each feature and normalized per sample so that they sum to 1.0.
For classifying each questioned document, we measure the weighted Euclidean distance between the question document qi and the set of known documents Di. An acceptance threshold is trained with all distances, maximizing the classification accuracy. 4.2
The Universal Background Model (UBM) is a large Gaussian Mixture Model (GMM)
trained to represent the distribution of features for all authors in the dataset. The idea
is to derive a model for one specific author by updating the trained parameters in the
UBM via a form of Bayesian adaptation [
The adaptation is a two-step estimation process, similar to the Expectation
Maximization (EM) algorithm. The first step is exactly the same as in the EM algorithm,
where estimates of the specific author features are computed for each mixture in the
UBM. In the second step of the algorithm the new estimates are combined with the old
statistics from the UBM mixture parameters using an adjusted mixing coefficient. This
allows mixtures to rely more either on old or new estimates depending on the amount
of data from the specific author that they explain [
Having trained the UBM and its resulting mixtures on the complete set of known authors, we take the feature vector for a specific author documents X = fx1; :::; xtg and compute, for each mixture i in the UBM:
We then use P r(ijxt) to compute statistics for the weight, mean and variance parameters, following the first step in the EM algorithm:
Then, these new statistics from the specific author documents are used to update old statistics on the UBM for each mixture i: (2) (3) (4) (5) (6) (7) (8) (9) (10) P r(ijxt) =
wipi(xt) PM
j=1 wj pj (xt) ni =
T X P r(ijxt) t=1 Ei(xt) = Ei(xt2) = 1 T
X P r(ijxt)xt ni t=1 1 T
X P r(ijxt)xt2 ni t=1 P (qjH0) P (qjH1)
, accept H0 < , reject H0 wi = [ iwni=T + (1 b bi = imEi(x) + (1 iw)wi] im) i bi2 = ivEi(x2) + (1 iv)( i2 + i2)
2 bi
The adaptation coefficients are f iw; im; ivg for the weights, means and variance. These are defined by i =
ni ni + r
Where r is a fixed relevant factor for all parameters , which was set empirically to 16. For tests, we used a fixed set of 2 mixtures
The classification was modeled as a hypothesis test between as proposed by Reynolds
et. al [
Having as little as one to five document per author, traditional discriminative methods
would fail to fit an accurate decision region. Attempting to improve generalization
capabilities, we proposed a feature-encoding scheme based on the Universal Background
Model (UBM) [
Ui)2 + 1 Ui)2 + 1 ji 2 [0 : : : N ) (11)
Then, we fed a Random Forest (RF) with each problem. In this way, we are building
a model that hierarchically determines the importance of each feature in the
identification of authorship. Random Forest is an ensemble discriminative method that trains a
set of predictive decision trees to classify a new instance [
Each feature would be valued with a number in the interval [0 : : : 1) if the features
computed for the unknown document is closer to the author than to the general
population, otherwise, it would be valued with a number in [1 : : : 1+). This encoding has the
advantage that it can model the triad: unknown document - author and population in a
single feature vector, making discriminative approaches feasible for this problem (i.e. it
does not depend on the number of documents per author but in the number of authors in
the dataset). However, as the features grow in an unbounded way with different scales,
assuming that all the features lie in the same scale may affect the results. Therefore, we
decided to use a Random Forest model which learns the decision region by considering
each feature independently [
On the training set for each language, both baseline models and the RF model were scored based on the measure of AU C c@1 using repeated randomly selected subsets: 80% of the samples for training and 20% of the samples for validation. Our RF approach scored higher than both baselines on the four datasets. Resumed details are explained in table 1.
In addition, test sets for each of the languages (Dutch, English, Greek and Spanish)
were provided for the competition. We submitted four runs on TIRA [
Table 2. Performance on the test corpus 7
In this work we have presented a supervised learning approach for authorship identification based on Random Forests. Previous attempts to solve this problem focused on the computation and interpretation of distance functions and threshold operations. However, it is a well known problem that nearest neighbor approaches are susceptible to the curse of dimensionality problem. In this sense, as we increase the number of discriminative features in the decision task, the amount of data needed to achieve good results grows exponentially.
Another difficulty related to the proposed problem is the small number of known documents per author, making per-author learning method unfeasible. Therefore, we adopted an discriminative approach with a encoding able to generalize the individual author information by measuring the distance relation between the unknown document, the author’s corpora and the entire dataset.
We obtained remarkable results in the Dutch and Spanish tracks, achieving AUCROC values of 0.82229 and 0.9076 in the final assessment of the competition, which are the second and third best results obtained in those tracks respectively.
Possible improvements for this approach include studying the separation of documents into paragraphs as a way to introduce more examples, analyze the relevance of the proposed features and include texts from other sources to broaden the dataset, given that our method depends greatly on the number of authors processed.