In this work we describe our approach to solve the author verification problem introduced in the PAN 2014 Author Identification task. The author verification task presents participants with a set of problems where each problem consists of a set of documents written by the same author and a questioned document with an unknown author. The task is then to decide whether the questioned document has the same author as the other documents or not. Inspired by a psychological personality model, our approach uses basic lexical feature extraction and fuzzy clustering. Using the created fuzzy clusters, the membership values of documents to the clusters can be computed. The distribution of the cluster membership values will be used finally to solve the verification problem.
Given a set of documents with known authors, authorship attribution is the task of
identifying the author of an unseen document. Having a small number of candidate authors,
this task can be easily solved using the state-of-the-art approaches[
The PAN 2014 Author Identification task focuses on the author verification problem. To be more specific, in this task a multi-lingual corpus is provided which consists of several problems. Each problem contains a maximum of 5 documents written by a single author and a questioned document by an unknown author. The task in then to determine whether the questioned document is written by the same author or not.
The fact that an author may consciously or unconsciously vary his or her writing
style, makes the task of author verification a hard problem[
In Section 2 we define the problem of author verification formally and introduce some notations. Section 3 addresses the process of feature extraction and normalization. The process of clustering and model construction is discussed in Section 4. Section 5 covers the process of verification and scoring. An overview of the evaluation results can be seen in Section 6. Finally in Section 7 the work will be concluded. 2
In this section we formally define the problem of author verification in the context of the PAN 2014 Author Identification task.
Let P = fD; dug be a problem consisting of a set of documents D = fd1; : : : ; dng
with 1 n 5 written by a single author, and a questioned document du with an
unknown author. The task in author verification is to determine whether the questioned
document du is written by the same author or not. We denote the author of a document
di by A(di). In other words an author verifier ' is a binary classification function of
the following form:
In the PAN 2014 Author Identification task, problems are from 4 different languages,
namely Dutch, English, Greek and Spanish. The author verification algorithm has to
be able to deal with documents from the specified languages. The performance of the
author verifier will be evaluated according to the area under the ROC curve (AUC) of
its probability scores and also based on the c@1 measure[
In the following section, we start the description of our algorithm by discussing the feature extraction and normalization step. 3
Feature extraction is considered as one of the important steps in author verification[
We represent documents as vectors in R4. Each component of these 4-dimensional vectors can be computed using the feature extraction functions. Independent of the document language we use the following functions to compute the feature vector components of documents: Average Sentence Length (fsl) : Using a sentence detector, sentence boundaries of the document will be detected (In our case we use a regular expression based sentence detector for optimizing the performance). For each sentence s in the document, its length l(s) will be computed. We denote the set of all sentences inside a document with S. Finally the average sentence length of the document can be computed as follows: Punctuation Marks Usage (fpm) : Using a predefined set of punctuation marks T = f ( ) ; : ; ! ? g the frequency of the elements of the set T inside the document will be computed and finally normalized by the length of the document. With f (t; d) we denote the frequency of the punctuation mark t in document d.
fsl(d) =
Ps2S l(s)
jSj fpm(d) =
Pt2T f (t; d) jdj Space After Comma (fsac) : Our experimental results show that whether a space is used after a comma or not, can be a good discriminating feature in the author verification task. Let denote the number of times a comma is followed by a space and be the number of times a comma is not followed by a space. In his way fsac can be defined as follows: fsac(d) = jdj Analogue to fsac we define fsbc which is the Space Before Comma feature. Through this feature, authors that use a space before comma can be discriminated from the ones who do not use a space before comma.
As the extracted features may exhibit significant differences in their range and distribution, out learning algorithm could be more sensitive to features that are in a wider range (e.g. Average Sentence Length). In order to avoid this behavior we use feature normalization through which we can modify the mean and variance of the features using a transformation function. The transformation function that we use in this work is the min-max function. Given a feature f , the min-max transformation function which is defined as follows: In the above formula f denotes the feature vector and f 0 is the transformed feature vector. 4
In this section we illustrate the main idea behind our learning algorithm. We believe
that different personality dimensions have a close relationship with the writing style of
authors. In psychology, the Big Five Personality Traits are 5 dimensions of personality
that are used to describe the personality of humans[
For this we collect all the documents in our training set from which we know that
they are written by the same author and extract their features (See Section 3). This will
result in a matrix Z = [z1tr; z2tr; : : : ; zNtr] 2 R4 N where N is the number of collected
documents and ztr denotes the transpose of the vector zi. As already mentioned the
i
personality of an author can be determined using his or her membership values to the
available clusters. Due to this consideration, we use the Fuzzy C-Means[
Step 1: Compute the cluster centers with m 2 [1; 1) v(l) = i
k=1( i(kl 1))mzk PN
k=1( i(kl 1))m PN ; 1 i c Di2k = v(l) 2 i = (zk vi(l))T (zk vi(l)); 1 i c; 1 k
N (7) Step 2: Compute the distances Step 2: Update the partition matrix: zk k for 1
N otherwise if Dik > 0 for all i = 1; 2; : : : ; c i(kl) =
1 Pc
j=1(Dik=Djk)2=(m 1) i(kl) = 0 if Dik > 0; and c i(kl) 2 [0; 1] with X
i(kl) = 1 i=1
U (l 1) < (6) (8) (9)
We use the cluster information produced by the cluster algorithm, to verify whether two documents are written by the same author or not. The process of author verification will be discussed in the following section. 5
In order to find an answer to an author verification problem P , we compute the cluster membership values for documents with known authors and documents with unknown authors. Then using the membership values we will decide if the documents have the same author or not.
Given a problem P = fD = fd1; : : : ; dng; dug and c cluster prototypes (centroids) V = fv1; : : : ; vcg we compute the membership values of the documents with known authors to the constructed clusters. In this way, for each document di; 1 i n a cluster membership vector i = f i1; : : : ; icg will be computed where the j-th element in the vectors represents the membership value of the document di to the cluster j.
In the same way we compute the cluster membership values of the document with unknown author du. This would result in the membership vector u. At this step the cluster membership values for all documents in the problem P are known. Notice that the documents d1; : : : ; dn are assumed to be written by the same author. Theoretically we would expect that the cluster membership vectors of these documents look very similar to each other. Experimental results show that this is usually not the case, which relies on the fact the authors write in different psychological states.
In order to solve the above problem, for the documents with known authors, we compute a mean cluster membership vector. Through this vector a more stable estimation of membership to available personality dimensions can be made. The mean cluster membership vector of a set of documents d1; : : : ; dn with known authors can be computed as follows:
Now using the cosine similarity between the average cluster membership vector of
documents with known authors and the questioned document, the similarity between
these two vectors can be computed. The cosine similarity between these two vectors is
defined as follows[
Pn i=1 i n S~; u = ~
u k~k k uk (10) (11)
Through the cosine similarity measure we compute the angle between the vectors. A cosine values of 0 means that the vectors are orthogonal to each other and a cosine value of 1 means that the vectors are identical. Through the cosine similarity measure we assigned a score to each problem. Additionally we need a transformation function which can return binary values for author verification problem. In Section 2 we defined the function '(du; D). Here we modify this definition and redefine the function: (1; if S~; u
Using the above function definition, for each problem P it can be decided if the documents inside P belong to the same author or not. A value of 1 means that the documents inside P have the same author and a value of 0 means that the questioned document has a different author than the documents with a known author. 6
In order to evaluate our approach we used the training set provided by the PAN 2014 Author Identification task. The training set consists of documents belonging to 4 different languages, namely Dutch, English, Greek and Spanish. Dutch documents are divided into essays and reviews, and English documents into essays and novels. Greek and Spanish documents belong only to the genre Articles. In total we constructed 6 models, where each model corresponds to a specific language and a specific genre.
For constructing the clusters of language L and genre G, we randomly selected 20%
of the available training data to create the clusters. The experiments have been repeated
1000 times and average c@1 measure of the iterations has been computed. The c@1
measure of a single iteration can be computed as follows[
n where, n = number of problems, nc = number of correct answers and nu = number of unanswered problems. The results are summarized in Table 1. Dutch Dutch English English Greek Spanish
Essays Reviews Essays Novels Articles Articles
In Table 1 the number of created clusters and the parameter m are also specified. These parameters are the ones that returned the best results during our experiments. As we can see the algorithm returns the best results for the English novels with an c@1 value of 0.852. The worst results are also for the English documents but the ones in the genre essays. Even though the c@1 values for all languages and genres are greater than 0.66.
Beside the above approach, we evaluate the performance of our algorithm according
to the area under the ROC curve (AUC)[
For ranking the performance of participants in the competition a test corpus has
been provided. We have evaluated our algorithm using Tira[
From the performance results based on the test corpus it can be seen that our algorithm performs very well for English Novels, and Essays reaching a final score of 0.508 and 0.349 respectively. But for the other languages the results are not as satisfactory as expected. This difference between the results indicates that for languages other than English, a deeper feature engineering is needed. Dutch Dutch English English Greek Spanish
Essays 0.635 0.594 Reviews 0.500 0.493 Essays 0.580 0.602 Novels 0.715 0.711 Articles 0.540 0.543 Articles 0.650 0.640
The run-time of our algorithm on different data sets also shows that the introduced algorithm can be efficiently used for large collections of author verification problems. This is due to the small number of features that we extract from documents. This has from one side the advantage that the author verification problems can be solved very efficiently, but from the other side, it will result in a lower performance for specific languages. 7
In this work we have described our approach to solve the author verification problem introduced in the PAN 2014 Author Identification task. Using the fuzzy c-means clustering algorithm, we partitioned the provided training set (Section 4) into several clusters. Given an author verification problem, we used the membership values of the documents inside the problem to verify whether two documents have the same author or not.
In order to design an efficient algorithm we only considered a limited number of features for each language. This resulted in very low run-times for our algorithm. Accordingly we acquired the 1st place among the participants regarding the run-time of algorithms.
Our introduced approach also revealed sound results for the English language achieving the 1st place for English Novels and the 5th place for English Essays among the 13 participating teams. For other languages we did not get the expected satisfactory results. The reason for this lies in the small amount of training set that we use for constructing our fuzzy clusters. We also use the same set of features for all available languages which is probably the main reason for insufficient results for languages other than English.