In this work, we explore how well can we perform the PAN'13 authorship identification task by using correlation of frequent significant features found in documents written by a known author. We extract features from the context of four types of words: first words of sentences or lines, nouns, verbs, punctuation. We compute the Matthews Correlation Coefficient for all pairs of extracted features and by using principal component analysis, transform them into a form suitable for using simple Euclidian distance as a metric. By hypothesizing that the distances of different documents written by a same author belong to same distribution, we can provide educated guesses for the authorship identification. Our system achieves F1-score of 66.7% for English, 56.7% for Greek and 80% for Spanish texts with an overall F1-score of 67.1% on the PAN'13 dataset.
A single subtask of the PAN‘13 authorship identification task contains up to ten
documents written by a known author and a single document, that needs to be classified as
either written by the same author or not. The dataset contains documents in English,
Greek and Spanish languages. Similar task was also presented at PAN‘11 [
Our approach is to first extract all frequent and statistically significant features from all given documents in a single subtask. Then, we use the correlation of the features to compute the distances between all given documents. We use Student‘s t-test to determine if the distances come from the same distribution, hence same author. On the preliminary data, our approach achieved F1-score of 66.7% for English, 56.7% for Greek and 80.0% for Spanish texts with an overall F1-score of 67.1%
Given a single subtask, we started by extracting lexical and morphological features.
Lexical features are the original word, does the word start with an uppercase character,
are all characters uppercase, does contain punctuation, does contain digits, is first or
last word in a sentence or a line. Morphological features are the word lemma,
part-ofspeech, other language specific features such as the case, mood etc. Similar features
have traditionally been used in authorship identification tasks [
For the feature extraction, we used NLTK toolkit for POS-tagging English [
(feature, offset, value) , where offset determines the feature/value combination of the word relative to the position of current word. For instance, a feature (case, -1, nominative) would say that the case of the previous word was nominative. We enumerated all possible features with the offset ranging from -2 to 2. Additionally, we were interested in extracting composite features that could express the co-occurrence of two or more simple features. For example, a composite feature could be (pos-tag, 1, noun) & (case, 0, partitive). Frequent composite features. We a say a feature is frequent, if its support (number of occurrences) is greater or equal to a predefined threshold. We set the treshold so that every feature matching at least 5% of tokens would be frequent. Particularly, we are interested in extracting frequent features for four different types of tokens: first words of sentences or lines, nouns and adjectives, verbs, punctuation. The tresholds were set respectively to the number of tokens in each particular group.
Counting the support of simple features is straightforward. To obtain frequent
composite features, we can use the monotonicity property of support. Given two features A
and B, we can assure that
support(A&B)
min(support(A); support(B)) ;
(1)
which means that if either of the features A and B are not frequent, then also their
conjunction A&B is not frequent. The monotonicity property makes it possible to extract
all frequent composite features efficiently using the Apriori algorithm used in frequent
itemset mining [
Statistical significance. Not all frequent features are relevant to authorship
identification due to common frequent words and language-specific grammatical and stylistic
patterns. For this purpose, we compiled a random subsets of documents from Brown
corpus [
For each frequent feature, we first compute the percentage of matched tokens on every document in the reference corpus of the subtask language. Then, we estimate the p-value as the fraction of documents with higher or equal percentage of matches than was obtained on the documents in the subtask. Next, we filter out the features that are not statistically significant (p-value 0.025). This will help to reduce the number of irrelevant features. 2
We use Matthews correlation coefficient (MCC) [
As we extracted features for four different groups of words: first words of a sentence or a line, verbs, nouns, punctuation, we compile a total of four such matrices for each subtask and by row-wise concatenation put them in a single matrix M containing all the pair-wise correlations for features in every particular group of words.
Next, we find n 1 principal components from matrix M and project the data as a new matrix P . As a result, we can view each document as a point in (n 1)-dimensional Euclidian space and use Euclidian distance to compute the distance between any two documents. As an alternative, we also measured the distance between the documents by a computing mean-squared errors (MSE) between respective correlations, but MSE did not prove to be as discriminating as the approach using PCA.
We hypothesize that if the unknown document is not written by the same author, its mean distance to known documents should be significantly larger than the mean similarity between known documents. If average mean distance of known vs unknown document was less or equal to mean distance between known and known, we automatically report the unknown document to written by the same author. Otherwise, we use onetailed Student’s t-test to obtain the p-value, which describes the statistical significance between the means. We use rather large threshold 0:5 to determine the authorship: result = (pvalue
0:5 pvalue < 0:5
Y (same author)
N (different author) :
Some subtasks of PAN‘13 dataset only contained one known document from the author. In such cases, we split the known document into two and compared the first and second part of the document as our approach requires at least two known documents from an author. 3
In Figure 1, we have depicted the distances our method for all three languages on training subset of the data that was available during writing this notebook. We see that our distance metric is quite discriminating for English and Spanish languages as documents written by same authors are on average more similar than the documents written by someone else. Our approach seems to have most difficulty with Greek language as the distances do not reflect very well the authorship of the documents.
On the full evaluation dataset, our approach achieved following F1-scores: English - 66.7%, Greek - 56.7%, Spanish - 80.0% with overall F1-score of 67.1%. On Spanish subset, our approach was ranked second. Overall, our solution shared fourth and fifth ranking from a total of eighteen participants. 2 e c n a it s d 1 0
N ● known vs unknown known vs known
N
Y
In this work, we have shown that the features we extract contain rather strong signal relevant for authorship identification. At least our t-test based approach handles certain cases correctly. However, we might improve the results by using more sophisticated novelty detection algorithms instead. Also, we might get stronger features by performing PCA separately on each of the four token groups. The next important step in our future work is evaluating these options.