The aim of this paper is to explore the usefulness of using features from different linguistic levels to email authorship identification. Using various email datasets provided by PAN'11 lab we tested several feature groups in both authorship attribution and authorship verification subtasks. The selected feature groups combined with Regularized Logistic Regression and One-Class SVM machine learning methods performed well above average in authorship attribution subtasks and below average in authorship verification subtasks.
Authorship identification refers to the connection of a text of unknown authorship to
a specific author using a set of quantifiable text features as indicators of the author’s
style. Since the late 1990s authorship identification has known a new impetus based on
developments in a number of key research areas such as Information Retrieval, Machine
Learning and Natural Language Processing [
The authorship identification dataset provided by the 5th International Workshop on Uncovering Plagiarism, Authorship, and Social Software Misuse PAN’11 provided a test bed for comparing different strategies in both feature selection and classification algorithms. Our approach to authorship identification is based mainly on the idea that an author’s style is a complex multifaceted phenomenon affecting the whole spectrum of his/her linguistic production. Following the old theoretical notion of "double articulation" of the Prague School of Linguistics we accept that stylistic information is constructed in blocks of segments of increasing semantic load, from character n-grams, to word n-grams. In order to capture the multilevel manifestation of stylistic traits we should detect these features, which belong to many different linguistic levels, and utterly combine them for achieving the most accurate representation of an author’s style.
Authorship identification research has used an impressive array of stylometric features
ranging from characters to syntactic and semantic units. We selected our features taking
into consideration the best practices established in authorship identification research
published from the 1990’s till today [
As mentioned above we decided to focus on features that cover a wide range of
linguistic levels and at the same time are easy to implement and are language independent.
We used five single feature groups and in a later stage we combined them in a feature
group labeled "All", a methodology that gave us the best results in the validation set and
is generally accepted as better strategy [
Character n-grams approach phonology and morphology capturing quantitative information regarding syllable structure, phonotactics, consonant clusters, prefix and suffix structure. Word n-grams on the other hand approach syntax organization including different lexical bundles, phrases, collocation structures among others.
The most frequent unigrams were detected using a custom PERL script which identified tokens as a sequence of alphanumeric characters using the regular expression \w+. Later a custom PERL script took as input a list of the most frequent tokens in the training corpus and produced a vector containing text length normalized frequency of occurrence of each token in all the texts contained in the datasets.
The most frequent n-grams were detected using the Ngram Statistics Package (NSP)
[
Output files from NSP were converted to vectors using custom PERL script which aggregated n-gram counts from each text file and normalized their frequency to the text length. Given the rather huge dimensionality of the extracted features, the task of training models is time and memory consuming, even for moderate number of training instances. Therefore a method for solving efficiently large scale classification problem was required.
For the purposes of the authorship attribution tasks, we used Regularized Logistic
Regression (RLR) as implemented in LIBLINEAR [
3.1
In the Authorship identification task we trained our classifier (RLR) using the default values (Costparameter = 1, Epsilon = 0:01). In order to evaluate the performance of our classifier in the training set we used both accuracy and F1 averaged in 10-fold cross-validation of the testing sample. The trained model we obtained from this procedure was used for prediction in the LargeValid dataset. Its performance was also measured using accuracy by comparing the predictions made by the classifier with the labels provided by the GroundTruthLarge and Small Valid files. We compared different feature groups both single and in different combinations. The best results obtained from the "All" feature group which contained the 1000 most frequent features from each single feature group. The results from our experiments in LargeTrain, LargeValid and SmallTrain, SmallValid datasets are shown in the following table. We report only the "All" feature group and the single group results since the number of combinations we examined was large (25 feature group combinations for each dataset):
From the above table it is obvious that combining the feature groups we get the best classification accuracy over all the datasets. In order to ensure further the superiority of the combined feature group we conducted a series of pairwise t-tests comparing the "All" feature group with each of the single feature groups. Since we had multiple comparisons a Bonferroni correction was applied to the p level of significance (p = 0:01) of all the t-test conducted. In all the comparisons employed, the "All" feature group obtained a statistical significant better classification accuracy and F1 over each one of the single feature groups providing support to our claim that a combined feature group consisting of features from multiple and different linguistic levels capture more efficiently an author’s style. 3.2
In authorship verification tasks (+ datasets) two subtasks were defined: – Combined authorship attribution and verification: In this subtask the aim was to find which of the given texts in the LargeValid+ and SmallValid+ were written from authors within the corresponding training set and which from external authors. In the first case we trained a One-Class SVM model with RBF kernel using the LargeTrain and the Small Train datasets as one class. The trained model was applied to the LargeValid+ and SmallValid+ datasets in order to identify the texts that were written from the authors of the training sets. All "unknown" cases were assigned the label "unknown" and they were removed from the Valid+ datasets. Then we applied the previously trained models from the authorship identification subtask to the reduced Valid+ datasets and performed authorship attribution. The final results obtained from this procedure are shown in the Table 2 below: – Authorship verification task: The aim of the second subtask was to find if 3 specific authors (Verify1, 2 and 3 datasets) had written any and what texts from the provided validation datasets (Verify1+, 2+ and 3+). The procedure followed in this case was the training of One-Class SVM model using RBF kernel and its subsequent application to the respective validation datasets. Furthermore, since our training set was small we used only the 2000 most frequent character bigrams of the training set in order to train our classifier. The final results from this procedure are shown in Table 3: The Author Identification competition organized by the PAN 2011 Lab was an interesting and challenging task in which we had the opportunity to test the usefulness of both features and machine learning methods in a variety of authorship attribution and verification scenarios. Our features covered a wide range of linguistic levels, from sub-word entities (character bigrams, trigrams) to word and hyper-word formations (word unigrams, bigrams and trigrams). In the authorship attribution subtask we used the above mentioned features combined with the Regularized Logistic Regression. This approach scored well in all the performance indices except the macro-averaged precision probably due to the large dimensionality of our solution. In total our system ranked in the 5th (out of 13 groups) and 3rd (out of 12 groups) position in the LargeTest and SmallTest dataset correspondingly.
On the subtasks of authorship verification our approach scored below the average performance of the participating research groups. In the combined scenario of authorship attribution and verification we trained a One-Class SVM in the training datasets with the same features used in authorship attribution subtask in order to identify and exclude the texts that weren’t part of the training set. Then we performed an authorship attribution to the remaining texts. This approach was ranked in the 6th (out of 9 groups) and 5th (out of 9 groups) position in the LargeTest+ and SmallTest+ dataset correspondingly. In the plain authorship verification task we trained a One-Class SVM using the 2000 most frequent character bigrams with little success in the respective datasets. Our approach obviously suffered from overtraining since we obtained high recall but low precision values in Verify2+Test and Verify3+Test datasets. More specifically our system ranked in the 4th (out of 7 groups), the 7th (out of 7 groups) and 5th (out of 7 groups) position in the Verify1+Test, Verify2+Test and Verify3+Test dataset correspondingly. We believe that some of the factors affecting the performance of our system in the verification tasks were the usage of the LIBSVM One-Class algorithm combined with the usage of the character bigrams in the plain verification subtask. There is a constant decline of our system’s performance when we employ LIBSMV One-Class in the combined subtask (Large+, Small+ datasets) and a further decrease when we shrink our features to character bigrams.
The results obtained from the authorship attribution subtask are encouraging and support our claim that authorship is based on textual features that are scattered in a wide spectrum of linguistic levels. Future research will be directed to detect features from other linguistic levels and use them in attribution tasks taking into consideration not only their frequency but also their discriminative power especially in small classes in order to improve our macro-average performance indices. Furthermore, we will continue our experimentation in the verification subtask with different one-class learning algorithms and varying feature groups.