In this paper, we describe a modified version of the profile-based approach for the Authorship Attribution (AA) task of the PAN 2012 challenge. Our PAN system for AA utilizes the concept of linguistic modalities1 on profile-based (PB) approaches. We concatenate all the training documents from the same author and build author-specific sub-profiles, one per linguistic modality. Then instead of using all the different types of features to compute the similarity of a test document against an author's profile in a single step, we compute several similarity scores using one set of features (modality) at a time. Each modality will assign the test document to the author whose sub-profile has the highest cosine similarity in that modality. Final classification decisions are based on the combination of decisions from each modality using majority voting. We achieved competitive results on PAN 2012, with encouraging results on the closed-class authorship attribution.
Authorship attribution is the task of identifying the author of an unseen document. AA
has a long history with multiple application areas that include spam filtering [
In a recent work, Solorio et al. (2011) showed that representing documents as a
set of separate linguistic modalities in a standard machine learning approach yields
good results in AA. On the other hand, profile-based (PB) approaches have consistently
shown promising results for the same task [
We participated in both the closed-class and the open-class problem and submitted one run per sub-task. The parameters of the final authorship attribution system, for both the closed-class and the open-class were adjusted using only the training data. Our system yielded very competitive results in the competition, reaching the best accuracies for several tasks. In what follows, we will discuss in more detail the process we followed during the parameter tuning of our system, as well as the evaluation results. 2
Authorship attribution problems have been solved mainly using either machine learning
or profile-based approaches [
Profile-based approaches have already been successfully used for attributing an
unknown text to its real author. Successful examples of this approach include: [
A lot of emphasis has also been placed on the use of machine learning for AA.
Successful examples include a wide variety of learning algorithms, such as Support
Vector Machines [
The main difference between our approach and the standard PB approaches is that we
use a set of sub-profiles to make the predictions, instead of computing a single one. The
different sub-profiles correspond to the different linguistic dimensions in the writeprint
of authors. We followed the notion of modalities proposed in previous work [
simm(Pa; Pt) =
Pa:Pt jPajjPtj ∑lim=1 Pai
Pti = √∑lm i=1 Pai2 √∑lm i=1 Pti2 (1)
We allow each modality to make authorship predictions based on the cosine
similarity of the authors sub-profiles in that modality. Each modality will assign the test
document to the author whose sub-profile has the highest cosine similarity in that
modality. Final classification decisions are based on the combination of decisions from each
modality using an appropriate voting mechanism [
Due to the limited number of training instances per author, here we use the majority voting to combine the decisions of each modality in the ensemble. Let x be a test instance. The final prediction pt for test instance x is determined as:
k
pt = argmax ∑
y
m=1
(hm(x); y);
y 2 Y;
(2)
Where hm(x) is the prediction of modality m, k is the number of modalities, (i; j) is
an indicator function whose value is 1 iff i == j and 0 otherwise, and Y = y1; :::; yl
is the set of possible classes (authors). To increase the prediction performance as well
as decrease the computation overhead, we decided to perform feature selection based
on information gain. To determine the percentage of features to be selected from each
modality, we used 50% of the training data as validation set and remaining 50% as the
training. We experimented using different percentage of the features for each modality:
20%, 40%, 60%, and 80%. Looking at the performance of the system on the
validation set, we determined the number of features to be used for the attribution of test
instances. After analyzing the performance on validation set, we decided to use 80%
of the stylistic features, 60% of the syntactic features, 20% of the semantic features,
20% of character n-grams, and 40% of word n-grams. To perform the feature selection
based on information gain, we converted the feature vector file to arff format and used
WEKA [
We also participated in the open class authorship attribution task of PAN 2012. The open class task is much harder than the closed class as test documents may belong to an unknown class. We did not do feature selection for this task. However, for the open class problem, we have an extra modality containing perplexity values from 4-gram language models at the character level. The extra modality was added after evaluating the performance of the open-class AA system on the validation set. When the improvement in the performance was observed with the addition of extra modality, we decided to include it for the open-class problem.
To deal with the “out of training set" cases, we decided to use a threshold on the difference between the cosine distance of the 1st and 2nd prediction of each modality. More specifically, to determine if a test document belongs to an unknown author, we compare the highest and the second highest cosine similarity score of the author subprofiles for each modality. If the difference between them is smaller than the threshold value , we decide that the test instance belongs to none of the authors in that modality. We consider the threshold a sort of filter for confusing cases, and we assign those confusing cases to the “unknown" category. The idea behind this approach is that if there is indecision, as indicated by a close similarity score between different authors, this indecision is coming from trying to force an assignment of authorship to a document that has no clear match inside the training set, then we should not assign that author to any of our candidate authors, and the system should predict “unknown" in that case. 4
All the parameters in our system were defined based on different experiments using
only the training data. We used 50% of the training instances as validation data and the
remaining 50% as the training data to set the parameters. In the cases of n-grams, we
selected the top 3000 features, except for word n-grams because the performance on
the validation set was better when the top 1500 of each word n-gram were used, where
n = 1, 2, 3. We used the Stanford parser [
There are a total of six datasets, three from the closed-class problem and three from the open-class problem. For each dataset in the closed-class problem, there is a corresponding dataset in the open-class problem. Therefore, the training data is the same for both the closed-class and open-class problems while test data is different. There are three kinds of datasets for each problem: one with three authors, another with eight authors, and the last with 14 authors. As described in Section 3, we first create the author-specific sub-profiles, one per linguistic modality. Then the prediction of the test instances is carried out separately on a modality basis. This means each modality will have its own prediction on the same test instance. The predictions are combined using a majority vote.
Modality Type
Stylistic Syntactic Semantic Character n-gram Word n-gram Majority Voting
Table 2 presents the performance of the individual modalities as well as the combination of the predictions from each modality on three different datasets provided in PAN 2012 authorship attribution competition. Our system achieves an overall accuracy of 100% on two out of the three datasets. The accuracy on the remaining dataset (14 authors) is also very high. As it can be seen, different modalities have different prediction accuracies on different datasets. For 3 Authors, syntactic as well as character n-grams obtain 100% accuracy, while all other modalities obtain 83.33%. Similarly, for 8 Authors, each modality performs differently. As expected, when we combine the individual predictions of the sub-profiles in an ensemble, the accuracy of the final combination is either higher or equal to that of the best individual classifier. The final accuracy for 8 Authors is 100% , which is more than the accuracy of each individual classifier. After looking at the accuracies of other PAN participants on each dataset, we observed that no participants obtained accuracy higher than ours. This proves that our AA system for the closed-class problem worked consistently well across different datasets and was among one of the the top AA closed-class systems in the competition.
In the open-class problem, the challenging part is to determine the threshold value. We performed experiments against different datasets using 0.02, 0.04, 0.06, and 0.08 as the threshold value ( ) and measuring accuracy on the validation set. Based on the performance of the AA system on validation set against the different values of we decided to set to 0.08. We even tried by increasing the value of beyond 0.08, but very poor performance on the validation set was obtained. As a reminder, we compare the highest and the second highest cosine similarity score of the author sub-profiles for each modality. The difference between them is compared against the threshold to determine if an instance belongs to an “unknown" category.
Table 3 presents the accuracy from using individual linguistic modalities, as well as the combination of the predictions from each modality for the open class problem of the PAN 2012 competition. Even though our results for the third dataset with 14 authors was not good, for the other two data sets our system yielded promising results. After combining the predictions from individual modalities, an accuracy of 60% was observed for the 3 Authors data set. This accuracy can be considered competitive when compared with that of other competitors on the same dataset. Our open-class AA system performed very well on the 8 Authors data set, reaching an accuracy of 76.47%, which is the highest accuracy obtained on that dataset. After analyzing the performance on the open-class datasets, we see that in only one case (row 2 of Table 3), the combined accuracy is higher than that of the individual classifiers. While for other two datasets, the combination of predictions from the sub-profiles is not the one with the best results. This could be due to having some very weak predictors as part of the ensemble.
For both closed-class and open-class problem, sub-profiling by linguistic modality achieved a very good performance overall in the PAN 2012 authorship attribution challenge. However, further experiments with additional datasets is required to evaluate the performance of the algorithm that we introduced in this paper. For the open-class problem, we proposed a new way of assigning a test instance to an “unknown" category. This new algorithm successfully worked for all the three datasets. However, additional empirical analyses are needed in order to explore different parameter values for this setting. 5
In this paper, we described an approach to the authorship attribution task adopted for the PAN 2012 competition. We introduced the notion of generating sub-profiles that represent the author writeprint along specific linguistic dimensions. Although the idea of linguistic modalities has been explored by previous work, it was used in a machine learning setting, while here we adopt that framework and combine it with a profilebased approach. The evaluation results on different datasets show this is a competitive AA framework for closed-class as well as open-class AA problems. For each dataset in the closed-class AA problem, our system matched the highest accuracy reported in the competition. This clearly illustrates that the proposed algorithm for closed-class problems is very competitive and consistent.
In the open-class AA framework our system did not reach the same high performance, but was also very competitive in the challenge. For one out of the three openclass datasets, we were able to get the best accuracy of the PAN 2012 challenge. The performance of this system for the dataset with 14 authors was not impressive. This leaves us with an impression that we need to keep investigating for a better way to handle open class settings.
This work was partially supported by ONR grant N00014-12-1-0217. The authors would like to thank the organizers of the PAN 2012 competition for putting together this event.