In this paper we address style change detection problem at PAN'18 author identification task. For this task one should determine whether text is written by the same author or not. We consider supervised problem statement with the whole text as a training object. The roposed approach is based on three types of features: text statistics, hashing and high dimensional text vectors. The final algorithm is the ensemble of classifiers that were independently trained on each feature group.
Authorship detection is a class of open problems in natural language processing. This
class contains a bunch of the tasks that were featured in previous PAN competitions,
namely:
1. Author clustering [
PAN’18 consists of the following tasks: author identification task [
However, in this paper we develop a framework that considers the whole text as a training object without any segmentation. On the one hand, such problem statement was inspired by the fact that we deal with binary classification, on the other hand we try to contribute slightly different point of view on the problem.
First, we perform preprocessing procedure that is different for each specific
classifier. Next, we extract three types of features: text statistics, hash code of a text, and
high-dimensional sparse representation of a text, obtained by simple counting of word
n-grams appearance in range 1-6. Such n-grams counting showed success in different
tasks from intrinsic plagiarism detection [16] to author profiling [
In this section we state the problem formally. Consider text documents collection D of size m and denote i-th document of collection by Di, where i 2 1; : : : m. Let f be the mapping, such that each document of the collection is mapped to fixed-size vector:
where class label 1 is for documents written by more than one author and 0 for singleauthor documents. Let LD be a empirical risk defined by: f : D
! Rd: h : Rd ! y 2 f0; 1g; LD(h) = jfi : h(Di) 6= yigj ;
m h^ = arg min LD(h);
h2H where yi is class label for i-th document.
We want to find h^ that minimizes LD on a given collection D: where H is parametric family of functions. 3 3.1
The data corpus consists of user posts from various sites of the StackExchange network. Data is split into training and validation sets that contain 2980 and 1492 texts respectively. 3.2
To evaluate the quality of proposed algorithm, the accuracy score was used. Accuracy is the fraction of correct predictions. More formally, for binary classification accuracy has the following definition:
Accuracy =
TP + TN TP + TN + FP + FN ; where TP = True Positives, TN = True Negatives, FP = False Positives, and FN = False Negatives. 3.3
Our model consists of three independent classifiers: Statistical, Hashing, and Counting Classifier. Each classifier returns the probability of the fact that text contains style changing. And the final probability is the weighted sum of three probabilities — ps; ph; pc respectively.
Statistical Classifier. Statistical classifier uses 19 statistical features for a text analysis. The most important of them are: – number of sentences; – unique words fraction; – text length; – punctuation symbols fraction; – letter symbols fraction.
To produce final probability Random Forest Classifier was used.
Hashing Classifier. This model uses hashing function to build term frequency counts in a text. The hash function employed is the signed 32-bit version of Murmurhash31. As a result, a text is maped into 3000-dimensional vector space. These vectors contains information about occurrences of char n-grams in text. Text representation vector is used to classify whether a text contains style changes or not. Random Forest Classifier was used to produce probability.
Counting Classifier. Counting Classifier uses high-dimensional (3 million) representation of a text. Different dimensions were tried but they showed lower quality. It counts word n-grams form 1 to 6 and turns it to a vector. Logistic Regression is then used to get the probability.
Statistical, Hashing, and Counting Classifiers were trained on the train set in order to maximize performance measure — accuracy — independently from each other. Resulting performances are shown in the table below. 1 http://scikit-learn.org/stable/modules/generated/sklearn.
feature_extraction.text.HashingVectorizer.html Model. The final score for text d is the weighted sum of probabilites:
Hashing Classifier Counting Classifier where coefficients s; h; c are selected from (0; 1).
If the score for a text exceeds the threshold , then this text is marked as text with change of style:
) d has change of style: Coefficients s; h; c and threshold were tuned on the validation set by grid search in order to maximize accuracy. Each of the coefficient s; h; c shows the importance of corresponding classifier. Optimal parameters for the final model are: s = 0:4; h = 0:2; c = 0:4: We can see, that Statistical and Counting Classifiers are the most informative. And the value of threshold is: = 0:55: The relation between accuracy and value of threshold is shown on the figure below. 3.5
The proposed model was tested on PAN’18 data set. The results of its performance are shown below.
We proposed an algorithm for style change detection task. This algorithm uses three independent classifiers: Statistical, Hashing, and Counting. Each classifier gives its own probability that a text may contain a change of style. Final score is computed as weighted sum of three probabilities. And if the score exceeds the threshold, a text will be marked as it containing a change of style.
The method was implemented for the PAN’18 style change detection task. The model has achieved accuracy score 0.803 on the test dataset. 13. Safin, K., Kuznetsov, M., Kuznetsova, M.: Methods for intrinsic plagiarism detection.
Informatics and Applications (2017) 14. Safin, K., Kuznetsova, R.: Style breach detection with neural sentence embeddings. In:
CLEF (2017) 15. Samdani, R., Chang, K.W., Roth, D.: A discriminative latent variable model for online clustering. In: Xing, E.P., Jebara, T. (eds.) Proceedings of the 31st International Conference on Machine Learning. Proceedings of Machine Learning Research, vol. 32, pp. 1–9. PMLR, Bejing, China (22–24 Jun 2014) 16. Stamatatos, E.: Intrinsic plagiarism detection using character n-gram profiles (2009) 17. Stamatatos, E., Tschuggnall, M., Verhoeven, B., Daelemans, W., Specht, G., Stein, B., Potthast, M.: Clustering by authorship within and across documents. CEUR Workshop Proceedings (2016) 18. Stein, B., Barron, Cedeno, L., Eiselt, A., Potthast, M., Rosso, P.: Overview of the 3rd international competition on plagiarism detection. CEUR Workshop Proceedings (2011) 19. Zechner, M., Muhr, M., Kern, R., Granitzer, M.: External and intrinsic plagiarism detection using vector space models. Proc. SEPLN. vol. 32 (2009)