This paper summarize our approach to author profiling task - a part of evaluation lab PAN'13. We have used ensemble-based classification on large features set. All the features are roughly described and experimental section provides evaluation of different methods and classification approaches.
Main goal of authorship analysis is to retrieve information carried by the text about specified characteristics of its author. Characteristics may relate to demography, culture, nationality, personality, etc. Such analysis, for example, may be applied to discover the relation between profile of a person and his/her opinion on particular subject. It may also help in recognition of criminal and terrorist activities. In this paper we describe our approach to Author Profiling task, which was a part of the PAN 2013 competition3. The goal was to determine gender and age of given chat conversations’ authors.
We decided to apply ensemble-based classification methods because of their potential for more effective recognition of complex patterns. Ensemble methods combines several weak classifiers into one classifier, which is more effective than the individual ones. The features were created on the base of structure, stylometry and semantics of the text. The diversity of explored properties of the dataset assures high independence of the features. The dataset was divided into two language groups (English and Spanish). Each group included conversations stored in XML files (grouped by author). There were 236 000 files (564 413 conversations, 180 809 187 words) in English and 75 900 files (126 453 conversations, 21 824 198 words) in Spanish. 3 http://pan.webis.de/
In the data preprocessing phase we employed standard techniques for text cleaning
and tokenization. Regular expressions was used to strip out most of html tags leaving
special tokens for hyperlinks and embedded images. Some of feature extraction
procedures required word and sentence tokenization. Word tokenization is easy task and can
be addressed by regular expressions. For sentence splitting we have used unsupervised
algorithm [
We noticed that many conversations in dataset seemed to be a spam (probably produced by some chatterbots used for commercial purposes). It is reasonable to assume that age and gender characteristics in spam-like text have different rationale. One can say, for example, that advertisement of certain products is more oriented towards one gender group than the other one. Also mixing together chatter bots and humans conversations can introduce unnecessary noise in phase of learning of a classifier. Because of that we put extra effort to discriminate spam-like over human-like chats. 3
With each document we associated a collection of features, which were employed by classification algorithms to identify the age and gender of the document’s author. In the final versions of our analytic dataset (that was used for classifier training) there are 311 and 476 features, respectively for the texts written in English and Spanish.
These features can be divided into groups roughly described below. 3.1
Structure of a document is proven to be important feature in various classification problems regarding text mining and authorship profiling. There were not much previous work, however, that can be easily adopted in our approach. Therefore we have engineered very simple features, that not necessarily carry information about authors profile but can be used to group the documents into similarly structured conversations (eg. long and shorts ones). This approach enables us to discover more subtle high-level features in various document groups.
Some examples of such structural features are: the number of conversations, paragraphs, sentences and words per sentences, number of special characters, etc. If the person conducted more than one conversation we measure minimum, maximum and average conversation length. The usage (absolute count and ratio) of hyperlinks and images, and whether they were used at the beginning or at the end of the conversation. After more thorough investigation we discovered that this particular feature can be also useful in spam detection. We also noticed that chatterbots seem to perform very similar conversations therefore we measured the Jaccard similarity coefficient of individual conversations and enclosed this average edit distance into an analytic dataset. 3.2
Argamon et.al [
We employ n-gram language model (separate for age and gender problem) to create
a vector of features. An n-gram in our case is a contiguous sequence of n pos-tagged
words from a given sequence. In the learning phase for each class C we selected all
files ascribe to it and we count the number of occurrences of each n-gram and
(n-1)gram in chosen files. Next, on the basis of determined values, for the given class C, the
probabilities PC (xijxi n+1; :::; xi 1) are estimated, for more details see [
Obtained probabilities are employed to generate vector of features for a given conversation. To this end for each sentence x n+2; ::xl+1 in the conversation and for each class C is calculated a vector of the posterior probabilities P(Cjx n+2; ::xl+1). In order to calculate this value, the probability of occurrence of the sentence in the files ascribe to the class C is calculated:
PC (x n+2; ::xl+1) = l+1 Y PC (xijxi n+1; :::; xi 1): i=1 (1) In the equation 1 we make a (n 1)th order Markov assumption. Next, the parameter PC (x n+2; ::xl+1) is multiplied by the prior class probability P(C). Values obtained in this way for all classes and normalized to sum up to 1, create a vector of the posterior probabilities P(Cjx n+2; ::xl+1). We sum these vectors from all sentences in the conversation. The obtained vector is again normalized to sum up to 1 and is returned as the final feature vector.
For both languages we employed 3 models for n 2 f4; 5; 6g. So, for age we have
2 3 = 6 features and for gender 3 3 = 9. A n-gram language model has a very
large number of parameters. For both languages, we applied part-of-speech taggers,
which maps words to the set of 46 different symbols, what results in 46n different
possible n-grams. Even with a huge set of training sentences, many of the n-grams
do not occur in the training set. It is serious problem, since some of these sequences
appear in conversations, which we want to classify. We apply discounting method
[
PC (xijxi n+3; :::; xi 1). For English conversations = 0:2; = 0:5 for gender and = 0:9; = 0:9 for age. For Spanish conversations = 0:2; = 0:5 for gender and = 0:7; = 0:9 for age. These values were chosen experimentally. 3.4
In order to determine text difficulties we applied several readability tests for the
documents: Flesch Reading Ease, Flesch-Kincaid Grade Level and Dale-Chall readability
formula. They are based on the number of words, sentences, syllables and difficult
words (there is Dale-Chall list of 3 000 familiar words [
We wanted to examine the intensity of words and expressions of particular types. In
each document we counted number of abbreviations, emoticons and badwords. We
useed NodeBox [
Numbers of errors and language mistakes is determined by using LanguageTool [
M
Uk kVk;
(2)
where M is tf-idf weighted term-doc matrix, Uk and Vk can be interpreted as term-topic
matrix and topic-document matrix both in low rank approximation. We have computed
this decomposition for English and Spanish corpus separately using rank k = 150
(chosen experimentally). Next, previously unseen document can be represented in latent
(topic specific) space by “fold-in” operation:
where d is vector (bag-of-words) representation of a document and d0 = wi is k-length
vector in which jwij indicates how this document contribute to i-th topic. We enclosed
those values in analytic dataset avoiding classifier overfitting using 10-folds to obtain
those topic specific features. Each 10% of documents was treated as unseen and folded
into latent representation estimated with 90% rest of corpus.
We employed a cluster analysis on two subsets of features: structural and topic specific
(defined in previous section). This is used to differentiate behavioral profiles of authors.
The basic behavior profile can be perceived as a preliminary authorship analysis; most
significantly we can distinguish human from chatter bots (similarly to Gianvecchio et.
al. [
We obtained clusters using K-Means algorithm with Silhouette score as criterion for estimation the number of clusters. Below table depicts structural centroids. N-grams on words were used in order to preserve language model. We estimated top-n n-grams for each gender-age group, next we merged the results and computed Mutual Information (equation below) to measure how much information every ngrams carry about each group.
I (C; T ) = X X P(C = c; T = t)log2 where P(C = 0) represents the probability that randomly selected author is a member of particular age-gender group and P(C = 1) represents probability that it isn’t. Similarly, P(T = 1) represents the probability that a randomly selected chatter contains a given n-gram, and P(T = 0) represents the probability that it doesn’t. 4
We tested how individual classifiers (Naive Bayes, Random Tree, SVM) as well as ensemble methods (Random Forest, Classifiers Committees) work with our features set. The random forest method gives the best results and those are presented in Table 2.
Experiment was conducted using k-cross validation with (k = 10). The final Random Forest classifier was trained on 12-core machine using about 30GB of RAM. Training took 45 minutes. Parameters of the classifier were estimated by trial-and-error: minimum samples per leaf = 5, size of feature set for each tree was equal to pn_f eatures. The classification accuracy convergence for the number of tree in the forest larger than 1000, but due to memory constraints on test machine we lowered this parameter to 666, which surprisingly fits the memory almost exactly. Also because memory constraints we did not used a n-gram model from subsection 3.9.
Among individual classifiers we tested also popular classifiers such as kNN (k = 5), linear SVM, SVM with RBF and Naive Bayes. We also conducted experiments for two simple ensembles composed of those classifiers: majority and weighted committee. Due to the time constraint we did not performed k-fold cross-validation. All classifiers were trained on 99% of feature vectors and tested on the remaining 1%. The date set was divided into training and test set by means of stratified cross-validation. The experiments were conducted on 48-core machine with 200 GB of RAM available. The obtained results are presented in Table 3.
We also performed similar tests on the subsets of features. The description of each of 9 subset can be found in Sect. 3 - "Feature engineering". We tested all of the four classifiers: kNN, Linear SVM, SVM with RBF and Naive Bayes. Further, for each subset of features we chose the best classifier and built committee from all 9 thus obtained classifiers. The results that we achieved are presented in Table 4.
The subsets of features: (I) Structural features, (II) Parts of speech, (III) Exploration of sequences of parts of speech, (IV)) Test difficulty, (V) Dictionary-based features, (VI) Errors, (VII) Topical features, (VIII) Topical centroids, (IX) Structural centroids.
Feature
Feature
Inf. gain Inf. gain 1 min_conv_len 0.0653 gram_n4_30s 0.0416 2 total_connective_words/total_sents 0.0653 gram_n5_30s 0.0363 3 avg_conv_len_words 0.0647 gram_n4_20s 0.0337 4 avg_conv_len 0.0644 gram_n5_20s 0.0246 5 total_abbreviations/total_sents 0.0642 gram_n4_male 0.0228 6 C1 0.0635 gram_n4_female 0.0228 7 gram_n6_20s 0.0631 total_uncategorized_errors/total_sents 0.0209 8 max_conv_len 0.0625 gram_n4_age 0.0207 9 C0 0.0624 gram_n5_age 0.0201 10 gram_n5_20s 0.0622 total_errors/total_sents 0.0197 11 gram_n6_age 0.0612 total_typographical_errors/total_sents 0.0177 12 total_badwords/total_sents 0.0604 new_line_count/sentence_count 0.0172 13 C3 0.0559 gram_n4_gender 0.0169 14 gram_n4_20s 0.0539 gram_n5_female 0.0163 15 gram_n6_30s 0.0524 gram_n5_male 0.0163 16 gram_n5_30s 0.0523 gram_n4_10s 0.0134 17 gram_n5_age 0.0518 gram_n5_gender 0.0127 18 total_abbreviations 0.0514 Fc_n 0.0107 19 word_count 0.0508 sps00_n 0.0107 20 gram_n4_30s 0.0503 gram_n5_10s 0.0100 21 total_badwords 0.0478 href_count 0.0095 22 total_persuasive_words/total_sents 0.0458 sentence_count 0.0090 23 sentence_count 0.0430 total_connective_words/total_words 0.0087 24 new_line_count/word_count 0.0404 Fp_n 0.0086 25 href_count 0.0397 UNK_n 0.0077 26 new_line_count/sentence_count 0.0385 href_word_ratio 0.0073 27 gram_n4_age 0.0380 new_line_count 0.0071 28 gram_n6_female 0.0369 word_count 0.0067 29 gram_n6_male 0.0369 rn_n 0.0066 30 gram_n4_male 0.0345 Fat_n 0.0061 31 gram_n4_female 0.0345 C2 0.0061 32 gram_n5_female 0.0344 Fs_n 0.0060 33 gram_n5_male 0.0344 avg_conv_len_words 0.0059 34 C2 0.0308 total_difficult_words/total_words 0.0057 35 total_difficult_words/total_words 0.0284 max_conv_len 0.0056 36 total_syllables/total_words 0.0283 C1 0.0055 37 gram_n4_10s 0.0268 new_line_count/word_count 0.0055 38 gram_n5_10s 0.0265 total_abbreviations 0.0054 39 gram_n6_gender 0.0252 C3 0.0053 40 gram_n6_10s 0.0250 ncmp000_n 0.0053 41 flasch_reading_easy 0.0241 avg_conv_len 0.0053 42 gram_n5_gender 0.0230 vmip1s0_n 0.0053 43 gram_n4_gender 0.0227 topic-85 0.0052 44 dale_chall_readability_formula 0.0216 topic-55 0.0050 45 total_badwords/total_words 0.0210 topic-17 0.0050 46 flesch_kincaid_grade_level 0.0209 topic-116 0.0049 47 total_emoticons/total_words 0.0206 topic-8 0.0049 48 total_emoticons 0.0202 topic-147 0.0048 49 total_abbreviations/total_words 0.0187 topic-36 0.0048 50 total_emoticons/total_sents 0.0180 pp1cs000_n 0.0048