In this notebook, we describe our approach to cope with the Author Profiling task on PAN17 which consists of both gender and language identification for Twitter's users. We used our MicroTC (µTC) framework as the primary tool to create our classifiers. µTC follows a simple approach to text classification; it converts the problem of text classification to a model selection problem using several simple text transformations, a combination of tokenizers, a term-weighting scheme, and finally, it classifies using a Support Vector Machine. Our approach reaches accuracies of 0.7838, 0.8054, 0.7957, and 0.8538, for gender identification; and for language variety, it achieves 0.8275, 0.9004, 0.9554, and 0.9850. All these, for Arabic, English, Spanish, and Portuguese languages, respectively.
Recently, forensic text analysis about originality, authorship, and reliability has attracted a lot of attention by researchers and practitioners because of practical applications in security and marketing [19]. In this context, author profiling is an important task of PAN@CLEF forum that focuses on analyzing some characteristics of the author (profiling aspects) based on the written author's text, such as gender, age, political preferences, personality, language variety, among others [14].
Generally speaking, author profiling task is tackled using, mainly, machinelearning approaches, i.e., models, for predicting profiling aspects, are built considering a set of general features that represent different categories of authors, e.g., gender, range age, and language variety, among others [16].
PAN forum 2017 3 provides a dataset of tweets for training and test the performance of each participating system. In this edition, the profiling aspects to be analyzed are gender and language of Twitter's users. The corpus is annotated with authors' gender and their particular variation of their mother tongue that includes Arabic, English, Spanish, and Portuguese.
Our approach is language independent, that is, we deliberately avoid the use of linguistic procedures such as part-of-speech tagging, lemmatization, or stemming. In the same way, linguistic resources, like lexicons and WordNetbased, are disallowed. In contrast, we take advantage of multiple tokenizers, an entropy-based term-weighting scheme, and an SVM classifier, see Section 3 for details.
The rest of the paper is organized as follows. Section 2 presents few of the gender, age, language, and region identification related works, and Section 3 describes our system and the general approach to model the problem. Section 4 detail the experimental methodology and the achieved results. Finally, conclusions and future work are given in Section 5.
Author profiling is a repetitive and important task in PAN contest since 2013 [16]. Before 2017 edition, only age and gender classification tasks were considered [17,18]. This year, the PAN considers the region aspect while removes the age identification subtask from the competition [14].
Several works have been proposed to solve age and gender identification subtasks. Agrawal & Gonçalves [1] use a combination of classifiers along with a model based on user's activities to predict the profile of the unknown users. The TFIDF representation was employed, and a dimension reduction was performed in this matrix. The authors use Naive Bayes and Linear SVM as classifiers.
With the purpose to find the differences between writing styles of males and females in different age groups, the usage of several stylometric features is considered in [4]. Another stylometric approach was presented in [5] where two groups of features were considered, trigrams and complementary-weighted Second Order Attributes. An SVM classifier is used in the classification step. A combination of features based on word n-grams, sentences starting with capital letters, finish the sentences with a dot, emoticons, word's length and sentence's length is also used along with grammatical aspects are explored in [23].
Lopez-Monroy et al. [12] propose a representation for documents that capture discriminative and subprofile-specific information of terms. Under the proposed representation, terms are represented in a vector space that captures discriminative information. On the other hand, more traditional representations, like TFIDF, are broadly employed in the author's profiling literature, that is the case of [8], [22], and [13]. Classification ensembles are also frequently used; for instance, [24] generate several classifiers using sets of features such as word ngram, character n-gram, and part-of-speech n-gram features.
Language variety identification is a new subtask introduced in PAN17 that consists in determining the specific variation of the native language of authors' text [11,10]. Another approach to region classification is presented in [7] where twitter geolocation and regional classification was conducted through sparse coding and dictionary learning. Another region prediction approach based on Modified Adsorption, removing "celebrity" nodes and analyzing a graph model propagation is proposed in [15].
MicroTC (µTC) is a generic framework for text classification task, i.e., it works regardless of both domain and language particularities. µTC is an extension of our previous work on sentiment analysis, see [20]. A full description of µTC can be found in [21]. The core idea behind µTC is to tackle a text classification task by selecting an appropriate configuration from a set of different text transformations techniques, tokenizers, and several weighting schemes, using as a classifier a Support Vector Machine (SVM) with linear kernel. In some sense, the text classification problem is transformed into hyper-parameter optimization, also known as model selection.
Briefly, µTC contains the following parts: i) a list of functions that normalize and transform the input text to the input of tokenizers (preprocessing), ii) a set of tokenizer functions that transform the filtered text into a multiset of tokens, iii) a function that generates weighted vectors from the multiset of tokens; and finally, iv) a classifier that knows how to assign a label to a given vector.
i. Preprocessing functions We use trivalent and binary parameters. The trivalent values can be set to {remove, group, none} which means that the term matching the parameter is removed, grouped in set of predefined classes, or left untouched. In this kind of parameters, µTC contains handlers for hashtags, numbers, urls, users, and emoticons. The binary parameters are boolean, and basically, indicate if the parameter is activated or not. In this parameter set, we support for diacritic removal, character duplication removal, punctuation removal, and case normalization. ii. Tokenizers After all text normalization and transformation, a list of tokens should be extracted. We allow to use n-grams of words (n = 1, 2, 3), q-grams of characters (q = 1, 3, 5, 7, 9), and skip-grams. For skip-grams we allow to select a few tokenizers like two words with gap one, (2, 1), also we allow to use (2, 2), (3,1). Instead of selecting one or another tokenizer scheme, we allow to select any combination of the available tokenizers, and perform the union of the final multisets of tokens. iii. Weighting schemes. After we obtained a multiset (bag of tokens) from the tokenizers, we must create a vector space. MicroTC allows to use the raw frequency and the TFIDF scheme to weight the coordinates of the vector. It contains a number of frequency filters that were deactivated for this contribution, see [21] for more details.
iv. Classifier We decide to use a singleton set populated with an SVM with a linear kernel. It is well known that SVM performs excellently for very large dimensional input (which is our case), and the linear kernel also performs well under this conditions. We do not optimize the parameters of the classifier since we are pretty interested in the rest of the process. We use the SVM classifier from liblinear, Fan et al. [9].
We select to model a user using all its tweets, that is, an user u is a collection of small texts u = {t 1 , . . . , t n }. For each text, we apply the preprocessing step and tokenizers, then we create a multiset from the union of all multisets in u. After this, a vector u is created using a term weighting scheme. Thus, we modeled each user as a high dimensional sparse vector. For instance, since we do not remove any kind of terms, and in fact we promote the usage of combinations of tokenizers, the user's vectors can contain millions of coordinates, and thousand non-zero entries.
The weighting schemes for this modeling are described in the following paragraphs. We also introduce entropy+b, a new weighting scheme introduced in this notebook designed for classification tasks. In the following paragraphs we describe in detail the weighting schemes used in the experimental section.
The simpler scheme corresponds to freq, and it is defined as the term frequency of each term per user; we name it freq usr to avoid confusion with other functions. TFIDF is the product of TF and IDF where TF is the normalized frequency of a user's term, and IDF is the inverse document frequency defined as the logarithm of the inverse of the probability that a term occurs in the whole collection of users, more precisely, TF(w, usr) = freq usr (w) max w∈usr {freq usr (w)} , and
where N is the size of the training collection, i.e., the number of users. It is common to add 1 to the denominator expression to avoid numerical problems.
In this notebook, we introduce the entropy+b term-weighting that considers that each term is represented by a distribution over the available classes. Instead of using the raw probabilities per class, we weight each term with the Entropy+b function, defined as follows: Here, freq c denotes the frequency of the given term in the class c. The idea behind entropy b (w) is to weight each term using the entropy of the underlying distribution in a way that large entropy values (terms uniformly distributed along all classes) have a low weight while terms being skewed to some class are close to log |C|. The parameter b is introduced to absorb the possible noise that occurs in low populated terms.
The model selection is lead by a performance function score that is maximized (solved) by a meta-heuristic. The only assumption is that score slowly varies on similar configurations, such that we can assume some degree of locally concaveness, in the sense that a local maximum can be reached using greedy decisions at some given point. Clearly, this is not true in general and the solver algorithm should be robust enough to get a good approximation even when the assumption is valid only with some degree of certainty. From a practical point of view, a configuration is similar to another if structurally vary in a single parameter. We name the set of all similar configurations of m as its neighborhood. Therefore, the core idea is to start from a set of random configurations, evaluate their neighborhoods and greedily move to the most promising set of configurations, The procedure is repeated until some condition is achieved, like the impossibility of improve the score function, or when a maximum number of iterations is reached. There are several meta-heuristics to solve combinatorial optimization problems, the proper survey of the area is beyond the scope of this notebook; however, the interested reader is referred to [20,21,6,2].
In particular, µTC uses two types of meta-heuristics, Random Search [3] and Hill Climbing [6,2] algorithms. The former consists in randomly sampling C and selecting the best configuration among that sample. Given a pivoting configuration, the main idea behind Hill Climbing is to explore the configuration's neighborhood and greedily move to the best neighbor. The process is repeated until no improvement is possible. We improve the whole optimization process applying a Hill Climbing procedure over the best configuration found by a Random Search. We also add memory to avoid a configuration to be evaluated twice 4 .
The experiments with the training set were run in an Intel(R) Xeon(R) CPU E5-2640 v3 @ 2.60GHz with 32 threads and 192 GiB of RAM running CentOS 7.1 Linux. The gold-standard were evaluated in the TIRA platform using a virtual machine with 4GiB of RAM and one core. We implemented µTC5 on Python.
We partition the full training dataset into two smaller sets, a new training set containing 30% of the users, and a validation set with the resting 70%. The partition where selected to ensure the generalization of our scheme. On the new training set, from now on just training set, we run µTC using random search and hill climbing to perform the hyper-parameter optimization. Random search was allowed to select 32 random configurations. On the other hand, Hill-climbing starts with the best configuration found by random search; the procedure was left to finish its optimization process. We use 3-fold cross validation for the model selection procedure. Once the model selection finished, we use the configuration found to train a µTC machine with the whole (small) training set and measure the performance of that classifier in the validation set.
Table 1. Performance of our approaches for gender using 30−70% partition for training and test datasets.
Table 1 shows the performance of µTC for gender identification. In particular, we show macro-recall, macro-f1, and accuracy scores. We show three different term-weighting schemes, detailed in §3. We select the FREQ scheme to describe the improvement of each scheme. The FREQ and T F IDF schemes are implemented in µTC; for entropy+b, we show the performance for five different values of b. Table 1 indicates that T F IDF performs poorly as compared with FREQ. Entropy+b illustrates the dependency of b, showing better performances for small b values, except b = 0 which has a poor performance for gender identification. The table shows that b = 3 and b = 10 performs much better than the rest of the classifiers. Between entropy+3 and entropy+10, the first one performs better; however, entropy+3 was evaluated after the deadline of the second run. Therefore, entropy+10 was used to classify the gold standard, see Table 3. Table 2 shows the performance of our systems in the language variety task. As before, we use FREQ as the baseline method. In this task, FREQ also performs The official performances on the PAN17 gold standard are shown in Table 3. We send our baseline based on the FREQ weighting scheme and the profiler based on entropy+10. The table indicates the accuracy for gender and variety tasks, as well for the joint accuracy (the same example was correctly predicted in both tasks). As predicted in Tables 1 and 2, entropy+10 has a better performance than FREQ, in some languages by a large margin, e.g., close to five percentual points for Arabic, and six percentual points for Portuguese.
In this notebook, we describe the INGEOTEC's system used to solve the Author Profiling task in PAN17. We used our MicroTC (µTC) framework [21] as the primary tool to create our classifiers. µTC follows a simple approach to text classification; it converts the problem of text classification to a model selection problem using several simple text transformations, a combination of tokenizers, a term-weighting scheme, and an SVM classifier. It is designed to tackle textclassification problems in an agnostic way, being both domain and language independent.
To effectively tackle the task, we introduce a new term-weighting scheme based on the distributional representation of each term and the entropy over that distribution. We call it entropy+b. More work is needed to characterize the new weighting scheme yet it demonstrated to be superior to raw term frequency and TFIDF, at least, for the Author Profiling task and our µTC framework.
We like to thank the PAN organizers, in particular to Francisco Rangel and Martin Potthast for their kind and quick response to our questions and requests.