Author profiling is the problem of determining the characteristics of an author of an anonymous text. In this paper, we detail a method to determine the language variety and the gender of the authors of tweets, as a submission for the Author Profiling Task at PAN 2017. This method seeks to select the most significant character n-grams for each class considered, combining them with style features for gender identification. The experimental evaluation shows that the proposed method gives good performance to determine the language variety and the gender of authors of tweets.
As computational power grows and artificial intelligence techniques evolve, more problems come to the hands of machine learning researchers. One such problem is author profiling. Different from the traditional authorship identification, in which a closed set of possible authors is known, author profiling aims to determine what are the characteristics of the authors: their age, gender, native language among others. Interest in this field has been growing. One of the reasons for this is how much text is produced in the internet, and a considerable number without a defined author behind. There are multiple practical applications on this field: forensic investigation of criminal messages; linking of certain opinions to profiles of people; better targeting in advertisement.
Since 2013, PAN organizes various tasks in author profiling, with variations on the characteristics of the authors to be determined and the source of the texts used. Each year, the task receives many submissions. In 2017, the texts came from Twitter in four languages: English, Spanish, Arabic and Portuguese. The task was classifying authors in regard to their gender and variety of the language used. This last characteristic was included in PAN for the first time, although it already has been the focus of other tasks, specifically the DSL (Discriminating Similar Languages). However, while this last one presented only two or three variations for each language, the language varieties in this year PAN author profiling task can reach seven (in the specific case of Spanish), offering a more challenging scenario.
In this paper, our approach to the problem will be described. We will introduce the data and how we preprocessed it, our classification method, present the features built from the texts, and the results of our experiments. Based on these, the features we submitted in our solution to the Author Profiling task in PAN will be described.
The data consists of a series of XML files, each one corresponding to an author, containing 100 tweets. These are in raw format, containing links, mentions to other users and hashtags. There are data to four different languages: Portuguese, English, Spanish and Arabic. These languages are divided into varieties. The varieties for each language are:
In each language, the number of files is divided equally by gender and language
variety. The split for gender is always of 50%, including for the varieties. After extraction
of all tweets from the XML, html tags, links, punctuation and mentions of users are
removed. All tweets are lowercased and treated as one single text. This text is tokenized
into words through the Natural Language Toolkit (NLTK) [
In most research on author profiling, Support Vector Machines (SVM) has been used
with satisfactory results. Comparisons with methods supposed to be more effective,
especially deep learning, revealed SVMs to be more discriminatory. Therefore, the
implementation of linear SVM in the library scikit-python [
In order to validate the approach, the data for each language was split in 66% for training and 33% for test. For the experiments on gender, this split is done in order to maintain the number of authors from each gender equal. Although the experiments are made from a subset, the classification in the final task will be made using all the training data. 3.1
Most papers use character n-grams as features in the identification of the variety of the
language of the text. Approaches diverge in the n considered, and way the grams are
used. In [
From that, n-grams were decided as the feature to be used in classification of language variety. But the high dimensionality of this sort of feature is a problem, especially due to our limited time to prepare for the task. Table 2 shows the number of distinct ngrams for all languages in the corpus, with n going from 2 to 5. For 4 and 5 grams, most results are around half a million and some more than one million. Therefore, a method was used to select a subset of the n-grams for each language, in such a way that this subset is able to differentiate between their varieties.
A language L contains j language varieties, indicated by Lp, with 0<p<j-1. Existing k n-grams in L, each one of them expressed by gi, in which 0<i<k-1, Lj (gi) is the number of times gi appears in a particular language variety. Each gram receives, for a particular variety Lp, the following ranking:
j 1 rp = X
; m 6= p
Lp(gi) m=0 Lm(gi) + 1
So, the more gi appears in other language varieties, lower will be its ranking relating to Lp. If is more frequent in Lp than other varieties, however, its ranking in this variety will be higher. Therefore, selecting the first N grams with higher values of rp will give features characteristic of Lp. 3.2
There are basically two approaches found in former works to finding the gender of an author: – Using content-based features, that reflect the particular topic the text is about, albeit powerful, they are susceptible to overfitting, if the classification is too dependent of the subject treated in the training texts; – Style-based features, reflecting stylistic structures in the writing of individual authors. They have the advantage of being less dependent of the particular conditions of a given text (topic or size, e.g.).
Usual content features used are individual words, word n-grams and character ngrams. A great number of style features appears in literature: function words, use of punctuation signs, orthographic errors, vocabulary richness, parts of speech, sentiment analysis. Some of these features, however, require specific tools to be used in a language.
For example, sentiment analysis require a Dictionary of Emotions, to find parts of speech in a text a tagger is needed. The lack of familiarity with one of the languages treated here, Arabic, and the low amount of work on another, Portuguese, makes hard to apply many style-based features present in past research, specially one of the most promising, function words.
Given that, the features used to classify texts by gender will be the same selection of character n-grams used in language variety, but considering only the amount of grams relative to if the author is male or female, as content features. As style-based features, were chosen: – number of repeated vowels; – number of hashtags used; – number of mentions to other users; – vocabulary variety, i.e., the ratio between the number of unique words and the number of total words in the text; – cleanliness of text, i.e., the ratio between number of characters it had after and before the preprocessing step. 4
This section will describe the experiments done, their methodology, the results achieved and the decisions taken for the evaluation of the Author Profiling task. 4.1
There is support in research that if the number of characters of the n-grams grows too much, performance will fall, because the terms computed will start to coincide too much with individual words. Some works suggest the average length of words in a corpus to be the maximum value of n considered to build n-grams. Therefore, the average length of words for all languages was calculated, the results presented in Table 3. The average length oscillates between 4 and 5 for all languages. So, 3 to 6-grams were built for all English and Spanish, and 3 to 5-grams for Arabic and Portuguese, in order to check their individual accuracy in classification. An optimal number N of n-grams must be chosen for be used as features to classification, after their ranking has been calculated. Choosing the first N with higher ranking for each class is the obvious approach, in which N must not be too low, so the features will not be powerful, nor too high, making the selection of a subset of n-grams useless. Figure 1 show the results of the gradual raise of N for classification on English using 3grams. As can be seen from them, the accuracy increases along with N and determining a optimal point would take doing experiments on much higher values of N, which was not possible due to the deadline.
The number of first N n-grams to be used was fixed by class used, resulting in a total of features N*number of classes for language. Each language has a different number of classes, therefore, different values of N were chosen for each one, seeking both good accuracy and a system not too heavy.
Considering the corpus size, experiments were realized using the following values to select the best ranked character n-grams. For English and Spanish, languages with larger corpus, we used 2000 features per class to model variety, and 6000 in gender. For Arabic, were used 4000 features per class in variety, and 8000 in gender. And in Portuguese, 10000 features per class to classify both variety and gender. Moreover, the gender classification used the style features described in section 3.
Defined the number of features for each language, the experiments were ran varying the size of the n-grams. Figures 2 to 5 show the results regarding language variety for each language. There is a peak in accuracy in 4 or 5 grams, with values stabilizing or falling in the next value of n. This is consonant to what was predicted in the preceding section. Given that 6-grams tend to present a fall in accuracy, 3, 4 and 5 were chosen as the size of the n-grams used in the submission.
Due to lack of time, the full tests regarding gender were made only for Portuguese and Arabic. The results are in the Figures 6 and 7. The accuracy tends to grow with the raise of n, but the increase between 4 and 5-grams is smaller. This suggests that in 5-grams accuracy for gender is stabilizing.
From the experiments, the value of N for variety and gender chose for each language is in the Table 4. Due to good results using the 2000 best ranked grams in English and Spanish, we raised the number of features for the 3000 best ranked n-grams. In Portuguese, due to the poor performance of 5-grams, we reduced the number of these features to 2500 per class.
Each text in the corpus was represented as a binary vector marking the presence or absence of a specific n-gram previously selected as a feature. 4.3
An issue that was noted when obtaining the vectors for each text: some texts would have too few of the features selected previously (some less than 100). In order to supplement such cases, 3 and 4-grams weighted through tf-idf were chosen as additional features. For Portuguese, English and Arabic, the 15000 most frequent 3-grams and 4-grams were selected, for Spanish, given the amount of texts, the 7500 first. No experiments could be conducted on these features due to time constraints. 5
The approach here defined performs well when compared with the ones in the state of
the art. In [12] and [
Regarding gender, the experiment that we managed deliver a performance
equivalent or superior to the two most recent PAN tasks in Author Profiling. In [10], the
authors report the best results ranging from 70% to 90% in the 2015 task. In [
On our submission to the 2017 Task, although we applied for all languages, a execution issue in our software involving memory resulted in only the output for Portuguese and Arabic being generated. The results are in the Table 5. The accuracy for variety in Portuguese is the best result achieved by us, and the best in all results along with three other teams. The detection of variety in Arabic was rather poor, being the third worst result. The results for gender were in the average range for both languages. These results follow closely the ones obtained in the experiments, with exception of gender for Portuguese, that exceeded in more than 6% the experimental results.
Portuguese was the language with the least number of varieties, so would naturally be the one with best results regarding this task. It was the one with more number of character n-grams by variety selected, so the model is a better classifier. Although the total number of n-grams used for training the classifiers for Portuguese and Arabic was similar, the amount by specific variety in this latter language is much lower, presumably reducing the discriminsating power of the model trained.
Moreover, the rankings for the n-grams for each class are obtained comparing their frequencies in that class with the ones from all the other classes. In Portuguese, with only two varieties, this is not a problem, but in Arabic, there is the possibility that information from some of the other classes was ignored or neglected, and only some relations between classes considered. Therefore, for future works, we propose investigation of selection of the character n-grams combining varieties in pairs in languages with more than two varieties. 10. Rangel, F., Rosso, P., Potthast, M., Stein, B., Daelemans, W.: Overview of the 3rd author profiling task at pan 2015. In: CLEF (2015) 11. op Vollenbroek, M.B., Carlotto, T., Kreutz, T., Medvedeva, M., Pool, C., Bjerva, J.,
Haagsma, H., Nissim, M.: Gronup: Groningen user profiling (2016) 12. Zampieri, M., Tan, L., Ljubešic, N., Tiedemann, J., Nakov, P.: Overview of the dsl shared task 2015. In: Joint Workshop on Language Technology for Closely Related Languages, Varieties and Dialects (LT4VarDial). pp. 1–9 (September 2015) 13. Zampieri, M., Gebre, B.G.: Automatic identification of language varieties: The case of portuguese. In: KONVENS2012-The 11th Conference on Natural Language Processing. pp. 233–237. Österreichischen Gesellschaft für Artificial Intelligende (ÖGAI) (2012)