Author profiling problem is about detecting some characteristics (age, gender, for example) of the author of some piece of text depending on the features (eg. lexical, syntactical) of this text. Men and women, and of different ages, write in different ways. Having a dataset in hand, written by different authors of different characteristics, we can train the machine using this dataset so it can predict these characteristics of an unseen piece of text fed to it. PAN 16 1 author profiling task provides a dataset of tweets for the sake of developing an author profiling system. The task is about predicting the age and the gender of the author. Machine learning technique suits to achieve this goal. Support Vector Machines (SVMs) [3] can be used as a multi-class classifier which could be trained using the dataset provided to produce a model which can be consulted on an unseen set of tweets written by some author to predict his age and gender. Bag-of-Words (BOW) [14] is a simplified representation of the text corpus which contains all the words used in it with their frequencies. BOW representation is used in many areas like Natural Language Processing [13], Information Retrieval [5], Document Classification and among others [14]. In our work we use SVMs and BOW representation. We use the python machine learning library, scikit-learn [7]. After we produced the best possible model trained on PAN 16 author profiling dataset, we ran some tests over the test sets provided by Tira [2,9]. The work presented in this paper was reviewed and is part of the PAN 2016 overview [11].
This paper is organized as follows: in section 2, the Implementation is described; in section 3, we present the results with features selected and evaluation criteria; in section 4, a retrospective analysis of the work is preformed and a future vision is suggested.
In this section we describe all the steps of creating the model. We first analyse the dataset, then we present the architecture of the system and at the end we explain the implementation of it.
We used the dataset2 provided by the PAN 2016 in our study. The corpus contains 436 files, each file contains a set of tweets and these files are written by different authors. The information about each file written by which author is indexed by a file called turth file. The file structure is shown in (1) and is explained in Table 1.
AID ::: G ::: AR The Table 2 shows the distribution of the data after analysing it. For example, the corpus contains 14 files written by female authors which have ages between 18 and 24.
Our system has three modules: preprocessing, training and testing modules. In figure 1 we show the architecture of the system in the training phase. In figure 2 we present the architecture of the system in the testing phase. Both of them use the preprocessing module. Social Media like twitter is a very noisy environment where informal texts can thrive. As the space is noisy and it does not comply with the syntactic rules of the natural language, NLP (Natural Language Processing) [13] can not be exploited to the best extent.
In our study, we use the BOW [14] representation of the corpus as set of features. Before the BOW generation the data had been transformed. The objective is to opti-Figure 2: The architecture of the system: testing phase mize the BOW representation by reducing the words set of the corpus without losing information. This preprocessing is done in three steps.
The data in the corpus comes from twitter and has the nature of being noisy containing a lot of abbreviations and special expressions. These special expressions can hold important clues that can differentiate the characteristics of the authors. A regular expression parser has been created in order to replace all of these special expressions with predefined tags. This first step allows to group expressions and reduce the words set, without losing information. The list of tags with few examples of tokens replaced by them is shown in Table 3.
The Second step consists of removing the stopwords form the corpus. Stop words are a set of words like prepositions ("in", "on", "to") and conjunctions ("and", "or"). Usually they carry no information and they are used a lot in the context. The Natural Language Toolkit (NLTK) [1] has a list of English stop words and the scikit-learn [7] too. In the work presented, two lists were merged and used to filter out the corpus.
The third step in the stemming. Stemming [6] is the process of finding the root (the lemma) of a given word. Stemming is used in Information Retrieval [5] such that, for example, words like "connect, connected, connecting, connection and connections" would be considered as one search word which is the stem of these words "connect". It is useful for the BOW representation such that it reduces the number of the tokens as it may reduce many words to their root and use them as if they were one word. NLTK provides many algorithms for stemming. We used the SnowballStemmer [8] algorithm in our work. The result of the preprocessing module is the BOW model as a list of lists such that each list represents a file of the dataset. The list length is equal to the number of features chosen. The numbers in the list represent the frequency of each word of the Bag-of-Words (the features) in each file in a descending order.
Our training module is the core of the work done. It uses the data preparing module to convert the training dataset to the BOW representation as explained before. Each word in the BOW is considered as a feature. We do not use the whole BOW as features but we limit the number, this will be discussed in the result section. After getting the BOW representation of the dataset we divide it into two parts; one part for training (it is two thirds of the whole dataset and we call it the development set) and another part for testing (it is one third of the whole dataset and we call it the evaluation set). We divide the dataset using the scikit-learn function train_test_split.
Then, this module seeks to get the best parameters to train an SVMs classifier on the development set. The parameters we seek to get for our SVMs classifier are the kernel, gamma and C parameters. To achieve that we do a hyperparameter tuning through a grid search provided by the scikit-learn library using GridSearchCV function. We define a set of parameters to be used by the grid search function as we show in Table 4.
Grid search uses stratified cross validation once for each pair of the parameters provided keeping track of the results it gets. We used a k-fold cross validation with k = 3. It is more usual to use this technique with k = 10 but due to the small number of some classes, it was not possible as can be seen in Table 2 there are some age ranges with only 3 elements (files). In other words, with classes of small number of files, it was not possible to apply a stratified cross validation with k = 10 correctly.
With "rbf(radial basis function)" kernel in our work. We explain how grid search works by a pseudo code (Code 1).
After we get the model trained on the development set using the best parameters we do a test on the evaluation set. Getting the result of this test, we produce a classification report to show the results in terms of precision, recall, f1-score and support. This will be discussed in the result section. We then used the best parameters we obtained from the grid search to train a classifier on the whole PAN 16 dataset and produce our model which we used to do the Tira tests.
This module will take again the benefit of the preprocessing module to get the dataset in a suitable format (BOW representation) to consult the model was produced by the training module. It will consult the produced model to predict the age and gender of the author of each file of the test dataset and it will produce an XML file for each one of them. The description of the XML file format is shown in Description 1. The set of XML files will be the input of the Tira evaluation where accuracy will be calculated as a performance measure.
We hint here that our system was developed just for English language.
We did many tests over many datasets (the evaluation sets of them) using different sets of features. Our features, as we mentioned before, are the words formed by the BOW such that each word is considered as a feature (taking the frequency of it in each document).
Our results are produced using the classification_report, provided by scikit-learn, over the testing results on the evaluation sets. After we obtain the model using grid search over the development set, we use it to predict over the evaluation set and we run the classification report over the result of prediction. Classification report takes the real target and the predicted target to calculate the precision, recall, f1-score and support for each class was predicted and it calculates the average of these metrics.
First we present some results of the tests on the PAN 16 dataset which has ten classes.
In table 5, we show the results after using a number of features equal to 10000. In table 6, we show the results after using a number of features equal to 100.
We hint here that the class 10 does not appear in the classification report and that is because of the PAN 16 dataset which contains 436 files, has only 3 files of this class and the way we divided the dataset into a development set and an evaluation set did not give any file of the class 10 to the evaluation set. Now we show some results of tests we did on the PAN 15 dataset which has only eight classes. Using a number of features equal to 10000, we present the results in Table 7.
And in Table 8 we present the results after using a number of features equal to 100.
We further discuss the results in the conclusion.
We decided to use the BOW representation as features for our classifier after observing the nature of texts in the social media like twitter. The process of making a parser to replace the special pieces of texts which may mean important in this kind of text and making the BOW (after stemming and stopwords removal) of the resulting tagged text may suit well for this task. But, selecting the right features for SVMs is not an easy task. There are many issues that should be taken into consideration. The scale range of each feature can be a problem [4]. We notice that the results were better for PAN 15 than for PAN 16. That could be because of the tagging process, when we tag the dataset to match special mentions like links and smiles, these special mentions could be found more often in the PAN 15 dataset than in the PAN 16 dataset. In other words, the tagger behaviour is not guaranteed and that depends on the essence of the dataset.
We also notice, from these tests on PAN 16 and PAN 15 datasets that increasing the number of features does not mean necessarily better results. For example, when we used a number of features equal to 100 in the test done for PAN 16 dataset, we got a precision equal to 0.3 and we got the same value of precision for a number of features equal to 10000 for the same test. This is known as the Peaking Phenomenon [12] (PP) and it can occur when using a high number of features. The performance of a model is not proportional to the number of features used, there is a point where the performance deteriorates when more features are added to the model. Procedures already presented in Section 2 like preprocessing the text using tagging, stopwords removal and stemming before creating the BOW representation can help to minimize this problem.
There are many things that could be done or improved in order to continue this study. A true Random Search could be implemented in order to improve the features selection and parameters tuning. It could also be improved by adding features extracted with respect to the natural language (syntactic and semantic features, for example). Natural Language Processing [13] can be exploited to achieve that. But as we mentioned before it may not be possible to exploit it to the best extent as the nature of this environment is noisy.
The use of the term frequency-inverse document frequency (tf-idf) technique [10] and tuning the maximum size of the BOW can help too. In fact the scikit-learn provides the necessary functions to use tf-idf technique and it could be a good experiment to do as a future work.
We want to address our thanks to the Departamento de Informática da Escola de Ciências e Tecnologia da Universidade de Évora, for all the support to our work.