This paper describes two methodologies applied to the author profiling task submitted to the PAN 2013 competition of the CLEF 2013 conference. The first methodology was applied only to the English language, whereas the second one was executed only over the corpus written in Spanish language. The aim was to evaluate the performance of both methodologies in the above mentioned task. The obtained results were quite positive for the first methodology which considers a classicaly approach of classification, using diverse features extracted from the texts in order to feed a classifier based on random forests. The second methodology, based on graph mining techniques, obtained a very poor performance for the author profiling task.
1.1
For the English corpus, we applied a methodology based in classical techniques of machine learning. The set of features were extracted in order to feed a Random Forest classifier. Figure 1 shows the methodology used for this corpus which is twofold: pre-processing and classification.
In the pre-processing step, we attempt to normalize terminology by replacing unrecognizable terms, smiles, and weird symbols (e.g. URLs, pictures) from the dialogues by their corresponding normalized term. In order to apply this normalization procedure, we used three lexical resources that we have constructed for this purpose. The lexical resources are described as follows: 1. Emoticons: A list of emoticons constructed on the basis of a preliminar dictionary1, and enriched by adding the predefined emoticons of Windows
Messenger, Facebook and Gmail. The dictionary constructed contains 344 entries. 2. Contractions: The list contains around 65 of the most used contractions in the United States. 3. Dictionary: A English dictionary in TXT format. This list was used for determining unrecognized words.
All the term occurrences in the dialogues of the training and test set that matches with some entry in “emoticons” or “Contractions” were replaced. The “dictionary” lexical resource was used only for determine the existence of a given term of the conversation.
In the classification process we used the frequencies of the following sets of features: – Emoticons – Contractions – Conversation length (in words) – Conversation length (in characters) – Mispelled words – Average length of words in the dialogues – Words capitalized – Words in uppercase – URLs – Each different POS tag – Each different suffix – Each different punctuation symbol – Each stopword
All these features were used for representing each one of the dialogues in the training set which further were used for feeding a Random Forest classifier (included in the WEKA tool[1]). The gender was used as the classifier attribute (class) for determining whether a given dialogue was written by a male or a female person.
A second classification model consider the discrimation of the age. In this case, the classifier attribute is the range of age (“10s”, “20s” or “30s”) given at the competition. The result of the first classifier (gender classifier) determines which type of the second classifier will be used, the one that was trained only with a corpus of male persons, or the one that was trained only with female persons. In summary, we obtained three classification models: 1. Classification by gender (Gender.model ). 2. Classification of age range for male persons (AgeMale.model ). 3. Classification of age range for female persons (AgeFemale.model ).
We extracted the sets of features in all documents and three models was create, Gender.model, AgeMale.model and AgeFemale.model. The system for the classification of the test dataset takes into consideration the following steps: 1. The extracted dialogues of the test corpus were preprocessed using the lexical resources in order to normalize the texts. 2. We obtained the set of features for the gender classifier. Afterwards, we classified the test dialogues for obtaining a gender label for each dialogue (“male” or “female”). 3. We separated the dialogues according to the gender label, then, we classified the female dialogues with AgeFemale.model, and the male dialogues using AgeMale.model. 4. Finally, each test dialogue has two categories assigned, age and gender. Using these categories, the system prepare the sytem output in XML format.
The results obtained in the competition are given in Section 2. 1.2
The methodology proposed for the Spanish corpus focuses on the use of graphs as a strategy for feature extraction. Moreover, this methodology uses the extracted features with the purpose of feeding a supervised classification algorithm which allows to determine the gender and age of the authors. As carried out with the English corpus, we performed a pre-processing step in order to normalize the input texts. Afterwards, the texts are represented by means of graphs which will be further used for extracting relevant features. Graphs are mined using the SUBDUE tool2. The obtained results of the mining graph phase are used as features in machine learning algorithms with the aim to obtain a classification model.
A graphic idea of the solution scheme is shown in Figure 2.
Graph Generator The Graph Generator module receives the training set split into two subsets, one made up of the dialogues labeled with age classes, and the other one labeled with the gender classes. A graph-based representation for each class is built using a star topology. The star topology allows depict text in sentences. This graph-based representation generates a vertex in the graph for each word in the sentence. Furthermore, the representation establishes the relationships between words through of the edges of the graph. A graphical scheme is shown in the figure 3
Before the generation process begins all dialogues are tagged used the freeling3 tool with the aim of finding the word lemmas and the Part-of-Speech tags (POS tags). Thereafter, an order between words is established; that given order is symbolized by a generic conector “next”. The process begins establishing an inicial vertex v0 representing a sentence, from which adjacent vertices are derived v1, v2, . . ., vk, where each one represents a lemmatized word, and k represents the number of words in the sentence. Each edge having an inicial node v0 and
a final node vk, with k = 1, . . . , no words, is assigned with a “POS tag” representing the POS tag obtained with Freeling. Finally each edge having an initial node vi and final node vi + 1, with i = 1, . . . , k − 1 is assigned with the generic conector “next”.
This module generates a graph-based representation for each class of the gender set (male-female) and the age set (10s,20s,30s).
Feature Extraction The feature extraction is conducted over all graph-based representations; the first one over the gender set (male and female), whereas the second on is conducted over the age set (10s, 20s, and 30s). Finally, based on the analysis of the extracted features of those two sets, a feature set is built for the combined class (10s-male, 10s-female,20s-female,20s-female,30s-female,30sfemale).
The feature extraction process over all graph-based representation is done by mining those graphs with the SUBDUE tool. The result of the mining process is analyzed as follows: data containing the highest support values are considered, that is, the substructures found from the graph mining process that frequently appear are used. For this particular step, the MDL measure of the mining tool is used. The results of each mining process for the age set and the gender set are single words or n-grams of words that exclusively appear in one class of the set, but not in the other. For example, words or n-gram that appears in the male set, but no appearing in the female set and viceversa. The feature vector obtained is later used by a supervised classifier.
The test set feature extraction is performed based on the features obtained with the train set, i.e., every feature in the train set is extracted and counted in the test set, generating a feature set with an unknown class.
Classification Process This module receives the train feature vectors with its corresponding class obtained in the previous module, i.e. D = (C1, C2, . . . , Cn), where Ci represent a particular characteristic and n is the total number of the characteristics. We have used the random forest classification algorithm, included in Weka4 tool in order to obtain a classification model for the six classes (10s-male, 10s-female,20s-female,20s-female,30s-female,30s-female). Thereafter, the test Set feature vectors are evaluated with the generated model, and every vector is labeled with one age-gender class. 2
In this section we present the results obtained with the two proposed methodologies. First we describe the dataset used in the experiments, and thereafter, the results for each language.
The English training corpus contains 236,600 dialogues of different types of authors. The corpus is balanced with 118,300 dialogues per gender. Table 1 shows some characteristics of this corpus.
The male gender part of this corpus contains a larger vocabulary than the female one, however, the female part of this corpus is more extensive in average of words per dialogue.
The Spanish training corpus is structured as shown in the table 2. The number of dialogues is much more smaller than the English one.
In this edition of the PAN competition, we were required to submit the entire system to a virtual machine in which the execution of the experiments will be carried out. We did not have access to the test dataset, therefore, the description of this corpus is not given in this paper. 2.2
Tables 3 and 4 show the results obtained at the competition for the English and Spanish corpus, respectively. As already mentioned, the first methodology was only applied to the English corpus. In this case, we can see that the Accuracy obtained is 0.5923 which rank the system in the 7th position. However, the second methodology performed even worse than the baseline, with an Accuracy of 0.2915. As future work, we would like to evaluate the first methodology in the Spanish corpus as well in order to determine its performance with the Spanish language.
We have presented two different methodologies for tackling out the author profiling task. The main difference between the two approaches is the feature extraction process. The first approach uses a number of features which we consider to be related to the two classes to be discriminated (age and gender). The second approach is based only on the word frequencies and POS tags, but uses a graph-based representation for extracting n-grams of words.
We succeed in the first approach obtaining the 7th place in the competition, but the second one was not able to capture regularities or patterns from the graphs. The results obtained by the first approach indicates that the features selected allow to discriminate gender and age of a given author with an Fmeasure of 0.59. We are interesting in evaluating the first methodology presented in the Spanish language. baseline ayala13 mechti13 weka data mining software: an update. SIGKDD Explor. Newsl. 11(1) (November