We present our participation in the PAN 2018 Author Profiling shared task, classifying authors on gender for English, Arabic and Spanish. We participated in all sub-tasks and propose a system for classification with text, images and the combination of those two. Our final submitted system is a Logistic Regression classifier that uses word and character n-grams as textual features and a set of automatically derived image-based features such as the presence, proportion and number of faces to detect selfies as well as the faces' emotions and gender. We experimented with word embeddings, which negatively affected our system's performance. Our cross-validated training results shows slight improvements in performance for Arabic and Spanish when image-based features are added to text-based features. Our highest scores on the PAN 2018 test dataset are accuracies of 81.2% for English using only text-based features, 78.7% for Arabic using both text-and image-based features and 80.3% for Spanish using only text-based features. Overall, we finished 6 th in the global ranking with an average accuracy for our text and image combination system of 79.6%.
The field of author profiling is about inferring traits from an author such as gender, age and personality. With the rise of social media platforms, such as Twitter and Facebook, the field of author profiling has gained more interest. From multiple viewpoints, it is desirable to profile an author. Examples of such viewpoints could be from a security point of view in order to detect authors with criminal intentions and from a marketing point of view in order to narrow down target audiences for online advertisements.
In the past years, multiple shared tasks have been organized on the topic of author profiling [18,16]. In this paper, we describe our approach for the Author Profiling shared task at PAN 2018 [17]. This year's Author Profiling task, is the 6 th iteration of this task and is slightly different from the previous years, since the gold standard data now includes images. The task is to built a system to classify a Twitter author's gender by 100 of its tweets and 10 posted images. Though the images are new for this shared task, previous work already created systems that are capable of detecting gender, emotional expressions and personalities from images [2]. By combining such image classification systems with textual classification systems, it can be determined whether this addition of images can improve the final accuracy.
In the last two years, the winning systems for 2016 [22] and 2017 [3] were both SVM classifiers that made use of word n-grams and character n-grams. Although deeplearning methods were introduced, such as Recurrent Neural Networks [7] and Convolutional Neural Networks [19,20], they haven't been able to beat those systems yet. Therefore, our approach will focus on the successful models of the previous iterations of the shared task, a SVM classifier such as in [3] and a Logistic Regression classifier as used in [10], in which we will take these systems as baselines and try to improve them by performing a parameter search, experimenting with word embeddings and adding image-based features. The latter includes a feature to indicate selfies, since females tend to post more selfies than males [5,21].
The PAN 2018 training corpus consists of tweets from three different languages, English, Spanish and Arabic. For each author there are 100 tweets and 10 images labeled by gender. The gender labels (male and female) are evenly distributed over the training corpus. Table 1 shows an overview of the PAN 2018 training corpus released by the organization.
Arabic 150,000 1,500 15,000 English 300,000 3,000 30,000 Spanish 300,000 3,000 30,000
The main set of features we used were n-grams. The winners of the previous year's Author Profiling shared task [3], as well as [1,4,6,9,10,12,13,19] showed that word ngrams and character n-grams are very robust features for this task. Another advantage of using n-grams is that they are non-handcrafted features and thus easy to generate. Also, there is no dependence on either pre-trained word embeddings, or large corpora of text to train word embeddings. For all three languages, we experimented with different lengths of word and character n-grams.
Aside from the n-gram features, we experimented with using word embeddings. For English, we experimented with pre-trained word embeddings from GloVe [15]. We used embeddings with vector lengths of 100 and 200 dimensions that were created from a corpus consisting of 2 billion tweets containing 27 billion tokens. For Spanish, we used pre-trained word embeddings from [8] with vector lengths of 200 dimensions. These embeddings were constructed from a total amount of 58.7 million Spanish tweets having 1.1 billion tokens. For Arabic, we trained our own word embeddings from roughly 70 million recently scraped Arabic tweets with vector lengths of 200 dimensions.
This year's new addition to the gender classification task is classification by images. Our approach to use the images for this task is to utilize existing image feature extraction tools from related research. In our system, we have used the software used in [2].
In that study, a convolutional neural network (CNN) was implemented to find and classify faces in images by gender and emotion. The CNN model contains of 4 residual depth-wise separable convolutions, whereby each convolution is followed by a batch normalization operation and a ReLU activation function. The last layer of the model applies a global average pooling and soft-max activation function to produce the prediction. The system achieved an accuracy of 95% on the IMDB gender dataset and 66% on the FER-2013 emotion dataset. That system, including all code and pre-trained models are available under an open-source license. 1The software from [2] was implemented in our system without preprocessing the images. The software converts the images from a Twitter user to a set of 13 features: The first two features are about the presence and number of detected faces in the images. In Table 2 are the number of faces detected for each language and gender. We see that there is little to no difference between the genders for which a face is detected.
Features three and four cover the relative area of images that are covered by a detected face. These features are intended to capture selfies. Previous research [5,21] studied selfie-related behaviours between males and females. One of the findings from those studies is that females tend to make and post more selfies on social media. Having a large face area on an image with only one detected face could identify that an image is a selfie, and therefore these features could be helpful in the classification task.
Features five and six are about the gender of the detected faces. The values of these two features are floats ranging from 0 to 1, indicating the proportion of each gender. In Table 3 are the probabilities for a gender posting more faces of a specific gender in a image. We see that for all languages, males are posting more images of male faces than female faces, especially for Arabic there is a large difference in male and female faces. English and Spanish females are slightly posting more images with female faces than male faces, except for Arabic, in which female user post more male faces than female faces, but still to a lesser extent compared to their male counterparts. Lastly, when one of the seven emotions from [2] could be detected, which the software was not always capable of, we stored the proportion of these emotions in seven float values ranging from 0 to 1. Table 4 shows an overview of the emotions for each gender and language. The table shows that, generally, there are small to no differences between male and female regarding emotions. The only conclusion that holds for all languages is that females tend to post more happy people. For English, males post more images of angry people, but for Arabic this is the opposite. Also, no one is ever surprised, raising the question whether the system of [2] can accurately detect this. Overall, we expect that these features will not be (very) beneficial for our system.
To get to our best system, we experimented with different classifiers. We used the Python package Sklearn [14] to implement the LinearSVC classifier as used in [3] and the Logistic Regression classifier with the parameters C = 1e2 and fit_intercept = False as used in [10], we also tried a K-Nearest Neighbour classifier. The results of all tested classification models can be found in Table 5. For every model,we measured its performance by accuracy in a 10-fold cross-validation setup. The models are all using the n-gram features used as in [3]. We found that using the Logistic Regression classifier resulted in the best performance, meaning we will use this classifier for our next experiments. For the logistic Regression model we performed a parameter search, mainly to find the optimal number of word and character n-grams. Our baseline n-gram model was the n-gram model used in [3], which was using word 1-and 2-grams and character 3to 5-grams. We tested different settings of n-grams but we were unable to outperform the settings from [3]. Table 6 shows the results for the best settings, as well as the best results found for word and character n-grams apart.
For the preprocessing of the text data we lowercased all tweets and subsequently tokenized the tweets with the NLTK Tweet Tokenizer. 2 We also replaced every username with @username and every URL to URL. Table 7 shows that our preprocessing methods do indeed improve performance for each language. Especially generalizing over URLs was beneficial.
In this section we will report the results of our systems on the training corpus (10-fold CV) and final test set.
The results in Table 8 shows that only using n-grams as features results in having a good baseline result that is in line with the findings in [3]. The model that only uses embeddings performs worse than the model that utilizes n-grams. Moreover, the model that combines embeddings and n-grams also performs worse.
The image-only model performed poorly with accuracies around 60%. However, using these features in combination with the n-gram features gave us slight performance improvements for Arabic and Spanish. Although we have found an increase of accuracy in Arabic, approximate randomization testing [11] 3 showed us that this improvement is not significant.
We handed in three final systems; one for classification based on text-only, one for images-only and one for the combination of text and images. For all of the three final systems, we used a Logistic Regression classifier. 9 shows our official results on the PAN 2018 test set. We see that our performance for English is a bit lower than we expected, but for Spanish and Arabic we obtain a better performance than on the training set. On average over all languages we scored an accuracy of 0.799 on the text-only submission. The images do not influence the score much, but since it now decreases the score, it is questionable whether our (simplistic) approach of processing the images is helpful for this task.
For the combination system with both text and image features we scored an average accuracy score of 0.7963 and became 6 th in the global ranking. In general we can say that our Logistic Regression model does quite well, making it a robust, straightforward and reliable method of doing gender classification.
In this paper, we presented our approach for the PAN 2018 author profiling shared task for predicting an author's gender using text-based and image-based features. We submitted a Logistic Regression classifier using word and character n-grams as text-based features and several automatically extracted image features. We found that only using text-based n-gram features gave us the best results for English and Spanish, whereas the combination of text-based and image-based features gave us the best results for Arabic.
As additional text-based features we tested word embeddings, but results on the training data shows that these rather hurt our system's performance.
For this shared task we experimented with using images to predict an author's gender. We used an image feature extraction tool to classify detected faces in images on gender and emotion. We also tried to construct a feature that could indicate selfies, as females tend to post more selfies than males. Our results showed that only using such image-based features are performing poorly with accuracy scores around 60% with a Logistic Regression classifier. Adding these features to a text-based n-gram model does not influence the score much. The images decreased the scores slightly on English and Spanish, but gave us a small improvement on Arabic on the PAN 2018 test dataset. Our submitted system only used image-based features extracted from detected faces, but data showed that not all images includes a face. Therefore, for future research we suggest a system that enlarges the set of image-based features.