We present our participation in the PAN 2018 Author Profiling shared task, classifying authors on gender for English, Arabic and Spanish. We participated in all sub-tasks and propose a system for classification with text, images and the combination of those two. Our final submitted system is a Logistic Regression classifier that uses word and character n-grams as textual features and a set of automatically derived image-based features such as the presence, proportion and number of faces to detect selfies as well as the faces' emotions and gender. We experimented with word embeddings, which negatively affected our system's performance. Our cross-validated training results shows slight improvements in performance for Arabic and Spanish when image-based features are added to text-based features. Our highest scores on the PAN 2018 test dataset are accuracies of 81.2% for English using only text-based features, 78.7% for Arabic using both text- and image-based features and 80.3% for Spanish using only text-based features. Overall, we finished 6th in the global ranking with an average accuracy for our text and image combination system of 79.6%.
The field of author profiling is about inferring traits from an author such as gender, age and personality. With the rise of social media platforms, such as Twitter and Facebook, the field of author profiling has gained more interest. From multiple viewpoints, it is desirable to profile an author. Examples of such viewpoints could be from a security point of view in order to detect authors with criminal intentions and from a marketing point of view in order to narrow down target audiences for online advertisements.
In the past years, multiple shared tasks have been organized on the topic of author
profiling [18,16]. In this paper, we describe our approach for the Author Profiling shared
task at PAN 2018 [17]. This year’s Author Profiling task, is the 6th iteration of this
task and is slightly different from the previous years, since the gold standard data now
includes images. The task is to built a system to classify a Twitter author’s gender by
100 of its tweets and 10 posted images. Though the images are new for this shared task,
previous work already created systems that are capable of detecting gender, emotional
expressions and personalities from images [
In the last two years, the winning systems for 2016 [22] and 2017 [
Method
The PAN 2018 training corpus consists of tweets from three different languages,
English, Spanish and Arabic. For each author there are 100 tweets and 10 images labeled
by gender. The gender labels (male and female) are evenly distributed over the training
corpus. Table 1 shows an overview of the PAN 2018 training corpus released by the
organization.
The main set of features we used were n-grams. The winners of the previous year’s
Author Profiling shared task [
This year’s new addition to the gender classification task is classification by images. Our
approach to use the images for this task is to utilize existing image feature extraction
tools from related research. In our system, we have used the software used in [
In that study, a convolutional neural network (CNN) was implemented to find and classify faces in images by gender and emotion. The CNN model contains of 4 residual depth-wise separable convolutions, whereby each convolution is followed by a batch normalization operation and a ReLU activation function. The last layer of the model applies a global average pooling and soft-max activation function to produce the prediction. The system achieved an accuracy of 95% on the IMDB gender dataset and 66% on the FER-2013 emotion dataset. That system, including all code and pre-trained models are available under an open-source license.1
The software from [
The first two features are about the presence and number of detected faces in the images. In Table 2 are the number of faces detected for each language and gender. We see that there is little to no difference between the genders for which a face is detected.
Features three and four cover the relative area of images that are covered by a
detected face. These features are intended to capture selfies. Previous research [
large face area on an image with only one detected face could identify that an image is a selfie, and therefore these features could be helpful in the classification task.
Features five and six are about the gender of the detected faces. The values of these two features are floats ranging from 0 to 1, indicating the proportion of each gender. In Table 3 are the probabilities for a gender posting more faces of a specific gender in a image. We see that for all languages, males are posting more images of male faces than female faces, especially for Arabic there is a large difference in male and female faces. English and Spanish females are slightly posting more images with female faces than male faces, except for Arabic, in which female user post more male faces than female faces, but still to a lesser extent compared to their male counterparts.
Lastly, when one of the seven emotions from [
To get to our best system, we experimented with different classifiers. We used the
Python package Sklearn [
The results of all tested classification models can be found in Table 5. For every
model,we measured its performance by accuracy in a 10-fold cross-validation setup.
The models are all using the n-gram features used as in [
For the logistic Regression model we performed a parameter search, mainly to find
the optimal number of word and character n-grams. Our baseline n-gram model was
the n-gram model used in [
Ar
Es For the preprocessing of the text data we lowercased all tweets and subsequently tokenized the tweets with the NLTK Tweet Tokenizer.2 We also replaced every username with @username and every URL to URL.
Table 7 shows that our preprocessing methods do indeed improve performance for each language. Especially generalizing over URLs was beneficial. In this section we will report the results of our systems on the training corpus (10-fold CV) and final test set.
The results in Table 8 shows that only using n-grams as features results in having a
good baseline result that is in line with the findings in [
The image-only model performed poorly with accuracies around 60%. However,
using these features in combination with the n-gram features gave us slight performance
improvements for Arabic and Spanish. Although we have found an increase of accuracy
in Arabic, approximate randomization testing [
We handed in three final systems; one for classification based on text-only, one for images-only and one for the combination of text and images. For all of the three final systems, we used a Logistic Regression classifier.
3 We used the script by Vincent Van Asch https://www.clips.uantwerpen.be/ scripts/art In this paper, we presented our approach for the PAN 2018 author profiling shared task for predicting an author’s gender using text-based and image-based features. We submitted a Logistic Regression classifier using word and character n-grams as text-based features and several automatically extracted image features. We found that only using text-based n-gram features gave us the best results for English and Spanish, whereas the combination of text-based and image-based features gave us the best results for Arabic. As additional text-based features we tested word embeddings, but results on the training data shows that these rather hurt our system’s performance.
For this shared task we experimented with using images to predict an author’s
gender. We used an image feature extraction tool to classify detected faces in images on
gender and emotion. We also tried to construct a feature that could indicate selfies, as
females tend to post more selfies than males. Our results showed that only using such
image-based features are performing poorly with accuracy scores around 60% with a
Logistic Regression classifier. Adding these features to a text-based n-gram model does
not influence the score much. The images decreased the scores slightly on English and
Spanish, but gave us a small improvement on Arabic on the PAN 2018 test dataset.
Our submitted system only used image-based features extracted from detected faces,
but data showed that not all images includes a face. Therefore, for future research we
suggest a system that enlarges the set of image-based features.
16. Potthast, M., Pardo, F.M.R., Tschuggnall, M., Stamatatos, E., Rosso, P., Stein, B.: Overview
of pan’17 - author identification, author profiling, and author obfuscation. In: CLEF (2017)
17. Rangel, F., Rosso, P., Montes-y-Gómez, M., Potthast, M., Stein, B.: Overview of the 6th
Author Profiling Task at PAN 2018: Multimodal Gender Identification in Twitter. In:
Cappellato, L., Ferro, N., Nie, J.Y., Soulier, L. (eds.) Working Notes Papers of the CLEF
2018 Evaluation Labs. CEUR Workshop Proceedings, CLEF and CEUR-WS.org (Sep
2018)
18. Rangel, F., Rosso, P., Verhoeven, B., Daelemans, W., Potthast, M., Stein, B.: Overview of
the 4th author profiling task at pan 2016: cross-genre evaluations. In: Working Notes Papers
of the CLEF 2016 Evaluation Labs. CEUR Workshop Proceedings/Balog, Krisztian [edit.];
et al. pp. 750–784 (2016)
19. Schaetti, N.: Unine at clef 2017: Tf-idf and deep-learning for author profiling. Cappellato et
al.[