Introduction

Author profiling is the analysis of shared content in order to predict different attributes of authors such as gender, age, personality, native language, or political orientation. Supported by the huge amount of information that is available on social media platforms, author profiling has gained a lot of interest. Being able to infer an author's gender, age, native language, dialects, or personality opens a world of possibilities-among others in marketing, where companies may analyze online reviews to improve targeted advertising, or in forensics, where the profile of authors could be used as valuable additional evidence in criminal investigations, and in security, where knowing the demographics of social media users (age and gender), as well as cultural and social context such as native language and dialects, may help to identify potential terrorists [51].

In the following we provide a historical outline of previous editions of this task. In the Author Profiling task at PAN 2013 1 [45], the identification of age and gender relied on a large corpus collected from social media, both for English and Spanish. In PAN 2014 2 [46], we continued focusing on age and gender aspects but, in addition, compiled a corpus of four different genres, namely social media, blogs, Twitter, and hotel reviews. Except for the hotel review subcorpus, which was available for English only, all documents were provided in both English and Spanish. Note that most of the existing research in computational linguistics [6] and social psychology [40] focuses on the English language, and the question is whether the observed relations pertain to other languages and genres as well. In this vein, in PAN 2015 3 [47], we included two new languages, Italian and Dutch, besides a new subtask on personality recognition in Twitter. In PAN 2016 4 [50], we investigated the effect of cross-genre information: the models are trained on a certain genre (here: Twitter) and evaluated on another genre different than Twitter. In PAN 2017 5 [19], we considered the language variety identification together with the gender dimension. We evaluated this new subtask in four languages: Arabic, English, Portuguese and Spanish.

Social media data cover a wide range of modalities such as text, images, audio, and video, all of which containing useful information to be exploited for extracting valuable insights from users. Consequently, the objective of this year's evaluation 6 is to address gender identification from a multimodal perspective: not only texts but also images are given. For this purpose a corpus with Twitter data has been provided, covering the languages: Arabic, English, and Spanish.

The remainder of this paper is organized as follows. Section 2 covers the state of the art, Section 3 describes the corpus and the evaluation measures, and Section 4 presents the approaches submitted by the participants. Sections 5 and 6 discuss results and draw conclusions respectively.

Related Work

The relationship between personal traits and the use of language has been widely studied by the psycholinguistics Pennebaker [41]. He analysed how the use of the language varies depending on personal traits. For example, in regards to the authors' gender, he found out that in English women use more negations or first persons, because they are more self-concientious, whereas men use more prepositions in order to describe their environment. These finding are the basis of LIWC (Linguistic Inquiery and Word Count) [40] that is one of the most used tools in author profiling.

Initial investigations in author profiling [6,26,13,28,54] focused mainly on formal texts and blogs. Their reported accuracies ranged from 75% to 80%. Nevertheless, nowadays researchers focused mainly on social media, where the language is more spontaneous and less formal. It should be highlighted the contribution of different researchers that used the PAN datasets. For example, the authors in [35] showed how to deal with a large dataset such as the PAN-AP-2013 with 3 million features with a MapReduce configuration. With the same dataset, the authors in [67] showed the contribution of information retrieval-based features. Following Pennebaker findings about the relationship between emotions and gender, the authors in [44] proposed the EmoGraph graph-based approach to capture how users convey verbal emotions in the morphosyntactic structure of the discourse and showed competitive results with the best performing systems at PAN-2013 and demonstrating the robustness of the approach against genres and languages at PAN-2014 [43]. Recently, Bayot and Gonçalves [10] used the PAN-AP-2016 dataset to show that word embeddings worked better in case of gender identification than TF-IDF. Finally, it is worth mentioning the second order representation based on relationships between documents and profiles used by the best performing team in three editions of PAN [30,31,4], as well as the performance of the combination of n-grams as shown by the authors [9] of the best performing team at PAN 2017.

The investigation in Arabic is more scarce and most of the research focused on other genres than social media. For example, Estival et al. [18] focused on Arabic emails. The authors reported accuracies of 72.10%. Similarly, Alsmearat et al. [2] focused on Arabic newsletters. They initially reported an accuracy of 86.4% that was increased to 94% in an extension of their work [1]. With respect to social media, AlSukhni & Alequr [3] focused on Arabic tweets and they reported accuracies of 99.50%. They improved a bag-of-words model with the use of the Twitter authors' names.

The use of visual features for author profiling has been less studied. A common approach for gender identification is the use of frontal facial images [37,60,17]. The authors in [37] trained SVM with 1,755 low resolution thumbnail faces (21x12 pixels) from the FERET face database7 obtaining an error of 3.4%. The authors in [60] used Principal Component Analysis to represent each image in a smaller dimensional space, reducing the error from 17.7% to 11.3% with a neural network. The authors in [17] experimented with 120 combinations of automatic face detection, face alignment and gender classification. They found out that the automatic face alignment did not increase the gender classification rates, whereas the manual alignment did. The authors evaluated several machine learning algorithms, obtaining the best results with SVM. They also saw that the classification did not depend on the size of the images. Recently, user annotated data have been used more and more. For example, Twitter has been used as repository to learn and evaluate gender identification systems. In this sense, the authors in [34] used automatic image annotations and the authors in [56] proposed a Multi-task Bilinear Model to combine the visual concept detector with the feature extractor to predict gender in Twitter. Similarly, the authors in [8] used 56 image aesthetic features to gender identification in 24,000 images provided by 120 FlickR users, obtaining 82.50% of accuracy.

Evaluation Framework

The purpose of this section is to introduce the technical background. We outline the construction of the corpus, introduce the performance measures and baselines, and describe the idea of so-called software submissions.

Corpus

The focus of this year's task is on gender identification in Twitter from a multimodal perspective: besides textual information, the participants are provided also with images. The task is framed as a multilingual task, covering the languages Arabic, English, and Spanish. The PAN-AP-2018 corpus is based on the PAN-AP-2017 corpus [49], extended by images that have been shared in the respective Twitter timelines. More specifically, PAN-AP-2018 contains those authors from the PAN-AP-2017 corpus who still have a Twitter account and who have shared at least 10 images. Table 1 overviews the key figures of the corpus. Moreover, the corpus is balanced with regard to gender and it contains 100 tweets per author.

Performance Measures

The participants were asked to submit per author three predictions according to the following modalities: a) text-based, b) image-based, and c) a combination of both. It was allowed to approach the task in a favoured language and a favoured modality; however, we encouraged them to participate in all languages and all modalities. 8For each language and for each modality the accuracy was computed. Note that the accuracy of the combined approach has been chosen as overall accuracy for the given language; if only the textual approach was submitted, its accuracy has been used. The final ranking has been calculated as the average accuracy per language as defined by the following equation: ranking = acc ar + acc en + acc es 3

(1)

Baselines

In order to assess the complexity of the subtasks per language and to compare the performances of the participants approaches, we propose the following baselines:

-BASELINE-stat. A statistical baseline that emulates random choice. As there are two classes and the number of instances is balanced, the random choice baseline is 50% accuracy. This baseline applies for both modalities, images and texts.

-BASELINE-bow. To approach the textual modality, we have represented the documents under a bag-of-words model with the 5,000 most common words in the training set, weighted by absolute frequency. The texts are preprocessed as follows: lowercase words, removal of punctuation signs and numbers, and removal of stop words for the corresponding language.

-BASELINE-rgb. To approach the image modality, we represent the photos as follows. For each author, we obtain the RGB color for each pixel in his/her photos. We represent the author with the following descriptive statistics of the RGB values: minimum, maximum, mean, median, and standard deviation.

Software Submissions

We asked for software submissions (as opposed to run submissions). Within software submissions, participants submit executables of their author profiling softwares instead of just the output (also called "run") of their softwares on a given test set. Our rationale to do so is to increase the sustainability of our shared task and to allow for the re-evaluation of approaches to Author Profiling later on, and, in particular, on future evaluation corpora. To facilitate software submissions, we develop the TIRA experimentation platform [21,22], which renders the handling of software submissions at scale as simple as handling run submissions. Using TIRA, participants deploy their software on virtual machines at our site, which allows us to keep them in a running state [23].

Overview of the Submitted Approaches

This year, 23 teams participated in the Author Profiling shared task and 22 of them submitted the notebook paper. 9 We analyse their approaches from three perspectives: preprocessing, features to represent the authors' texts, and classification approaches.

Preprocessing

Various participants cleaned the textual contents to obtain plain text. Most of them removed or normalised Twitter-specific elements such as URLs, user mentions, or hashtags [15,61,59,42,53,24,66,36,65,38,29]. Some participants also lowercased the words [66,65,38,11,29,59,53,24]. The authors in [15,59,24,65] removed punctuation signs; character flooding has been removed by the authors in [15,42]. Stopwords have been removed by the authors in [15,42,24,65], and contractions and abbreviations have been expanded by the authors in [59,42]. The authors in [15] applied specific preprocessing to Arabic texts, such as normalisation and diacritics removal.

Only three participants preprocessed images. The authors in [61] applied direct resizing and resizing with cropping, as well as normalisation by subtracting the average RGB value per language. The authors in [36] rescaled all images to 64x64 and used only those containing human faces, while the authors in [57] rescaled all images to 224 pixel width, maintaining the aspect ratio.

Features

In previous editions of the author profiling task at PAN as well as in the referred literature, features used for representing text documents have been distinguished as either content-based or style-based. However, this year several participants have employed deep learning techniques. It is interesting to differentiate among traditional features and these new methods in order to compare their performance in the author profiling task. While the authors in [36,65,11,33,61] represented documents with word embeddings, the authors in [53] used character embeddings. Moreover, the authors in [59,52,33] also used traditional features such as character, word, and/or POS n-grams. The authors in [39] combined word embeddings for English as well as stylistic features; however, for Spanish and Arabic they used LSA instead of word embeddings.

Traditional features such as character and word n-grams have been widely used [66,62,38,29,16,24,59,15]. Style features have been also used by some participants [39,27,24]. For example, the authors in [39] used the counts of stopwords, punctuation marks, emoticons, and slang words (only for English). The authors in [27] combined POS tags n-grams with syntactic dependencies to model the use of amplifiers, verbal constructions, pronouns, subjects and objects, types of adverbials, as well as the use of interjections and profanity. The authors in [24] counted the average number of characters and the average number of words per tweet. The authors in [66] also used emojis, whereas the authors in [20] used only the skewness calculated from a variation of the Low Dimensionality Statistical Embedding (LDSE) [48]. The authors in [5] combined ensembles of word and character n-grams with bag-of-terms and second order features [30,31,32], which relates documents with authors' profiles.

With respect to the representation of images several approaches have been presented. For example, some participants tried to detect faces in images [59,15,65]. In this regard, the authors in [65] used face vectors from images that contained only faces. Besides faces the authors in [15] detected also objects and quantified local binary patterns and color histograms. Other authors used image resources, such as [39], who applied an image captioning system [64]. Similarly, the authors in [38] used a known image feature extraction tool [7] to obtain features about the number of faces in the images, as well as the expressed emotions or their gender. The authors in [5] used ImageNet [58] to obtain VGG1610 features, and the authors in [53] built a languageindependent model with TorchVision. 11 The authors in [61] also used a pre-trained Convolutional Neural Network (CNN) based on VGG16. Other participants approached the task with their own set of features, such as the authors in [24] who combined three sets of characteristics: Shift, RGB histogram, and VGG. The authors in [62] designed a variant of the Bag-of-Visual-Words (BoVW) by using the DAISY [63] feature descriptor and encoded the images by the set of visual words.

Classification Approaches

Regarding the deep learning approaches, the authors with the overall highest accuracy [61] used Recurrent Neural Networks (RNN) for texts and CNN for images. CNNs have also been used by the authors in [5,53,55,36], while RNNs have also been used by the authors in [11]. Interestingly, the authors in [53] used CNN only for texts and ResNet18 [25] for images. In the same vein, the authors in [65] approached the images with SVM but used Bi-LSTM for texts. The authors in [59] used CNN for images and an ensemble of Naive Bayes and RNN for texts. Finally, the authors in [42] approached the task with dense neural networks. Some participants still used traditional machine learning algorithms such as logistic regression [52,24,66,38], SVMs [33,5,15,39,62,65], multilayer perceptron [24], a basic feed-forward network [29], and distance-based methods [62,27]. It is worth to mention the approach in [20], who used a simple IF condition with respect to only one feature, allowing the system to process the whole dataset in seconds while achieving a decent performance.

Evaluation and Discussion of the Submitted Approaches

Although we encouraged to consider both modalities, some participants approached the problem with text features only. We present the results separately to account for this fact.

Gender Identification with Text Features

As can be seen in Table 2, the best results were obtained for English (82.21%) [16] and Spanish (82%) [16], although being only slightly better than for Arabic (81.70%) [62]. This similarity is also reflected by the mean accuracies, which are 74.85% for Arabic, 76.93% for English, and 75.46% for Spanish. Taking a closer look at the distributions (Figure 1) shows a different characteristic for English: the median is higher and approximately equal to the Q3 of the other languages, while the interquartile range is smaller. The similarity in the mean value is due to the two outliers (55.21% [27] and 66.580% [52]). This fact is highlighted in the density chart (Figure 2), where the curve for the English language is more skewed to the right and the kurtosis is higher since there are more results concentrated around 80%.

The best result for Arabic (81,70%) is from the authors in [62]; they performed several preprocessing steps and trained an SVM with word n-grams, character n-grams, and skip-grams of different lengths and different weighing schemes such as boolean, tf, and tf-idf. There is no statistical significance with respect to the second (81.20%) [57] and third (80.90%) [16] best results. The authors approached the task with character n-grams and combinations of different types of n-grams. The best result for English (82.21%) comes from the authors in [16]. There is no statistical significance with the second (81.21%) [62] and third (81.16%) [38] best results. The authors in [38] used Logistic Regression with word and character n-grams. Finally, for Spanish, the best result (82%) is from the authors in [16]. Again, there is no statistical significance regarding the second (80.36%) [65] and third (80.27%) [38] best systems. The authors in [65] used a bi-LSTM with pre-trained word embeddings.

With respect to the provided baselines, we can discard the statistical one since its results are much lower than those obtained by the participants. The BOW baseline is at rank 17 out of 22 in the overall ranking. 12 Furthermore, for Arabic the obtained result (74.80%) is very close to the mean (74.85%), while 9 participants are below. For English and Spanish, most participants were better than the baseline. For English, the obtained result (74.11%) is lower than the mean (76.93%) and even lower than the Q1 (76.34%), with 4 participants below (including the aforementioned outliers [27,52]). For Spanish, the obtained result (72.55%) is below the mean (75.46%) and the Q1 (73.70%), with 5 participants below (including one outlier).

Gender Identification with Images

As can be seen in Table 3, the best results were achieved for English (81.63%), with statistical significance over Spanish (77.32%) and Arabic (77.80%). All best results stem from the authors in [61], who used a pre-trained CNN on the basis of ImageNet. Despite this higher value for the best obtained result for English, the distributions of accuracies are very similar for the three languages, as can be seen in the Figures 3 and 4.

The mean values are of 62.37%, 63.41%, and 61.86% for Arabic, English, and Spanish respectively, with standard deviations below 10% and following a normal distribution. For Arabic, the second best result (72.80%) has been obtained by the authors in [57], who used VGG16 and ResNet50 from ImageNet. The third best result (70.10%) has been obtained by the authors in [15]. Besides color histograms they have detected faces, objects, and local binary patterns. Although there is no statistical significance between them at 95% of confidence, there is with respect to the best result (not at 99%). For English, the second (74.42%) and third (69.63%) best results are from the authors in [57] and [15] respectively. In both cases the difference is statistically significant. Similarly, for Spanish the second (71%) and third (68.05%) best results are from the authors in [57] and [15] respectively. Again, the difference is statistically significant.

As before, we can discard the statistical baseline. Similarly, most of the participants have achieved better results than the RGB baseline (52.60% on average); two participants achieved slightly lower results (50.23% and 50.22%) [24]). For all languages the baseline (54.10%, 51.79%, and 51.91%) is below the respective Q1s (55.57%, 56.89%, and 56.40%). Also note that this baseline is only slightly better than the statistical one, we shows that it is not suitable for the task.

Table 3. Accuracy per language in the gender identification task with images.

Ranking Team

Arabic English Spanish Average

Combined Approaches

We now analyse how images can help to tackle the gender identification task. Table 4 shows the basic statistics about the improvement (in %) for the different languages. On average, the improvement is very small (0.76% and 1.01% for Arabic and English), or even negative (-0.06%) for of Spanish. However, looking at Figure 5 it can be seen that some systems perform much better such as Takahashi et al., who achieved an improvement of 7.73% for English. The tables 5, 6, and 7 show the accuracies obtained with texts, with images, with their combination, and the percentage of improvement for Arabic, English, and Spanish respectively. Similarly, the Figures 6, 7, and 8 show for the same languages the density of the improvement distribution over text classification.

Table 5 shows the results for Arabic. As can be seen in Figure 6 the results do not follow a normal distribution; the improvement of most of the participants is between 0.53% and -0.26%, whereas three users obtain higher improvements: 1.82% [61], 2.93% [5], and 3.36% [39]. It is noteworthy that the systems that obtained the highest results tried to capture semantic features from images, and not only faces or colors. For example, Gopal-Patra et al. [39] used an image captioning system [39], Aragon & Lopez [5] ImageNet to obtain VGG16 features, and Takahashi et al. [61] a pre-trained CNN also on the basis of ImageNet. The distribution of improvements for English is even less normal, as can be seen in Figure 7. There are three groups of systems (see Table 6): i) systems with improvements between 0.72% and deteriorations of -4.65%, ii) one system with an improvement of 2.37% [39], and iii) one system with an improvement of 7.73% [61]. Similar to Arabic, the best results have been achieved by systems that exploit semantic features [61,39]. Furthermore, the less negative results have been achieved either with the use of Im-ageNet and VGG16 features [5] or with the combination of face recognition, object detection, local binary patterns, and color histograms [15]. For Spanish the systems' improvements follows a normal distribution, having two spikes in both extremes. In particular, there is i) one system whose deterioration is -4.47% [57], ii) a group of users with improvement/deterioration between -1.30% and 1.62%, and iii) one system with 3.75% of improvement [61]. In this regard, the best result has been obtained by Takahashi et al. with a pre-trained CNN from ImageNet, followed by the use of an image captioning system [39], the combination of faces, objects, and local binary patterns with color histograms [15], and the use of ImageNet to obtain VGG16 features [5].

Final Ranking and Best Results

This year 23 teams participated in the shared task; Table 8 shows the overall performance per language and user's ranking. The best results have been obtained for English (85.84%), followed by Spanish (82%), and Arabic (81.80%).

Table 8. Accuracy per language and global ranking as average per language.

Ranking Team Arabic English Spanish Average The overall best result (81.98%) is from the authors in [61] who approached the task with deep neural networks. For text processing, they used word embeddings from a stream of tweets with FastText skip-grams and trained a Recurrent Neural Network. For images, they used a pre-trained Convolutional Neural Network. They combined both approaches with a fusion component. The authors in [16] got the second best result on average (81.70%) by approaching the task only from the textual perspective. They used an SVM with different types of word and character n-grams. The third best overall result (80.68%) stems from the authors in [62]. They used an SVM with combinations of word and character n-grams for texts and a variant of the Bag of Visual Words for images, combining both predictions with a convex linear combination. According to t-Student, there is no statistical significance among the three approaches. This is also supported by the Bayesian Signed-Rank test [12] between Takahashi et al. and Daneshvar, as shown in Figure 9. However, for Takahashi et al. and Tellez et al., the probability of the first system to perform better (62.96%) is higher than the sum of being equal (20.64%) or worse (16.39%), as shown in Figure 10. The complete results of this test are presented in the Appendix B. With respect to the different languages, the best results have been obtained by the same authors. The best results for Arabic (81.80%) stem from the authors in [62], the best results for English (85.84%) from the authors in [61], and the best results for Spanish (82%) from the authors in [16]. Note that the only result that is significantly higher is the one obtained for English (85.84%). Table 9 shows the best results per language and modality. The results achieved with the textual approach are higher than the results obtained with images, although being very similar to those for English. It should be highlighted that the best results were obtained by combining texts and images, where in the case of English the improvement is higher.

Conclusion

In this paper we presented the results of the 6th International Author Profiling Shared Task at PAN 2018, hosted at CLEF 2018. The participants had to identify the gender from Twitter authors, considering both a multimodal and a multilingual perspective: the provided data contains both tweets and images and cover the three languages Arabic, English, and Spanish. The participants used different approaches to tackle the task, with deep learning approaches prevailing. However, the best results regarding the textual subtask have been obtained with combinations of different types of n-grams and traditional machine learning algorithms such as SVM and Logistic Regression. Only the second best result for Spanish was obtained with a bi-LSTM, which has been trained with word embeddings.

For the classification of images the approaches can be grouped in three types: i) approaches based on face recognition, ii) approaches based on pre-trained models and image processing tools such as ImageNet, and iii) approaches with "hand-crafted" features such as color histograms and bag-of-visual-words. Regarding the second type, the best results were obtained with semantic features extracted from the images. Approaches based on face recognition do not belong to the best, which may be rooted in the fact that many images do not show faces-and if, the contained faces do not depict the author.

According to the achieved results, text features discriminate better between genders than do images. However, the combined use of both modalities provides insights: On average, there is no improvement when images are used, which is due to the low performance of some inferior approaches. However, for more elaborated representations, which obtain semantics from the images with the use of tools such as ImageNet, the improvement is up to 7.73% for English (taking into account that the accuracy obtained only with text features is even high).

The best results in the shared tasks are over 80% on average, with the highest result for English (85.84%) [61], followed by Spanish (82%) [16], and Arabic (81.80%) [62]. Takahashi et al. [61] approached the task with deep learning techniques: word embeddings and RNN for texts and ImageNet-based CNN for images. Daneshvar [16] approached the task using the textual modality only. The author trained an SVM with combinations of word and character n-grams. Finally, Tellez et al. [62] used SVM with different kinds of n-grams, combined with a variant of the Bag of Visual Words (BoVW) using the DAISY feature descriptor. Altogether, traditional approaches still remain competitive, while some new approaches based on deep learning are acquiring strength.

Appendix A Pairwise Comparison of all Systems

For all subsequent tables, the significance levels are encoded as follows: A1. Significance of accuracy differences between system pairs. Textual modality in Arabic. Table A2. Significance of accuracy differences between system pairs. Image modality in Arabic. Table A3. Significance of accuracy differences between system pairs. Combined modality in Arabic. A4. Significance of accuracy differences between system pairs. Textual modality in English. Table A5. Significance of accuracy differences between system pairs. Image modality in English. Table A6. Significance of accuracy differences between system pairs. Combined modality in English. A7. Significance of accuracy differences between system pairs. Textual modality in Spanish. Table A8. Significance of accuracy differences between system pairs. Image modality in Spanish. Table A9. Significance of accuracy differences between system pairs. Combined modality in Spanish.