This overview presents the results of the cross-genre gender prediction task (GxG) organized at CLIN29. Teams were tasked with training a system to predict the gender of authors of tweets, YouTube comments and news articles. In the cross-genre setting, systems were trained on two genres, and tested on the other to assess domain adaptivity of the solutions. Eight teams participated in the shared task. Performance was generally better in the in-genre setting. In the cross-genre settings, performance on news articles declined the most compared to other target genres.
In this paper we give an overview of the GxG shared task at CLIN29. The next section explains the motivation and setup for the task. Section 3 gives an overview of the genres and the data that was used. We then brie y outline the participating systems and their solutions to the posed task in Section 4. Section 5 will list the nal results and highlights a few notable ndings.
Despite considerable progress and increasing research activity in author pro ling
from text, as witnessed for example by the PAN competition1, the problem is
far from solved. One obstacle is the absence of solutions to cross-genre pro ling:
when models trained on one genre are applied to another, accuracy usually
decreases dramatically. Typically, gender pro ling for languages like English is
in the 80-85% range, but drops to the 60% range in a cross-genre setting [
The task is cast as a binary classi cation problem. Given a text, a system has to predict whether its author was male or a female. The models are trained in two settings: within the same genre and in a cross-genre setting. Teams were allowed to submit up to two runs per setting, for a potential total of 12 runs per team, six in-genre and six cross-genre. 3
Data was collected for three di erent genres. Two genres (Twitter and YouTube) consisted of short user posts as documents. The third genre consisted of online news paper articles from ten Flemish and three Dutch news outlets. The labels were balanced in each of the genres and care was taken to provide equivalent numbers of tokens in the training portion despite di erent document lengths. See Table 1 for an overview of the data.
To determine an author's gender, Twitter and YouTube user pro les were cross-checked with lists of known Dutch male and female names. For the news collection, only articles that were written by a single author were considered. We then looked up their full name to determine whether the author was male or female. The nal collection of news articles was written by 767 di erent authors, 437 male and 330 female. A total of eight teams participated in the shared task. Table 2 summarizes the participants, with their a liations, the number of submitted runs, and the letter code used in the tables reporting the results. National Research University Higher School of Economics, Moscow, Russia
ADAPT, School of Computing, Dublin City University, Dublin, Ireland & Computer Science Department, Bar-Ilan University, Ramat-Gan, Israel Department of Information Science University of Groningen, The Netherlands Department of Swedish / Sprakbanken, University of
Gothenburg, Sweden H Fraunhofer IAIS Sankt Augustin, Germany (Fraunhofer
Center for Machine Learning) #Runs 6 6 12 6 12 12 12 12 System A uses a neural approach, more speci cally a BiLSTM, trained on both word and part-of-speech n-gram features.
System B (Rob's Angels) applied a basic SVM approach with many di erent features and preprocessing steps. They also experimented with trying di erent training genres in the cross-genre setting.
System C made use of lexical features like lemmata, syntactic features like dependency relations and more abstract character-level features based on a text bleaching approach. These features were tested separately and combined in a logistic regression classi er; lexical features proved to be most e ective overall. System D did not submit a system description paper and as such we cannot report their system setup.
System E experimented with word clusters based on word embeddings. Additional features used were word unigrams and character trigrams. The eventual (winning) setup used an ensemble of di erent neural models whose output was weighted by their validation score System F, (wUGs) tried a basic character n-gram SVM and a combination of SVM and logistic regression using co-training and pre-processing by normalization. System G used a basic logistic regression model using token and character unigram features, and word lengths as features. The paper focuses on two di erent ways to combine the di erent training genres: using an LSE-based formulation of the objective and pooling the data.
System H investigated a bidirectional LSTM on word sequences and topic modeling features, and a random forest classi er using topic modeling features and function word patterns. 5
Table 3 shows all results for the in-genre settings. The top runs performed best when training and testing on news. The di erence in performance between the genres may be caused by the number of tokens provided for each genre. More data was available for news in comparison to Twitter and YouTube.
As expected, the performance in the cross-genre settings (Table 4) was lower for all systems. The in uence of data size seems less apparent in this setting, as there is no clear di erence between the three genres. However, the scores on news data have clearly su ered the most in the cross-genre setting. This may be because the Twitter and YouTube data, both being social media texts, share more commonalities. In the setting where a system is trained on the social media genres and tested on journalism, it may pick up artifacts that a ect the decision process.
The outcomes of the submitted runs can be directly compared to the
crossgenre gender prediction shared task at Evalita 2018 [
In the cross-genre setting, results at Evalita were comparable. The best performing system at Evalita did better when training on other genres and testing on Twitter (.609 accuracy) but worse on YouTube (.510) and journalism (.495).
Because of the uctuating scores of teams in both tasks, we cannot conclude that cross-genre pro ling was more successful in either. However, the best performing team in GxG CLIN29 had consistent performance on all genres, achieving the best score for YouTube and news and the second best score for Twitter.
0.663 0.689 0.666 0.639 0.637 0.620 0.619 0.594 0.599 0.583 0.503 0.494 0.502 0.558 0.535 0.552 0.528 0.534 0.554 0.548 0.553 0.526 0.508 0.510 0.504 0.501 0.532 0.6493 0.6471 0.6461 0.6333 0.6312 0.6248 0.6179 0.6171 0.6074 0.5973 0.5514 0.5455 0.5041 0.6056 0.5626 0.5612 0.5485 0.5474 0.5440 0.5409 0.5377 0.5373 0.5342 0.5270 0.5260 0.5104 0.4975 0.5365 0.6501 0.6367 0.6475 0.6316 0.6235 0.6311 0.6115 0.6482 0.6099 0.6241 0.5945 0.5915 0.4848 0.6142 0.6349 0.6156 0.6247 0.6294 0.6331 0.6233 0.6231 0.6091 0.6133 0.5849 0.5566 0.5511 0.5254 0.6019 0.5589 0.5789 0.5427 0.5549 0.5567 0.5467 0.5425 0.5376 0.5406 0.5494 0.5428 0.5177 0.4946 0.5434 0.5710 0.5698 0.5507 0.5594 0.5413 0.5220 0.5227 0.5212 0.5360 0.5236 0.5252 0.5094 0.4969 0.5345 E-2 F-1 E-1 G-1 C-1 G-2 C-2 B-1 A-1 F-2 H-1 H-2 D-1 AVG E-2 E-1 A-1 B-1 C-1 C-2 H-1 F-2 F-1 G-2 G-1 H-2 D-1 AVG