We present our system for the CLIN29 shared task on crossgenre gender detection for Dutch. We experimented with a multitude of neural models (CNN, RNN, LSTM, etc.), more \traditional" models (SVM, RF, LogReg, etc.), di erent feature sets as well as data pre-processing. The nal results suggested that using tokenized, nonlowercased data works best for most of the neural models, while a combination of word clusters, character trigrams and word lists showed to be most bene cial for the majority of the more \traditional" (that is, non-neural) models, beating features used in previous tasks such as ngrams, character n-grams, part-of-speech tags and combinations thereof. In contradiction with the results described in previous comparable shared tasks, our neural models performed better than our best traditional approaches with our best feature set-up. Our nal model consisted of a weighted ensemble model combining the top 25 models. Our nal model won both the in-domain gender prediction task and the cross-genre challenge, achieving an average accuracy of 64.93% on the in-domain gender prediction task, and 56.26% on cross-genre gender prediction.
In recent years, author pro ling (AP) has gained a lot of interest. AP can be described as the task of predicting or identifying demographics (such as gender and age) of an author based on their writing. It has applications of growing importance in di erent elds such as security, marketing, etc. Of all the demographics, there has been a particular interest in gender pro ling which has Copyright c 2019 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0) been part of the shared tasks organized by PAN for the past six consecutive years (2013{2018). Tasks di ered from in-domain gender prediction achieving relatively high scores to multi-modal gender prediction and cross-genre gender prediction, initially focusing on English but gradually including other languages such as Dutch, Spanish, and others.
In this paper, we describe our best system submitted for the CLIN29 2019 shared task on gender prediction for Dutch with a particular focus on out-ofdomain data, along with all other systems we experimented with. The plethora of trained models and empirical evaluations we conducted for this task is one of our main contributions; the other is the ensemble system which wins both the in-domain and out-of-domain subtasks.
The rest of the paper is structured as follows: Section 2 describes the related work, Section 3 the experimental set-up. Results are discussed in Section 4. Finally, the conclusions and ideas for future work are presented in Section 5. 2
Over the last years, there has been a signi cant body of research dedicated to
AP, often including gender prediction. The main focus, however, has been on
predicting gender using in-domain data for training and testing. The various
approaches presented in the PAN shared tasks used preprocessing, lowercasing,
stop-word ltering combined with speci c linguistic features including character,
word and POS n-grams, punctuation features, topic modeling and features speci c
to the given domain (e.g. hashtags or links for Twitter data). The yearly PAN
evaluation campaigns (pan.webis.de) have led to the development of
state-of-theart (SOTA) in-domain gender prediction models on Twitter data for English
achieving accuracies in between 80% 85%
The PAN 2016 di ered from previous gender prediction tasks as it was the
rst shared task focusing on cross-genre gender prediction. Twitter data was
provided for training while the test data was another `unknown' type of social
media text. It should be noted, however, that although the test data di ered from
the training data, all the data still belonged to the broader 'social media' domain.
The best scores recorded for gender prediction were 62%, 73% and 76% for Dutch,
Spanish and English respectively
An important di erence between the two previous tasks on real cross-genre gender prediction and this year's CLIN shared tasks is that, unlike Russian and Italian, gender agreement with the rst person is very rare in Dutch1. In Russian and Italian, verbs, adjectives and nouns (can) re ect the gender of the speaker, which facilitates gender prediction. 3
In this section, we describe in more detail: the datasets provided (3.1) and the di erent models and features used (3.2). 3.1
The datasets initially provided belonged to three di erent domains: Twitter
(TW), YouTube (YT) and News (N). For the 3 in-domain scenarios we used 90%
of the data for training and 10% for validation. For the 3 out-of-domain scenarios
we used all the out-of-domain data available. For example, for the out-of-domain
YouTube prediction, we used all the given News and Twitter data and validated
on the YouTube data provided. For some of our models (i.e., our second set
of submissions), we also used external CSI
N I AN (90-10) OMTW (90-10) -NDYT (90-10) I N I AMN+TW jYT ON+YT jTW -TDTW+YT jN U O # Train # Valid 1 648 184 18 000 2 000 13 269 1 475 # Train # Valid 21 832 14 744 16 576 20 000 34 744 1 832 # Train 3 113 68 907 1 Exceptions would be certain sentences where the noun agrees in gender with the subject, e.g., `Ik ben een acteur' (masc.) vs `Ik ben een actrice' (fem) [English: `I am an actor/actress']
Our winning approach consisted of an ensemble model of our strongest models. First, we will rst describe all the neural and traditional models we experimented with as well as the e ects of di erent feature sets. After, the ensembling of the best models will be described in more detail.
Neural Models We experimented with the following neural networks: SpaCy
TextCategorizer models (SpaCy)
All neural network models we trained using the following parameters: batch
size = 32, hidden state size = 256 and embedding length = 300; The learning
optimizer was Adam
Traditional Models From previous shared tasks it resulted that more
traditional statistical models, in general, still outperform neural models. The
traditional approaches we experimented with include: Statistical Language Models
(LMs), Support Vector Machines (SVMs), K-Nearest Neighbour Classi er (KNN),
Logistic Regression Classi er (LogReg), Random Forest Classi er (RF) and
Bernouilli Naive Bayes (NB) using the scikit-learn toolkit
We ought to note some di erences between our LM-based models and the
other approaches. In particular, for the LM-based models, we: (i) split the data
into two subsets { a female and a male subset { which we (ii) use separately to
train two KenLM
Features As previous shared tasks showed the usefulness of features such as word n-grams, character n-grams, Part-Of-Speech (POS) tags, and article counts, we compared how the 'traditional' classi ers performed with such features. We experimented with 4 additional features: 2 As the fasttext embeddings are extracted from Wikipedia, yet a di erent domain, the usage of these embeddings does not interfere with the objective of the cross-genre prediction task. 3 https://github.com/prakashpandey9/Text-Classification-Pytorch 4 See: https://spacy.io/usage/training for details. clusters: Inspired by the cross-genre task and the necessity for generalization we added a cluster feature. To do so, we clustered words together based on their fasttext embeddings and subsequently replaced unique words by their cluster number. Clusters containing only singletons or more than 500 words were removed.
The three following features were inspired by the ndings of Keune (2012) whose thesis investigated the di erence between male and female speech.
words used more by men: Keune (2012) concluded from her research on the Corpus Gesproken Nederlands (CGN)5 that there were certain words linked to a speci c gender. Similar observations were made in Vanmassenhove et al. (2018), showing that certain words are more frequent in male/female speech. Words such as: `feitelijk', `voornamelijk', `degelijk', `oorspronkelijk', `tamelijk', `onmiddellijk', `je', `d'r', `ja', `nee' and `neen' are used more by men than by women. Therefore, we decided to count the number of occurrences of these words.
words used more by females: Similarly, some other concrete words are used signi cantly more by women: `ik', `hij', `dadelijk', `vriendelijk', `lelijk', `vrolijk', `eindelijk' and `verschrikkelijk'.
diminutives count: We also counted the diminutives appearing in every data
example as diminutives are used more by women than by men
We started by using the classi er with the strongest baseline on unigrams for the 3 out-of-domain validation datasets. This appeared to be the LogReg classi er with an average accuracy of 53:11%. We rst evaluated all features separately, then combinations of two and three features. Using 4 features or more lead to decline in the accuracy. Interestingly, as can be observed in Table 2, by using a single feature, the cluster feature outperformed all other features on all three cross-genre datasets. The highest average accuracy for the traditional models (53:24%) was obtained by combining clusters, words used more by men and char 3grams.
Features
Clusters Char 3-grams Unigrams liadTYowuitTtuebre 5533..0505%% 5522..9834%% 5503..9298%%
VNews 53.11% 52.78% 51.20% Table 2. The performance of the best traditional model (LogReg) on the di erent cross-genre datasets based on the three best single features: clusters, char 3-grams and unigrams
The Winning Approach From the experiments, it resulted that, both for in-genre and cross-genre experiments, the neural models outperformed (with many con gurations) the traditional approaches with di erent feature sets as well as the statistical language models. Therefore, we decided to continue working 5 `Corpus of spoken Dutch' with the neural models of which the best ones were SpaCy, LSTMa, and SA. We trained 5 instances of these models, each with a di erent random seed. In addition, for the LSTMa and SA, 5 instances were trained with pretrained frozen word vectors and 5 without. This resulted in a total of 25 models.
Ensembling We opted for a weighted ensemble, where each model contributes its deviation from randomness, i.e. if the validation score of a model is 0.55, it will have a positive weight of 0.05 (0:55 0:50), however, if the validation score is 0.40, the weight will be -0.10 (0:40 0:50). Afterwards, we computed the output for every model (1 or 1) and added the prediction multiplied by the weights. 4
In the following paragraphs, we present the results of our nal weighted ensemble model.
The shared task allowed to submit 2 sets of results for every scenario, allowing thus for maximum of 12 di erent models. Our rst set of results (E-1) are presented in Table 3. These results are all generated by the ensemble models described above. No external data was used here.
Twitter 64.75% 57.89% Youtube 62.47% 56.98% News 66.60% 53.50%
Our second set of submissions (E-2) were generated by models trained with
external data from CSI
Twitter 65.01% 55.89% Youtube 63.49% 57.10% News 66.30% 55.80%
AVG 64.94% 56.26% Table 4. The accuracies of our best performing submission on the di erent test sets for in-genre and cross-genre gender prediction. (E-2)
With respect to the results obtained in the cross-genre setting, we observe low accuracies (55:80{57:10). However, during our experiments we observed that the models consistently got an accuracy of more than 50% on all validation data, i.e., better than random predictions, indicating that there are speaker-speci c language or style features that aid to identify the gender of the speaker/writer. However, a further exploration of the outputs of the models is needed in order to draw more concrete conclusions. 5
More and more research has focused on author pro ling and gender prediction more particularly. The performance of models largely depends on the languages and domains involved. In this work, we explored di erent methods and models that achieve state-of-the-art results on various Natural Language Processing tasks and applied them on the task of gender prediction for Dutch. The plethora of empirical data we collected con rms the complexity of this task and we can not help but wonder whether we are to expect further improvements on this type of tasks for Dutch.
This work has been supported by Dublin City University Faculty of Engineering & Computing under the Daniel O'Hare Research Scholarship scheme and by the ADAPT Centre for Digital Content Technology, funded under the SFI Research Centres Programme (Grant 13/RC/2106) and Theo Ho enberg, founder & CEO of Reverso.
We would also like to thank the organizers of the shared task.