In this paper we describe our participation in the Evalita 2018 GxG crossgenre/domain gender prediction shared task for Italian. Building on previous results obtained on in-genre gender prediction, we try to assess the robustness of a linear model using n-grams in a crossgenre setting. We show that performance drops significantly when the training and testing genres differ. Furthermore, we experiment with abstract features in trying to capture genre-independent features. We achieve an average F1-score of 0.55 on the official in-genre test set - being thus ranked first out of five submissions - and 0.51 on the cross-genre test set.
In questo articolo presentiamo il nostro contributo per lo shared task GxG di Evalita 2018 per l’analisi predittiva del genere di un autore su corpora di dominˆı diversi. Usiamo un modello basato su una macchina a vettori supporto con kernel lineare che usa gli n-grammi come feature. Il modello, che ha ottenuto in passato risultati eccellenti nella predizione di genere quando allenato e valutato all’interno di un unico dominio (Twitter), crolla significativamente nella performance in questo lavoro, anche quando usato in combinazione con una serie di feature astratte. Sul test set ufficiale il nostro modello raggiunge una F1-score pari a 0.55 (permettendoci di piazzarci in cima alla classifica) nel contesto in-genre e 0.51 nel contesto cross-genre.
Gender prediction is the task of profiling authors
to infer their gender based on their writing. This
task has been carried out so far with success
within one-genre/single-domain data sets,
reaching state-of-the-art accuracy of 85% on English
tweets
The theoretical assumption behind gender
prediction from text is language variation: the same
meaning can be expressed in different forms and
this variation can be explained in terms of social
variables, such as social status, personality, age
and, indeed, gender
Proof of significant variation in language use
between men and women has been found at the
morpho-syntactic level
In this paper we use a model built, trained,
and tested for gender prediction on a single
domain (i.e. Twitter). Instead of experimenting with
new techniques, our aim is to sound the
existing model’s resilience in the different context of
a cross-genre train and test setting (RQ1). Since
we expect our model to fail on this task, we set
out to design an experiment using a set of
abstract features that has recently been applied for
cross-lingual gender prediction
We organise this work as follows: in Section 2 we present an overview of the data set released by the organisers; in Section 3 we describe the experimental set up, the model and the features used; in Section 4 we give an overview of the results and finally we conclude our work in Section 5.
The contributions of this work for the GxG task for EVALITA 2018 are the following: we test if a gender prediction model achieving state-of-the-art performances when trained and tested on the same domain can also achieve good results when tested in a cross-domain setting we experiment with abstract features in order to factor out domain-dependent effects we release all the code for further reproducibility at https://github.com/ anbasile/gxg-partecipation
Task description The GxG task is a document classification task. Given a document belonging to a given genre, we have to predict the gender of the author. Thus the task is a binary classification task. The task is composed by two sub-tasks: ingenre prediction and cross-genre prediction. In the first case, the training set and the test set belong to the same genre. In the cross-genre sub-task, on the other hand, models will be trained on four genres and then tested on the single genre which they have not been exposed to during training. 2
We use only the data released by the task organisers: that is, texts from five different genres. The genres are as follows.
From the data description given by the organisers we know that one author can possibly have authored multiple documents. We provide an overview of the data set in Table 1.
Diaries
Journalism
Twitter
Youtube
In this section we describe the feature extraction
process and the model that we built. We develop
one single model and train it in ten different
configurations, as required by the task assignment.
We train and test on five different domains,
indomain and across domains.
We decide not to pre-process the data in any way,
since we have no linguistic (nor non-linguistic)
reasons for doing so. As a tokenization strategy
for the lexicalized models we simply split on all
white space tokens. For building the abstract
feature representation we use spaCy’s Italian
tokenizer
As features we use n-grams extracted at the
word level as well as at the character level. We
use 3-10 n-grams and binary TF-IDF. We feed
these features to a Support Vector Machine (SVM)
model with a linear kernel; we use the
implementation included in scikit-learn
Furthermore, we experiment with feature
abstraction: we follow the bleaching approach
recently proposed by
Since the data set labels are evenly distributed across the two classes, we use accuracy to evaluate our model. First, we report results obtained via a 10-fold cross-validation on the training set; then, we report results from the official test set, whose labels have been released. 4.1
We report the development results obtained by using different text representations. Table 2 presents an overview of these results. Overall, we see that all the different feature representation formats lead to comparable results; the combination of words and characters seems to be the best combination. This is the same combination that we use for the bleached representation. 4.2
We present official test results in Table 4. We submitted only one run. For one genre (Diaries) we obtained exactly the same score in both the in- and cross-genre settings: this outcome is extremely unlikely, however we ran the models several times and inspected the code for bugs and yet the results remained identical. In the cross-genre setting, we did not tune the hyper-parameters of our model considering the target genre.
The overview of the results is puzzling. First,
compared to related in-domain work
GENRE
IN
CROSS
YouTube Twitter Children Journalism Average acc. 0.635 0.547 0.545 0.615 0.480 We presented our participation to the GxG crossgenre gender prediction task and we obtained good results using a simple system. On top of that, we experimented with abstract features and got suboptimal results.
Based on our experiment with a
genderprediction model which obtained state-of-the-art
in-domain performance in the past, we conclude
that genre plays a crucial role in gender
prediction: not only genre, but the notion of variety
space, as instructed by
Recently, Variational Auto Encoders (VAE)