English. The Gender Cross-Genre (GxG) task is a shared task on author profiling (in terms of gender) on Italian texts, with a specific focus on cross-genre performance. This task has been proposed for the first time at EVALITA 2018, providing different datasets from different textual genres: Twitter, YouTube, Children writing, Journalism, Personal diaries. Results from a total of 50 different runs show that the task is difficult to learn in itself: while almost all runs beat a 50% baseline, no model reaches an accuracy above 70%. We also observe that cross-genre modelling yields a drop in performance, but not as substantial as one would expect. Italiano. GxG (Gender Cross-Genre) è la prima campagna di valutazione per l’identificazione del genere di un autore di testi scritti in lingua italiana e fa parte di quell’area di studio detta author profiling. In questa edizione di GxG particolare attenzione è stata posta nella valutazione dei sistemi in contesti di analisi cross-dominio. I domini testuali presi in esame sono stati: Twitter, YouTube, Children writing, Journalism, Personal diaries. I risultati ottenuti da un totale di 50 diverse run (prodotte da tre diversi gruppi di ricerca) mostrano che il task è complesso: mentre quasi tutti i sistemi testati superano la baseline del 50%, nessun modello raggiunge un’accuratezza superiore al 70%. Si osserva inoltre che i risultati raggiunti nel contesto di analisi cross-dominio mostrano un calo delle prestazioni non così sostanziale come ci si sarebbe potuto aspettare.
As publishing has become more and more accessible and basically cost-free, virtually anyone can get their words spread, especially online. Such ease of disseminating content doesn’t necessarily go together with author identifiability. In other words: it’s very simple for anyone to publicly write any text, but it isn’t equally simple to always tell who the author of a text is.
In the interest of companies who want to advertise, or legal institutions, finding out at least some characteristics of an author is of crucial importance. Author profiling is the task of automatically discovering latent user attributes from text. Gender, which we focus on in this paper, and which is traditionally characterised as a binary feature, is one of such attributes.
Thanks to a series of tasks introduced at the
PAN Labs in the last five years (pan.webis.
de, and the production of a variety of
genderannotated datasets focused on social media
However, in the context of the 2016 PAN
evaluation campaign, a cross-genre setting was
introduced for gender prediction on English, Spanish,
and Dutch, and best scores were recorded at an
average accuracy of less than 60%
This is evidence that we have not yet found
the actual dataset-independent features that do
indeed capture the way females and males might
write differently. To address this issue, we have
designed a task specifically focused on
crossgenre gender detection, and launched it within the
EVALITA 2018 Campaign
While Italian has featured in multi-lingual
gender prediction at PAN
GxG (Gender Cross-Genre) is a task on author profiling (in terms of gender) on Italian texts, with a specific focus on cross-genre performance.
Given a (collection of) text(s) from a specific genre, the gender of the author has to be predicted. The task is cast as a binary classification task, with gender represented as F (female) or M (male).
Evaluation settings were designed bearing in mind the question at the core of this task: are there indicative traits across genres that can be leveraged to model gender in a rather genreindependent way?
We hoped to provide some answers to this question by making participants train and test their models on datasets from different genres. For comparison, participants were also recommended to submit genre-specific models, i.e., tested on the very same genre they were trained on. In-genre modelling can (i) shed light on which genres might be easier to model, i.e. where gender traits are more prominent; and (ii) make it easier to quantify the loss when modelling gender across genres. Therefore, the gender prediction task must be done in two ways: • using a model which has been trained on the same genre • using a model which has been trained on anything but that genre.
We selected five different genres (Section 3), and asked participants to submit up to ten different models, as per the overview in Table 1. Obviously, if one participant wanted to have one single model for everything, they could submit one model for all settings. In the cross-genre setting, the only constraint is not using in training any single instance from the genre they are testing on. Other than that, participants were free to combine the other datasets as they wished.
Participants were also free to use external resources, provided the cross-genre settings were carefully preserved, and everything used was described in detail in their final report.
Measures As standardly done in binary classification tasks with balanced classes (see Section 3), we will evaluate performance using accuracy.
For each of the 10 models, five in the in-genre settings, and five in the cross-genre settings, we calculate the accuracy for the two classes, i.e. F and M. In order to derive two final scores, one for the in-genre and of for the cross-genre settings, we will simply average over the five accuracies obtained per genre. In-genre: 5
P Accijn genre Accin genre = j=1 5 and cross-genre: 5
P Accjcross genre Acccross genre = j=1 5 We keep the two scorings separate. For determining the official “winner”, we can consider the cross-genre ranking, as more specific to this task. Baselines For all settings, given that the datasets are balanced for gender distribution, through random assignment we will have 50% accuracy.
Twitter in-genre model non-Twitter model for Twitter YouTube in-genre model non-YouTube model for YouTube Children in-genre model non-Children model for Children Journalism in-genre model non-Journalism model for Journalism Diaries in-genre model non-Diaries model for Diaries 3
In order to test the portability and stability of profiling models across genres, we created datasets from five genres. We describe them below, together with the format and train/test split of the materials distributed to participants. In Figure 1 we provide a few samples to illustrate the variety of the data, and the format provided to the participations (Section 3.3). 3.1
We selected data from the following genres grounding our choice of both availability and wide variety.
Twitter Tweets were downloaded using the
Twitter API and a language identification
module to restrict the selection to Italian messages1.
Names from usernames were matched with a list
of unambiguous male and female names.
YouTube YouTube comments were scraped
using the YouTube API and an available scraper2.
Videos were pre-selected manually with the aim to
avoid gender biases, resulting in a selection from a
few general topics: travel, music, documentaries,
politics. The names of the comments’ authors are
visible, and gender was automatically assigned via
matching first names to the same list of male and
female proper names used for the Twitter dataset.
Children writing This dataset is a collection of
essays written by Italian L1 learners collected
during the first and second year of lower secondary
school called CItA (Corpus Italiano di
Apprendenti L1,
2https://github.com/philbot9/ youtube-comment-scraper the second year. It was collected during the two school years 2012–2013 and 2013–2014. News/journalism This dataset was created scraping two famous Italian online newspapers (La Repubblica and Corriere della Sera) and selecting only single-authored newspaper articles. Gender assignment was done manually.
Personal diaries In order to include personal writing which is more distant from social media, we collected personal diaries that are freely available as part of the Fondazione Archivio Diaristico Nazionale della Città di Pieve Santo Stefano.3 The documents are of varying but comparable sizes, and the author’s name is clearly specified in their metadata. Gender assignment was done manually. 3.2
For each genre we have a portion of training and a portion of test data. The distribution of gender labels was controlled for in each dataset (50/50). Additionally, we aimed at providing sets of comparable sizes in terms of tokens so as to avoid including training size as a relevant factor. This was intended for test, too, so as to have the same amount of evaluation samples, but due to limited availability we eventually used a smaller test set for the Diary genre.
Table 2 shows the size of the training and the test sets in terms of tokens and authors. The datasets are composed by texts written by multiple users, with possibly multiple documents per user. It is also possible that in the Twitter and YouTube datasets, different texts by the same user ended up both in training and in test. For what concerns the Children writing dataset, training and test contain texts written by same 155 children. Differently from the Children training set, the Children test set is composed by texts written on the same 3http://archiviodiari.org/index.php/ iniziative-e-progetti/brani-di-dirai. html.
<doc id="778" genre="twitter" gender="M"> @edmond644 @ilsussidiario Sarebbe vero se li avessimo eletti ma, non avendolo fatto, "altri" se li meritano. </doc> <doc id="1" genre="children" gender="M"> Questa estate mi sono divertito molto perché mio padre ha preso casa nella località del Circeo. La casa era a due piani, al piano terra c’era un giardino dove il mio gatto sela spassava. C’era molta ombra nel giardino e io mi ci addormivo sempre. Il mare era poco lontano da casa e ci andavamo ogni giorno e giocavamo a fare i subacquei. Siamo andati a mangiare la pizza fuori ed era molto buona. </doc>
<doc id="8493" genre="youtube" gender="F"> alla fine esce sempre il tuo lato gattaro! sei forte! bellissimo video come sempre! </doc>
<doc id="118" genre="journalism" gender="F"> Elogio alla longevità, l’intervista bresciana a Rita Levi Montalcini Trent’anni fa il Nobel a Rita Levi Montalcini. Ecco l’ultima intervista bresciana a cura di Luisa Monini: «I giovani credano nei valori, i miei collaboratori sono tutte donne» Tra le numerose interviste che Rita Levi Montalcini ha avuto la bontà di concedermi, mi piace ricordare l’ultima, quella dei suoi 100 anni. Eravamo nello studio della sua Fondazione e lei era particolarmente serena, disponibile. Elegante come sempre. [...] </doc>
<doc id="107" genre="diary" gender="F"> 23.9.80 Sergio, volutamente stai coinvolgendo Alessandro in questa nostra situazione, invece di tenerlo fuori: sai quanto è sensibile, quanto è fragile, quanto è difficile anche - né puoi ignorare che non solo lui in particolare ma nessun ragazzino di 14 anni è in grado di subire o di affrontare o di sostenere una prova così dolorosa.
Lo stai distruggendo, impedendogli di riflettere da solo, martellando di parole (o scritti addirittura, come quella tua dichiarazione) per sentirti meno solo o per annullare la sua volontà e imporgli la tua, come volevi fare con me: ma non ti rendi conto che non è amore il tuo, [...] </doc> topic at the same time, at the end of the two school years. For News and Diaries we made sure no author was included in both training and test. We did not balance the number of users per genre, nor the number of documents per user, assuming these as rather natural conditions. The data was distributed as simil-XML. The format can be seen in Figure 1. Although we distributed one file per genre, we still included the genre information in the XML so as to ease the combination of the different files. 4
Following a call for interest, 15 teams registered
for the task and thus obtained the training data.
Eventually, three teams submitted their
predictions, for a total of 50 runs. Three different runs
were allowed per task, and one team experimented
with three different models submitting three
different predictions for each of the 10 subtasks. A
summary of participants is provided in Table 3,
while Tables 4 and 5 report the final results on the
test sets of the EVALITA 2018 GxG Task.
CapetownMilanoTirana proposed a classifier
based on Support Vector Machine (SVM) as
learning algorithm. They tested different n-gram
features extracted at the word level as well as at the
character level. In addition, they experimented
feature abstraction transforming each word into a
list of symbols and computing the length of the
obtained word and its frequency
In this section we provide both a discussion of the approaches and an analysis of the results. 5.1
Participants experimented with more classical machine learning approaches as well as with neural networks. Results show that while neural models achieve globally more accurate results, feature engineered SVMs are as competitive. This holds both in the in-genre and in the cross-genre settings.
All models suffer to some extent
from the shift to cross-genre, though the
CapetownMilanoTirana-SVM system appears to
be the most robust. This might be due more to
the choice of (abstract) features, rather than the
learning algorithm itself. This system also
employs bleaching (a technique to fade out lexicon
in favour of more abstract token representation)
in this GxG cross-genre setting, after it had
shown promise in a cross-lingual profiling task,
where it was firstly introduced
Research Group
Symanto Research, CoGrammar, freelance researcher
Università Orientale di Napoli
ItaliaNLP Lab, ILC-CNR, Pisa
# Runs
10
10
30
results on training data, where they also perform
an evaluation of feature contribution, it seems
that bleaching in this context does not yield the
expected benefits
The use of external resources was globally little explored, with the exception of generic word embeddings (ItaliaNLP Lab). While such embeddings do not seem to have contributed much to performance, specialised lexica or embeddings could be something to be investigated in the future.
From a learning settings’ perspective, teams chose quite straightforward strategies. In-genre, all models were trained using data from the target genre only, with the exception of the ItaliaNLP Lab-MTL model, where the adopted multitask strategy could use the knowledge from other genres, even in the in-genre settings. This seems confirmed comparing the ItaliaNLP Lab-MTL’s results with the twin model ItaliaNLP Lab-STL (the same architecture with a single-task setting).
In the cross-genre scenario, all systems have used as training all of the available datasets apart from the dataset from the target genre. It appears that no team tried to exploit functions of (dis)similarity among genres in order to select subportions of data for training when testing on a given genre. The only different model in the way the training data from the other genres is used is the ItaliaNLP Lab-MTL, but its performance on the cross-genre setting indicates that this approach is not robust for this task. 5.2
The in-domain results (Table 4) are useful to identify which genres are overall easier to model in terms of the author’s gender, and provide an overview of gender detection in Italian.
As could be expected, Diaries are the easiest
genre to model. This might result from the fact
that the texts are longer, and are characterised by
a more personal and subjective writing style. For
example, the collected diaries present an extensive
use of the first and second singular person verbs
and a higher distribution of possessive adjectives.
Due to availability, the Diaries test set is smaller
than the others, providing thus fewer instances for
evaluation (see Table 2) and possibly weaker
reliability of results. However, from the analysis of
results reported by
We also see that Children writings carry better signal towards gender detection than social media. This might be due to the fact that the Children test set is composed by documents characterised by a common prompt (in the original collection settings, this was meant to provide evidence of how students perceive the different writing instructions received in the considered school years). This feature makes the children texts a reflexive textual typology, typically characterised by a more subjective writing style as we observed also for Diaries.
Both Twitter and YouTube score above a 50%
baseline, but are clearly harder to model. This
could be due to the short texts, which in some
cases offer very little evidence to go by.
Compared to previous results reported for gender
detection on Twitter in Italian, as obtained at the
2015 PAN Lab challenge on author profiling
The most difficult genre is Journalism. While
texts can be as long as diaries, results suggest
that the requested jargon and writing conventions
typical of this genre overshadow the gender
signal. Moreover, while we have selected only
articles written by a single author, there is always
the chance that revisions are made by an editor
before the piece is published. This is the only genre
where some models fail to beat the 50% baseline.
However, it is interesting to note that the highest
score in-genre is achieved by UniOr, which uses
a selection of stylometric features
Cross-genre, we observe that results are on average lower, but only by 2:5 percentage points (55:8 vs 53:4), which is less than one would expect. Some models clearly drop more heavily from in-genre to cross-genre (ItaliaNLP Lab-MTL: :578 vs :528 average accuracy, ItaliaNLP Lab-SVM: :575 vs :532, UniOr: :535 vs :498). However, others appear more stable in both settings (CapetownMilanoTirana-SVM: :564 vs :549), or even better at the cross- rather than in-genre prediction (ItaliaNLP Lab-STL: :538 vs :562).
From a genre perspective, the drop is more substantial for some genres, with Diaries losing the most, with large variation though across systems. For example, the model that achieves best performance on Diaries in-genre (ItaliaNLP LabMTL, :676) suffers a drop of almost eight percentage points on the same dataset cross-genre (:595). Conversely, CapetownMilanoTirana preserve a high performance on the Diaries testset in both in- and cross-settings (:635), yielding the highest cross-performance on this genre. Twitter shows the least variation between in- and crossgenre testing. Not only the losses for all systems are minimal, but in some cases we even observe higher scores in the cross-genre setting. ItaliaNLP Lab-STL obtains the highest score on this test set in both settings, but the performance is higher for cross- than in-genre (:609 vs :595).
Finally, some possibly interesting insights also emerge from looking at the precision and recall for the two classes (which we do not report in tables as there are too many data points to show). For example, we observe that for some genres only one gender ends up being assigned almost entirely by all classifiers. This happens in the cross-genre settings, while the same test set has rather balanced assignments of the two classes in the in-genre settings. In the case of Journalism, all systems in cross-genre modelling almost only predict female as the author’s gender. At the opposite side of the spectrum we find YouTube, where almost all test instances are classified as male by almost all systems, cross-genre. In this case though we also see high recall for male in the in-genre setting, though not so dominant. While more grounded considerations are left to a deeper analysis in the future, we could speculate that some genres are globally seen by classifiers as more characteristic of one gender or the other, as learnt from a large amount of mixed-genre, gender-balanced data. 6
Gender detection was for the first time the focus of a dedicated task at EVALITA. The GxG task specifically focussed on comparing performance within and across genres, building on previous observations that high performing systems were likely to be modelling datasets rather than gender, as their accuracy substantially dropped when tested on different, even related, domains.
Results from 50 different runs were mostly
above baseline for most prediction tasks, both
in-genre and cross-genre, but not particularly
high overall. Also, the drop between in-genre
and cross-genre performance is noticeable, but
marginal. Neural models appear to perform only
slightly better than a more classic SVM which
leverages character and word n-grams. The use
of the recently introduced text bleaching strategy
among the engineered features (aimed at reducing
lexicon bias
In the near future, it will be interesting to
compare these results to human performance, as it has
been observed that for profiling human do not
perform usually better than machines
We would like to thank Eleonora Cocciu and Claudia Zaghi for their help with data collection and annotation. We are also grateful to the EVALITA chairs for hosting this task.