In this paper we describe the systems we used to participate in the task TAG-it of EVALITA 2020. The first system we developed uses linear Support Vector Machine as learning algorithm. The other two systems are based on the pretrained Italian Language Model UmBERTo: one of them has been developed following the Multi-Task Learning approach, while the other following the Single-Task Learning approach. These systems have been evaluated on TAG-it official test sets and ranked first in all the TAG-it subtasks, demonstrating the validity of the approaches we followed.
Author Profiling (AP) is a known Natural
Language Processing task consisting in the extraction
or the prediction of information about the authors
of some disputed documents. Such information
can include the age and the gender of the authors.
The AP problem is assuming more and more
importance in several fields, such as security,
forensics, marketing and sales, and so on. For example,
in forensics, detecting the age and the gender of
the author of a given document can be very helpful
for determining whether a person should be
considered as a suspect or not; from the marketing
and sales’ perspective, companies can understand
what kind of people may or not like their products
on the basis of the analysis performed on people’s
reviews or blog and social network posts
In the context of EVALITA 2020
Copyright © 2020 for this paper by its authors. Use
permitted under Creative Commons License Attribution 4.0
International (CC BY 4.0).
ural Language Processing and speech tools for
the Italian language, the task TAG-it
De Mattei and Cimino (2018) and Cimino et
al. (2018) demonstrated the validity of Multi-Task
Learning approach to establish the state of the art
for several Italian NLP task, in the context of GxG,
Cimino et al. (2018) developed the best system
for this task based on Bidirectional LSTM
(BiLSTMs) trained using a Multi-Task Learning
approach. For TAG-it we replicated the same
approach: we developed a baseline system based
on SVM, and two neural systems, the first one
exploiting a Single-Task Learning approach, the
second one a Multi-Task Learning approach.
Instead of the Bi-LSTM model used by Cimino et
al. (2018) for TAG-it we exploited a deeper neural
pretrained language model: BERT
We implemented and tested three different systems. Our early experiments were led on a training set and a test set obtained by shuffling and splitting (80% training - 20% test) the training set provided by the organisers in order to analyse the classifiers’ performances on a labeled dataset. At the end of our experiments, we trained our best classifiers on the whole training set and run them on the TAG-it test sets provided by the organisers.
For our experiments and runs, as a preprocessing phase, we filtered out all posts less than 20 characters in length and labeled each post of the dataset with the corresponding author’s id, gender, age and topic. In Table 1 we report the distributions of the classes of the TAG-it dataset.
Train Test1 M 15070 315 F 3113 96 0-19 2232 39 20-29 5412 131 30-39 3569 95 40-49 3577 69 50-100 3393 77 ANIME 3925 97 AUTO-MOTO 3648 76 BIKES 468 12 CELEBRITIES 1063 22 ENTERTAINMENT 534 9 MEDICINE-AESTHETICS 370 16 METAL-DETECTING 1471 26 NATURE 481 11 SMOKE 1574 30 SPORTS 4593 103 TECHNOLOGY 56 9 GAMES 0 0 ROLE-GAMES 0 0 CLOCKS 0 0
As a first step, our systems make their predictions by classifying the three dimensions post by post. Then they use a voting mechanism according to which the gender, the age and the topic of an author are represented by the most frequent values assigned by the classifiers to his/her posts.
The first system we implemented uses linear Support Vector Machine as learning algorithm and we used different features for predicting the core dimensions of the dataset, the second system is based on a Single-Task Learning BERT model and the third system is based on a Multi-Task Learning BERT model. In particular, we used UmBERTo1, an Italian pretrained Language Model developed by Musixmatch.
In the following subsections we will describe these systems in detail. 2.1
As regards the system based on three linear
SVM statistical models, we used the scikit-learn2
Python library and we conducted several
experiments by testing different configurations for
feature extraction. In all the experiments we used the
TF-IDF vectorizer, but we changed the tokenizer
and the n-grams context window. In particular we
tested five different kinds of features: character
n-grams, word n-grams, lemma n-grams,
Part-OfSpeech n-grams and bleached tokens. As regards
the bleached tokens features, they were extracted
after performing a bleach tokenization consisting
in fading out lexicon in favour of an abstract
token representation
These results led us to choose the best SVM classifiers for the official runs on the provided test set; analysing them, we can state that the best SVM classifiers tested in our experiments are the following: • Topic Detection: One-Vs-Rest Linear SVM using features extracted through a TF-IDF Vectorizer considering character n-grams; • Age Detection: One-Vs-Rest Linear SVM using features extracted through a TF-IDF Vectorizer considering lemma n-grams; • Gender Detection: Linear SVM using features extracted through a TF-IDF Vectorizer considering word n-grams.
1https://github.com/
musixmatchresearch/umberto
2https://scikit-learn.org/stable/
3https://spacy.io
word n-gram
char n-gram
lemma n-gram
pos n-gram
bleached words
Our second system consists of three different
BERT models and a classifier on top of each of
them. More precisely, we used the UmBERTo
language model, which was pretrained on a large
Italian Corpus: OSCAR
This language model have 12-layer,
768hidden, 12-heads, 110M parameters. On top of
the language model we added a ReLU classifier
In Table 3 we report the system’s performances in terms of f1-score obtained on the validation set.
Our last system is based on a unique UmBERTo
model and three classifiers on top of it, each one
responsible of predicting one of the three core
dimensions of the dataset according to the
MultiTask Learning approach used in
In Table 4 we report the system’s performances in terms of f1-score obtained on the validation set.
We run all our three systems on the test sets provided by the task organisers. The performances of our systems are reported in Table 5.
For the Task 1 scoring, TAG-it considers two different rankings. The first ranking is obtained using a partial scoring scheme, giving 0 points if no correct predictions are provided for the three dimensions of the dataset, 1/3 points if one out of three correct answers is given, 2/3 points if two out of three correct answers are given and 1 point if all the answers given by the system are correct. The second ranking assigns 0 points if no correct predictions are provided for the three dimensions of the dataset and 1 point if all the answers given by the system are correct. In both cases, the final score is the sum of the points achieved by the system across all the documents normalized with respect to the number of documents in the test set. For the Task 2, the micro-average f-score is used as scoring function.
Task 1 metric 1 Task 1 metric 2 Task 2a Task 2b
STL-SVM 0,6626 0,253 0,8519 0,3742
MTL-BERT STL-BERT 0,7178 0,7348 0,3090 0,3309 0,9247 0,9053 0,3667 0,4093
Analysing the scores in Table 5, we can state that the best system in the TAG-it context is the one based on BERT using the Single-Task Learning (STL-BERT) approach, obtaining the best scores in Task 1 and Task 2b (age prediction). In Task 2a, consisting in gender prediction with topic control, the best system is the Multi-Task Learning BERT-based system (MTL-BERT). Hence, the systems based on deeper neural models outperform the systems based on traditional machine learning techniques, i.e. the SVM (STL-SVM).
Task 1: In order to compare classifiers’ predictions on Task 1 with regard to each dimension and to understand the correlation between labels, we plotted and analysed some distributions.
In Figure 1, we reported the distribution of the labels in the test set and in the classifiers’ output. As regards the gender prediction (a), we can note that the STL-SVM classifier overestimates the M class, most likely because the M and F classes are very unbalanced in the training set. STL-BERT and MTL-BERT’s distributions, on the contrary, are closer to the test set’s one: in our setting the neural models appear less affected by the imbalance of a training set.
Observing the distributions of the Age classes in Figure 1 (b), we can observe that for all the three systems the distributions of the labels are not close to the distribution of the test set. The nearest distribution is the one of MTL-BERT’s output.
Looking at the Topic classes distributions in Figure 1 (c), we can observe, once again, that the SVM-based system’s one is the less close to the test set in that it has the tendency to overestimate the SPORT, ANIME and AUTO-MOTO classes and it does not recognise the BIKES and TECHNOLOGY classes as they are underrepresented in the training set (respectively the 2.574% and the 0.308% of training set). For the same reason, it has difficulties in recognising the classes ENTERTAINMENT, MEDICINE-AESTHETICS and NATURE (which are respectively the 2.937%, 2.035% and 2.645% of the training set). The two BERT-based systems, on the contrary, are less affected by this imbalance of the training set and their predictions reflect more the reality of the test set, even though, as STL-SVM, also MTL-BERT cannot recognise the BIKES and TECHNOLOGY classes.
In Figure 2 we report the distribution of the Age classes with respect to the Topic classes. Figure 2 (b) shows that in the STL-SVM’s output the 0-19 age class is only related to the ANIME topic, the age 20-29 is related more or less with all the detected topics, the 30-39 class is mostly related to SMOKE and MEDICINE-AESTHETICS, the 40-49 class to the METAL-DETECTING, AUTO-MOTO and SMOKE topics and the 50-100 class mostly to AUTO-MOTO, SPORTS and CELEBRITIES. This distribution is quite far from the test set one and it seems that the relation between the class 0-19 and the topics is overestimated. In Figure 2 (c), which refers to MTL-BERT, we can note that authors classified as having age 20-29 are predicted to talk mostly about ANIME, CELEBRITIES, NATURE and SPORTS and are less related to ENTERTAINMENT, MEDICINE-AESTHETICS and NATURE topics than in STL-SVM’s output; the relation between the 30-39 class and ENTERTAINMENT and MEDICINE-AESTHETICS categories on one hand, and 50-100 and AUTO-MOTO, MEDICINE-AESTHETICS, METAL-DETECTING, NATURE and SMOKE on the other is stronger than in STL-SVM’s results. Also this distribution, though, is quite far from the test set’s one, even if ages seem to be more distributed than in STL-SVM’s output. As shown in Figure 2 (d), in STL-BERT’s distribution, the age 0-19 seems mostly related to TECHNOLOGY and ANIME. The class BIKES, which has not been recognised by the other systems, is related to the classes 30-39, 40-49 and, mostly, 50-100. As regards the 20-29 class, its relations are quite similar to the ones found in the STL-SVM’s results, except for the class NATURE, which is related also to the ages 0-19, 40-49 and 50-100. Also this distribution is quite far from the test’s one. All the three distributions differ considerably from the test set because systems do not perform well enough in age prediction.
The distributions of the topics with respect to gender in the test set and the predictions are reported in Figure 3. As shown in the figure, all the three systems results relate the F class mostly to the ANIME topic, as it is also in the test set. In the STL-SVM’s output, though, this relation seems to be overestimated. Moreover, in STLSVM the F class, besides ANIME, is only related to a much lesser extent to SMOKE. The relation between M and SMOKE seems to be overestimated too with respect to the test set. As regards the F class in MLT-BERT and STL-BERT outputs, topics are more distributed than in STLSVM, but the nearest to the test set’s one is STL-BERT: MLT-BERT, in fact, seems to overestimate the relation between F and BIKES and ENTERTAINMENT and to underestimate the relation between F and MEDICINE-AESTHETIC and SPORTS. For what concerns the M class in MLT-BERT and STL-BERT distributions, we can state once again that the distribution which is closer to the test set one is given by STL-BERT: STL-SVM, MLT-BERT overestimates the relation between M and SMOKE and NATURE.
The results reported in Table 5 show that for Task 2a (gender prediction with topic control) the best classifier is MLT-BERT. In this subtask, BERT-based systems outperform in a significant way the system based on SVM.
As regards the Task 2b, consisting in the age prediction, the best metrics belong to the STLBERT. In the age prediction the gap between all the systems’ metrics is not very high. In this case, in which only the age dimension must be predicted, the best classifier is the one using a SingleTask Learning approach. 4
In this paper we reported the performances and the results of the systems we used to participate to the TAG-it task of EVALITA 2020. We compared our systems’ performances and noted that in the case in which the goal is to predict topic, age and gender dimensions at once, and in the case in which only the age must be predicted, the best classifier is the one developed using a SingleTask Learning approach and based on transformers. In the case in which the goal is the gender prediction only a Multi-task Learning approach combined with transformers have slightly better performances. These results prove that the proposed systems based on transformers, are more effective than traditional machine learning techniques in topic, age and gender classification achieving the state of the art for TAG-it shared task. Using deep pretrained language models on this task Multi-Task Learning does not provide any relevant boost of performances.