English. This report was written to describe the systems that were submitted by the team “TheNorth” for the HaSpeeDe 2 shared task organised within EVALITA 2020. To address the main task which is hate speech detection, we fine-tuned BERT-based models. We evaluated both multilingual and Italian language models trained with the data provided and additional data. We also studied the contributions of multitask learning considering both hate speech detection and stereotype detection tasks.
Organised as part of the 7th EVALITA
evaluation campaign
Beside the main task of binary hate speech classification - aimed at deciding whether a message contains hate speech or not - the HaSpeeDe 2 shared task has two more sub-tasks. One being stereotype detection, and the other the identification of nominal utterances. All tasks being evaluated both on in-domain (tweets) data, and outof-domain (newspaper headlines) data. Here, we
Copyright c 2020 for this paper by its authors. Use
permitted under Creative Commons License Attribution 4.0
International (CC BY 4.0).
tackle both the main task, and the first sub-task
of Stereotype Detection that is potentially useful
for the main task. For this sub-task the
organisers use the following definition of Stereotype: “a
standardized mental picture that is held in
common by members of a group and that represents
an oversimplified opinion, prejudiced attitude, or
uncritical judgment”
Here, we have two binary classification tasks. A
simple way to perform text classification is based
on bag-of-words representation counting the
number of occurrences of each word within text. It is
often combined with term frequency-inverse
document frequency
In
We have chosen to deepen the approach of finetuning a BERT based language model, comparing multilingual and monolingual settings. We also assessed the contribution of additional hate speech data from different online sources. We finally submitted the results of the same model fine-tuned with and without multitask learning between hate speech and stereotype detection tasks. 2.1
The chosen classification approach is to fine-tune
a BERT-based language model. This kind of
approach is the state-of-the-art for many text
classifications tasks today
It has been shown that it is possible to fine-tune these models for many downstream natural language processing tasks, including the one we are interested in, which is text classification. This can be achieved by removing the language modelling head and replacing it by a head appropriate for the target task. The designers of BERT prepared this by adding a token at the beginning of each text sequence, named CLS for classification. The purpose of this token is to contain the information useful for the classification task at the end of the forwarding process. Then a classifier head can just take this CLS token as input to classify the whole text sequence. In our case we decided to add a simple linear layer with a softmax on top of it, for simplicity and because it is efficient enough since the other layers are fine-tuned. 2.2
An important consideration of fine-tuning
described in
Moreover, the learning rate can be gradually decreased in the first layers of the models. It aims at limiting the update in these first layers that have been showed to contain the most primal information about the language. One can think of the classical example in computer vision neural networks where the basics shapes features are extracted by the first layers and the task-specific combinations are processed in the last ones. Thus we applied layer-wise learning rate with the following geometric equation: the learning rate in a layer is the one of the following multiplied by a decay factor between 0 and 1.
LRk 1 =
LRk where LRk is the learning rate of the k-th layer.
Then the case when is one is the case of classic fine-tuning with the same learning rate everywhere, and the case when is zero is the case of feature extraction with the whole language model weights that are frozen and only the parameters of the classification head are trainable. This hyperparameter was learned with the others during the hyper-parameters tuning process. 2.3
We compared the use of several language
models. Many models similar to BERT have been
trained since 2018, and a lot are available for use.
Although the models are often first and foremost
trained for English, multilingual models have been
trained on data of several languages in order to
counteract the lack of data for some languages. It
is the case of mBERT and XLM-Roberta
We evaluated two multilingual models, mBERT
and XLM-RoBERTa, and three Italian
monolingual models, AlBERTo, UmBERTo, and
PoliBERT. AlBERTo was pretrained on TWITA, that
is a collection of Italian tweets
We tried to use more data, with different settings. For the multilingual models, we could use all type of hate speech data. For the monolingual models, we used the little data available for Italian but we tried also to use translated multilingual data. These additions were not conclusive, so we stuck to the HaSpeeDe 2 data for the submissions. 2.4
The tuning of the hyper-parameters is relevant in
order to get good results, and that is especially
the case for the learning rate and the layer-wise
decay factor . We tuned hyper-parameters with
random search which has been shown to be
often more efficient than grid-search
Learning rate Layer-wise Batch Size Max Length Language Model
2.10-4 0.35
32 100 UmBERTo
It is very important that the learning rate and the layer-wise multiplier are tuned simultaneously because the choice of the multiplier strongly modifies the amplitude of the gradient. 2.5
We evaluated the usage of multitask learning
between the two classification tasks of the
competition that are hate speech detection and stereotype
detection. Multitask learning consists of learning
to perform several tasks. It can be done by
learning the tasks simultaneously with common first
layers but task-specific heads
A question that arises when doing multitasking
is the way to combine the loss of the tasks in one.
The simple solution is to sum them uniformly. It
might not be the best solution when there is
imbalance between the tasks, for instance when the scale
of the outputs of one is much higher than the
others. A solution brought by
Two submissions are allowed during the HaSpeeDe 2 test phase. We chose to submit a fine-tuned UmBERTo trained separately for each of the two tasks and a fined-tuned UmBERTo with multitasking on both Stereotype and Hate Speech detection. The hyper-parameters used to train these models were presented in Table 1.
Since we compared the different language models with 5-fold cross-validation, we then ensembled the 5 models obtained for each fold in order to get the final model. The ensembling was done by considering the mean of the probabilities returned by each model. 3
The organisers provided a train dataset of 6,839 tweets, annotated with Hate Speech and Stereotype labels (as described in Table 2).
Development Data (Tweets) 0.404 Test Data (Tweets) 0.492 Test Data (News) 0.362
0.445 0.450 0.350
The test data of HaSpeeDe 2 consists of two subsets: an in-domain set (1,263 tweets) and an out-of-domain set (500 newspaper headlines).
The hate speech labels are slightly unbalanced towards non-hate speech. Thus we tried to use adapted losses to prevent tendency towards nonhate speech predictions. We used class-weighted loss, which assigns a higher weight to the observations from the minority class in the computing of the loss. We also tried to use a smoothed F1score – a differentiable loss in phase with the F1. Neither approach improved the results in a significant way.
The pre-processing was simple. We removed emoticons and hashtags and we replaced urls and user names with associated tags as done in the evaluation data. Each tweet was padded with a size of 100. Then we used the pre-processing and tokenization pipeline specific to each language model as provided by the authors of the models. 4 4.1
The metric used for the evaluation is the macro F1-score. The F1-score of a class is computed by calculating the harmonic mean between the precision and recall for this class. The macro F1-score is the mean between the F1-scores for each class. It is less sensitive to the imbalance between the classes. 4.2
We used several baselines to evaluate our results during the development process. The first ones are those obtained by dummy classifiers, one that always predicts the most frequent class and the other one that makes a random stratified prediction according to the distribution of the classes in the training data. We also computed the results of more developed systems, that are a TF-IDF bag of words and a BiLSTM with trainable word vectors inputs.
The HaSpeeDe 2 organisers provided two baseline systems after the results were submitted. The first is a most frequent class predictor and the second is a linear SVM with unigrams, char-grams and TF-IDF representation. 4.3
We tuned the hyper-parameters for each evaluated language model as described in Section 2.4. For each language model, we then computed 5-fold cross-validation results on HaSpeeDe 2 training data. The averages of the 5 macro F1-scores are shown in Table 3.
HS
Most Frequent Class 0.374 TF-IDF Bag-of-words 0.703 Word vectors + BiLSTM 0.721
Multilingual language models mBERT 0.757 XLM-RoBERTa 0.761
Italian language models AlBERTo 0.773 PoliBERT 0.795 UmBERTo 0.799
Ster The scores of our two systems evaluated on the HaSpeeDe 2 test data are summarized in Table 4. These systems are 5 UmBERTo models trained on each of the 5 training folds and ensembled. The second system is the same as the first with the use of multitask learning.
Hate Speech Detection Most Frequent Class 0.337 Classic Features + SVM 0.721 UmBERTo 0.790 UmBERTo + Multitasking 0.809 Best HaSpeeDe 2 0.809
Stereotype Detection Most Frequent Class 0.355 Classic Features + SVM 0.715 UmBERTo 0.772 UmBERTo + Multitasking 0.768 Best HaSpeeDe 2 0.772 News 0.389 0.621 0.671 0.660 0.774 0.394 0.669 0.685 0.647 0.720 According to Table 3, multilingual models performed worse than monolingual models based on HaSpeeDe 2 data alone, although they achieved respectable results.
Moreover, even when we used additional data from other languages to train the multilingual models, they still did not manage to outperform the monolingual models, as we were hoping they would.
Within the Italian models, UmBERTo and PoliBERT performed better than AlBERTo on these tasks. While the good performance of PoliBERT can be linked to its pre-training for a tweet classification task (sentiment analysis) potentially useful for hate speech detection, it is more difficult to explain the competitiveness of UmBERTo, which was trained on data not coming from Twitter and less numerous than for AlBERTo. One explanation could be the better quality of this data, or a better optimisation by its creators. 5.2
Our results on the HaSpeeDe 2 test dataset are summarized in the Table 4. The results obtained on in-domain data correspond to what we expected from our cross-validation results. Our systems achieved the best macro F1-scores on the indomain test set (Tweets) for both hate speech and stereotype detection. However, the results on outof-domain data (News) are far from being as good. This can be explained by the different distribution of this data compared to the training data.
Table 5 shows the confusion matrix for our first system evaluated on out-of-domain data. The error is mostly due to the high number of false negatives. The classifier predicts too many sequences as non-hate speech. This suggests that this classifier trained with hate speech on Twitter is struggling to detect hate speech in newspaper headlines. It can be assumed that hate speech in newspapers is more subtle, with less coarseness and aggressiveness that make it easier to detect on Twitter.
312 117
Predicted True
7 64 We have chosen to submit a system with multitask learning on both Stereotype and Hate Speech detection and an other one without, in order to study the benefits of it. Indeed, the system with multitasking learning performed much better on the indomain data for the hate speech detection task. It is not the case however for the out-of-domain data, neither for the stereotype detection task.
Table 6 describes in more detail the differences between the predictions of the two systems for data containing stereotypes and data not containing stereotypes. We observed that the improvement linked to multitask learning consists mainly in a reduction in the number of false positives in favour of the number of true negatives in data not labeled as Stereotype. Assuming that hate speech makes significant use of stereotype, one could suppose that the multitask model has learned to discard some data that do not have the characteristics of stereotypes and are therefore unlikely to contain hate speech.
Predicted False Predicted True +3 -3 +7 -7
Predicted False Predicted True +28 -28 +1 -1 In this work, we compared the fine-tuning of multilingual and monolingual BERT-based language models for hate speech detection. We also investigated the addition of multitask learning with the Stereotype detection task linked to hate speech. We obtained the best macro F1-scores of HaSpeeDe 2 on the in-domain test data. However, the results were worse for out-of-domain test data, and further research could be conducted to better understand the reasons for this and address it.
AuxCoRR,