English. This paper explains the system developed for the Hate Speech Detection (HaSpeeDe) shared task within the 7th evaluation campaign EVALITA 2020 (Basile et al., 2020). The task solution proposed in this work is based on a fine-tuned BERT model. In cross-corpus evaluation, our model reached an F1 score of 77,56% on the tweets test set, and 60,31% on the news headlines test set.
The detection of Hate Speech has been a popular task in Natural Language Processing. Because there is no universal definition of the term ’hate speech’, we follow the EVALITA 2018 organizers in defining it as any expression ”that is abusive, insulting, intimidating, harassing, and/or incites to violence, hatred, or discrimination. It is directed against people on the basis of their race, ethnic origin, religion, gender, age, physical
Copyright © 2020 for this paper by its authors. Use
permitted under Creative Commons License Attribution 4.0
International (CC BY 4.0).
condition, disability, sexual orientation, political
conviction, and so forth”
Apart from being hurtful to the person or group
that the hateful message is aimed at, its
systematic usage can be the cause of hate crime and other
criminal acts towards these groups. Mass and
social media help to spread hate speech a lot faster
than traditional communication channels
In previous work, automated detection of hateful
messages has been approached in various ways,
starting from simpler lexicon-based approaches
and Naive Bayes classifiers to more state of the art
Convolutional Neural Networks
Early work in this area includes Spertus’
automatic recognition of hostile messages with
the Smokey system. She found that only 12%
of such messages contained explicit keywords.
Therefore, she compiled a set of rules resulting
in a 47-element feature vector per sentence to
capture semantic and syntactic information.
For instance, imperative statements have higher
chances of containing insulting content than
indicative utterances. The same applies to sentences
starting with you. For evaluation, decision trees
were trained on the vectors and the results were
compared to human assessments. Overall, in 36%
of the cases the instances labeled as insulting
matched with the human classification.
Another approach introduced by Greevy and
Smeaton in 2004 involved support vector
machines for classifying racist texts. In their work,
they compared part-of-speech distributions across
racist and non-racist documents as well as
different feature representations like bag-of words and
bigrams. The bag-of-words model was found to be
more useful than the bigram model (accuracy of
87.77% vs. 84.77%)
Since around 2015 and with the gaining
popularity of deep learning, various methods
involving neural networks have been proposed.
For instance, Kamble and Joshi compared a CNN,
LSTM, and BiLSTM to one another for detecting
code-mixed Hindi-English hate speech within the
context of ICON 2018. The CNN was fed with
domain-specific embeddings and showed the best
performance (F1 score of 80.85%)
In 2018, the EVALITA organizers presented
three subtasks: In the first task, Facebook data
was used to classify a message as not hateful (0)
or hateful (1) and in Task 2, the same challenge
was conducted on Twitter data. Task 3 asked the
participants to train on the Facebook data and
test on the Twitter data, and vice versa. With an
F1 score of 0.82, the best performance on the
Facebook task was achieved by a team that used
polarity and subjectivity lexicons as well as two
word-embedding lexicons as external resources
together with a 2-layer BiLSTM. The same team
reached the best performance for the Twitter data
(F1 score of 0.79). However, systems that were
cross-corpus tested performed significantly worse
with an F1 score of 0.65% with the Facebook
training set and 0.69% with the Twitter train
data. The former score was achieved with a
neural network with three hidden layers involving
word embeddings that were trained on previously
extracted Facebook comments; the latter was
once again achieved by the team with the 2-layer
BiLSTM
We participated in subtask A of HaSpeeDe – a
binary classification task to predict the presence
or absence of hate speech in Italian Twitter
messages
id
1940
6777
text
Ma quindi solo io sono
preoccupato che il terrorista stava in
Italia?
Cacciamo tutti gli immigrati visto
che sono un pericolo
hs
0
1
To solve the task, we fine-tuned the language
model Bidirectional Encoder Representations
from Transformers (BERT). BERT was developed
by Google and offers great possibilities not only
for hate speech detection, but for all kinds of
tasks that involve processing natural language
For faster and more efficient processing while fine-tuning the model, we used Google Colab (https://colab.research.google. com) in all experiments as it provides free GPU. We further experimented with the training data by comparing model performance on the data as it was provided by the event organizers and after cleaning it. Leaving data as is could have several advantages: On the one hand, it can be helpful to leave in junk characters that appear in tweets as well as trailing white spaces. For instance, a tweet written in all capital letters might indicate an insult and therefore contain useful information for the classifier. On the other hand, the task at hand did not solely require hate speech detection on social media but was evaluated on newspaper articles. Therefore, the model might adapt too much to the specific style of the Twitter genre and lower classifier performance when trying to generalize to another domain (like newspaper articles where these kinds of characters do not occur). For both our runs of the final model we cleaned the data as previous test runs showed better performance. 4.1
System Description To solve the task, we fine-tuned a BERT model. After experimenting with the different language models as described in the previous section, we found the bert-base-italian-cased model to be the best fit. The data was split into training and validation set during the first phase of the training. Cross-validation was used on the training set to prevent overfitting, and the validation set was used to assess how the model will generalize to unseen data. In the second training phase, the whole training data was used for training purposes.
Before experimenting with different estimators, the data was cleaned from @user-marks, trailing whitespaces, and we corrected errors like ”&” to ”&”. Since BERT is an already trained language model, extensive preprocessing of the data is not unnecessary. However, we assume that some preprocessing will be useful for crossdomain evaluation. After preprocessing, the text data was tokenized by the Italian BERT tokenizer (AutoTokenizer) that splits texts into tokens. It adds special [CLS] and [SEP] tokens to mark that the sentences can now be used for classification purposes and to separate sentences so that each token within a sentence can be assigned a segment token. Afterwards, the tokens are converted into token ids using the pre-defined indices of BERT’s tokenizer vocabulary. Additionally, those embeddings are also assigned attention masks that specify how much attention the system should pay to each of the words.
Since we implemented BERT with PyTorch,
we used the optimization module AdamW for
finetuning. Finding a good learning rate can be
difficult. AdamW takes care of this issue by
adapting the learning rates for different
parameters which makes the training process more
efficient
Even though we are dealing with binary classification, the model makes predictions by calculating probabilities using the softmax function. Moreover, we used a threshold of 0.9% to reduce prediction errors; 90% certainty is very high when we compare the default threshold of 50% that is typically used for this purpose. However, after manually going through some of the test data, it is sometimes fairly difficult even for a human to uncover hate speech, especially for the news dataset. Therefore, our goal was to produce realiable predictions. For both our runs we used the same system playing around with some of its parameters according to the results received from the first run. Therefore, our second run performs slightly better. 5
When evaluating our model with the two test
sets provided by the EVALITA organizers, we
received the scores shown in Table 2. Our model
performed 17% better on test data containing
Tweets
The organizers provided two baseline models (see Table 3 – most frequent class (MFC) and Linear SVM with unigrams, char-grams and TFIDF representation. Our model achieved higher scores for the news headlines and the twitter test set compared to the MFC baseline that achieved Macro-F1 scores of 38.94% and 33.66% respectively. However, our model failed to beat the baseline of the Linear SVM for the news test set which scored 62.1%. Nevertheless, it performed better on the tweets test set compared to the Linear SVM (72.12%).
Tweets
non-hate hate F1 P R F1 P R 0.82 0.70 0.98 0.39 0.25 0.9 0.79 0.75 0.83 0.76 0.81 0.72 As expected, model performance decreases in cross-corpus evaluation, especially in the news headlines test data. We assume that our model learned characteristics of the Twitter data alongside the characteristics of hate speech. Therefore, the model performs worse when applied to domains that entail different linguistic surface structures. The F1 macro-scores in Table 2 show that the scores for the two labels are evenly distributed (79% for non-hate and 76% for hate). Contrary to this, the model tested on the news data is a lot more likely to detect non-hate items (with 82%) whereas its performance on finding hate items only lies at 39%. The confusion matrices for both test sets for the second run can be seen in Table 4 and Table 5.
Positive Negative 314 5 136 45 la Positive u tc Negative
A Table 4: Confusion Matrix of news headlines test set
Positive Negative 534 107 175 447 la Positive u tc Negative A
Table 5: Confusion Matrix of tweets test set 6
Identifying hate speech in Twitter data was obviously easier for our model because it had been trained on similar data. However, the model had more difficulties in making predictions on the news headlines as hints towards hate speech were much more subtle and harder to grasp. This became especially clear when we tried to identify hate speech in the test data ourselves. For the tweets test data, the use of hate speech was more obvious and direct. Another and bigger problem might have been missing context information as we were limited to the headlines, thereby missing the content of the article. Since we had difficulties identifying especially hate speech for the news headlines test data it is only reasonable that our model had similar difficulties and performed worse compared to the tweets test set. Table 6 and 7 show some examples where our system failed to detect hate speech correctly. Table 6 contains examples with upper-cased words which are used to highlight strong ideas and opinions. In this context, the upper-cased language is used to highlight the rage of the user. Therefore, our model should have been made more sensible towards the intentional use of capital letters to classify content containing hate speech more accurately. Nevertheless, none of these examples, including Table 7 were correctly classified as hate speech. text @user A me pare una scelta politica suicida puntare tutto su una battaglia sicuramente perdente in favore dell’immigrazione incontrollata...Meglio cos`ı, spariranno piu` velocemente! Rosarno, le case popolari? Solo agli immigrati Hanno avuto bisogno di governi non eletti, di gente imposta ad un popolo disarmato. Una volta messi li, i VIGLIACCHI hanno dato inizio alla ns fine! Se e quando si scatenera` la rabbia vera, ne faro` parte!!URL I CRISTIANI ATTACCATI DAL MONDO ISLAMICO: IRAQ SIRIA SRI LANKA E ED EUROPA.E LA CHIESA DIVISA TRA DUE PAPI, BENEDETTO AUTOREVOLE RINTUZZA LA RIVOLUZIONE TRASGRESSIVA DEI COSTUMI, FRANCESCO LASCIA FARE.
CRISTIANI PERSEGUITATI MA IL PROBLEMA SONO I MIGRANTI
URL
Our goal was to develop a system for Hate Speech
Detection in Italian Twitter data. After cleaning
the data, we fine-tuned a BERT model with a batch
size of 16 and a learning rate of 5e-5. Overall, our
model reached an F1 score of 77.56% on the
Twitter test data, and 60% on the news data. Ideas for
future work include adding training data that has
been collected from other sources apart from
Twitter, incorporating a lexicon of hate words, such as
Hurtlex