English. This paper reports on the systems the InriaFBK Team submitted to the EVALITA 2018 - Shared Task on Hate Speech Detection in Italian Twitter and Facebook posts (HaSpeeDe). Our submissions were based on three separate classes of models: a model using a recurrent layer, an ngram-based neural network and a LinearSVC. For the Facebook task and the two cross-domain tasks we used the recurrent model and obtained promising results, especially in the cross-domain setting. For Twitter, we used an ngram-based neural network and the LinearSVC-based model.
Italiano. Questo articolo descrive i modelli del team InriaFBK per lo Shared Task on Hate Speech Detection in Italian Twitter and Facebook posts (HaSpeeDe) di EVALITA 2018. Tre classi di modelli differenti sono state utilizzate: un modello che usa un livello ricorrente, una rete neurale basata su ngrammi e un modello basato su LinearSVC. Per Facebook e i due task cross-domain, si e` scelto un modello ricorrente che ha ottenuto buoni risultati, specialmente per quanto riguarda i task cross-domain. Per Twitter, sono stati utilizzati la rete neurale basata su ngrammi e il modello basato su LinearSVC.
In this paper, we describe the submitted systems
for each of the four subtasks organized within
the HaSpeeDe evaluation exercise at EVALITA
2018
We adopt a supervised approach and, to select the
best model for each task, we perform grid search
over different machine learning classifiers such
as Neural Networks (NN), Support Vector
Machines (SVM) and Logistic Regression (LR). Both
ngram-based (unigram and bigram) and recurrent
models using embeddings were tested, but only
the ones that were submitted for the tasks will be
described. A LinearSVC model from scikit-learn
Since misspellings, neologisms, acronyms and jargon are common in social media interactions, it was necessary to carefully preprocess the data, in 1https://gitlab.com/ashmikuz/ creep-cyberbullying-classifier
2https://github.com/0707pinar/ Hate-Speech-Detection/
3To split the data we use the scikit-learn train test split function, using 42 as seed value. order to normalize it without losing information. For this reason, we first replace URLs with the word “url” and “@” user mentions with “username” by using regular expressions.
Since hashtags often provide important
semantic content, they are normalized by splitting them
into the words composing them. To this end,
we adapted to Italian the Ekphrasis tool
We used the following text-derived features:
Word Embeddings: Italian fastText
embeddings
ber of hashtags and mentions, the number of exclamation and question marks, the number of emojis, the number of words that are written in uppercase.
level emotion and sentiment tags for Italian
words extracted from the EmoLex
In order to classify hate speech in social media interactions, we believe that recurrent neural networks are a useful tool, given their ability to remember the sequence of inputs while considering their order, differently from the feed-forward models. In the context of our classifier, this allows the model to remember the whole sequence of words in the order they appear in.
More specifically, our recurrent models,
implemented using Keras
Batch normalization and various kinds of
dropout have been tested to reduce the variance of
the models. Experimental results suggested that
applying the former to the output of the
recurrent layer had a negative effect on performance.
For this reason, batch normalization was applied
only to the output of the hidden layers. As for
dropout, we tried three different mechanisms. A
simple dropout layer
Ngram-based neural networks are structurally
similar to the recurrent models. We first
compute the unigrams and bigrams over the
lemmatized social media posts. The resulting vector is
then normalized by using tf-idf from scikit-learn
and concatenated to the social-specific features.
One or two hidden feed-forward layers are then
used, and the same output layer as in the recurrent
models is used. The same dropout and batch
normalization techniques used in the recurrent models
have been tested for the ngram-based neural
networks as well. For the first submitted run of Task
2: HaSpeeDe-TW, we used unigrams and bigrams
along with the required preprocessing steps based
on tf-idf model.
We implemented a Linear Support Vector
Classification system (i.e., LinearSVC)
In this Section we describe the single runs submitted for each task and we present the results. The official ranking reported for each run is given in terms of macro-average F-score. 3.1
For Task 1: HaSpeeDe-FB, two recurrent models were used. The first submitted run used a single fully connected layer of size 200 and a GRU of size 100 as the recurrent layer. Recurrent dropout was applied to the GRU with value 0.2. The second submitted run used two fully connected layers of size 500 and a GRU of size 300 as the recurrent layer. Simple dropout was applied to the output of the feed-forward layers with value 0.5. The first run ranked third and the second ranked fourth out of 18 submissions (Table 1). As shown in Table 1, both runs yield a better performance on the hate speech class.
Category Non Hate
Hate Macro AVG Non Hate
Hate Macro AVG
First Run
P R 0.763 0.687 0.858 0.898 0.810 0.793
Second Run 0.716 0.703 0.859 0.867 0.788 0.785
F1 0.723 0.877 0.800 0.709 0.863 0.786 scribed in subsection 2.5. This run was ranked sixth out of 19 submissions. As our second run on the Task 2: HaSpeeDe-TW, an ngram-based neural network was used having a single fully connected hidden layer with size 200. Simple dropout was applied to the hidden layer with value 0.5. This run ranked fourth. Both runs show better performance when classifying the non hate speech class as displayed in Table 2. For Task 3.1: Cross-HaSpeeDe FB two recurrent models were used. In the first submitted run, two hidden layers of size 500 were used. An LSTM of size 200 was adopted as the recurrent layer. Embeddings dropout was applied with value 0.5 and a simple dropout was applied to the output of the feed-forward layers with value 0.5. The recurrent model for the second run had one hidden layer of size 500. A GRU of size 200 was used as the recurrent layer and no dropout was applied. The first run ranked second out of 17 submissions while the second run registered the best score in the Task 3.1: Cross-HaSpeeDe FB. In both runs, the models showed good performance over the non hate speech class, whereas the precision on the hate speech class does not exceed 0.5 (see Table 3).
Category Non Hate
Hate Macro AVG Non Hate
Hate Macro AVG
First Run
P R 0.810 0.675 0.497 0.670 0.653 0.672
Second Run 0.818 0.660 0.494 0.694 0.656 0.677 In the first submitted run for Task 2: HaSpeeDeTW, we used the LinearSVC-based model deFor Task 3.2: Cross-HaSpeeDe TW two recurrent models were used. In the first submitted run, two hidden layers of size 500 were used together with a GRU of size 200 as the recurrent layer. Simple dropout was applied to the output of the feedforward layers with value 0.2, whereas the recurrent dropout has value 0.2. In the second submitted run, one hidden layer of size 200 was used adopting an LSTM of size 200 as the recurrent layer. Embeddings dropout was applied with value 0.5. The first run ranked fourth out of 17 submissions, while the other run ranked second. Table 4 shows that in both cases the system showed good performance over the hate speech class, while detecting negative instances proved difficult, in particular in terms of precision over the non hate speech class.
Category Non Hate
Hate Macro AVG Non Hate
Hate Macro AVG Although all our runs obtained satisfactory results in each task, there is still room for improvement. In particular, we noticed that our models have problems in classifying social media messages containing the following specific phenomena: (i) dialects (e.g. “un se ponno sent`ı...ma come se fa`...”) or bad orthography (e.g. “Io no nesdune delle due.....momti pesanti”); (ii) sarcasm, “Dopo i campi rom via pure i centri sociali. L’unico problema sara` distinguere gli uni dagli altri”; (iii) references to world knowledge, typically used for an indirect attack not containing an explicit insult (e.g. “un certo Adolf sarebbe utile ancora oggi con certi soggetti”); (iv) metaphorical expressions, usually referring to ways to physically eliminate the targets of hate speech messages (e.g. “Ruspali”).
As for false positives, some errors come from the misclassification of messages containing the lemmas “terrorista”, “terrorismo”, “immigrato” that are extremely frequent in particular in the Twitter dataset. These lemmas are associated to the hate speech class even when they appear in messages reporting the title of a news, eg. “Il Giappone senza immigrati a corto di forza lavoro”.
In Task 2: HaSpeeDe-TW, when the classifier relies on sentiment and emotion features, we registered several misclassified instances containing relevant content words not covered by EmoLex. This is due to the fact that for every English word, EmoLex provides only one translated entry, thus limiting the overall coverage. For instance, “to kill” is translated in Italian with “uccidere” not considering synonyms such as “ammazzare” often used in the dataset.
Finally, we noticed some inconsistencies in the gold standard. For example, the message “Al solo vederle danno il voltastomaco!” is annotated as hate speech while, the almost equivalent, “Appena le ho viste ho vomitato” is considered a non hate speech instance while our models identify it as hate speech. Similarly, an insult like “ridicoli” is annotated as non hate speech in “CERTO CHE GLI ONOREVOLI DEL PD SI RICONOSCONO A KILOMETRI ... RIDICOLI” but as hate speech in “Ci vorrebbe anche qua Putin, invece di quei RIDICOLI...PAROLACCE PAROLACCE”. 5
In this paper we presented an overview of the runs submitted for the four subtasks of HaSpeeDe evaluation exercise. We implemented a number of different models, comparing recurrent neural networks, ngram-based neural networks and linear SVC. While RNNs perform better in three of four tasks, classification on Twitter data achieves a better ranking using the ngram based neural network. Our system was ranked first among all the teams in one of the cross-domain task, i.e. Cross-HaSpeeDe FB. This is probably due to the fact that considering the whole sequence of inputs with a recurrent neural networks and using a prelearned representation by using word embeddings help the model to learn some common traits of hate speech across different social media.
Part of this work was funded by the CREEP project (http://creep-project.eu/), a Digital Wellbeing Activity supported by EIT Digital in 2018. This research was also supported by the HATEMETER project (http:// hatemeter.eu/) within the EU Rights, Equality and Citizenship Programme 2014-2020.