English. In this report we describe the hate speech detection system for the Italian language developed by a joint team of researchers from the two universities of Perugia (University for Foreigners of Perugia and University of Perugia). The experimental results obtained in the HaSpeeDe task of the Evalita 2018 evaluation campaign are analyzed. Finally, a suggestion for future research directions is provided in the conclusion.
Italiano. In questo documento
descriviamo il sistema di hate speech detection
per la lingua Italiana sviluppato da una
squadra di ricercatori dell’Universita` per
Stranieri di Perugia e dell’Universita` degli
Studi di Perugia. I risultati sperimentali
ottenuti nel task HaSpeeDe, organizzato
nell’ambito di Evalita 2018, sono
riportati e analizzati. Infine, una possibile
direzione di ricerca e` fornita nelle
conclusioni.
In the recent years there was an exponential
growth of social media that has revolutionized
communication and content publishing. However,
social media are also increasingly exploited for the
propagation of hate speech. This issue motivates
the recent research on hate speech detection
systems
In this paper, we provide the description of our hate speech detection system for the Italian language. The system, namely HSD4I PG, has been developed by a joint team of researchers from the University for Foreigners of Perugia and the University of Perugia. The code of HSD4I PG is provided online at the url https://github.com/Gabriele91/HSD4I PG.
The rest of the paper is organized as follows. The main system architecture is provided in Section 2, while the single software components are described in Sections 3-6. Experimental results are provided in Section 7, while conclusion and future lines of research are depicted in Section 8. 2
The hate speech detector we have developed,
namely HSD4I PG, is composed by several
software components:
a tokenizer for Italian posts from social
media,
the popular FastText tool
Moreover, the following resources have been
adopted:
the Ita Twitter corpus
As any other supervised classifier system,
HSD4I PG requires a training stage, that is
depicted in Figure 1. The word embedding model
is trained by FastText using the Ita twitter corpus.
Numeric features are obtained by aggregating the
FastText features and by generating some ad-hoc
extra-features. These numeric features are finally
fed to a Support Vector Machine (SVM)
After the SVM classifier has been trained, the prediction of (unlabeled) posts is performed following the scheme depicted in Figure 2. 3
A tokenizer for the Italian language adopted in
social media has been designed by modifying the
output produced by the ”TweetTokenizer” class
of the popular Python library NLTK
A variety of corrections have been introduced.
The most important ones are:
1. two or more consecutive occurrences of the
same vowel have been replaced by a single
occurrence (e.g., ”ciaooo” is replaced with
”ciao”),
2. alternative spellings of some bad words have
been normalized (e.g., ”vaffa” is replaced
with its most popular form),
3. some common mispellings and abbreviations
have been corrected (e.g., ”cmq” is replaced
with ”comunque”),
4. hashtags have been split into multiple tokens
using the Python library
”compound-wordsplitter”,
5. apostrophes have been considered as token
separators,
6. tokens composed by digits characters have
been replaced with the token NUM,
7. tokens corresponding to Twitter mentions
have been replaced with the token MEN,
8. tokens corresponding to web links have been
replaced with the token URL,
9. emojis have been kept as tokens on their own,
while other punctuation characters have been
removed,
10. all the textual tokens have been replaced with
their stemmed form by using the NLTK
implementation of the Snowball stemming
algorithm for the Italian language
Moreover, in order to provide additional experimental results, we have also tried a lighter variant of the tokenizer that only perform the tasks numbered from 5 to 10. 4
A word embedding model is generated by
FastText
Fed with the Ita Twitter corpus, FastText produces a numeric vector representation for every ngram contained in the corpus’ posts in such a way that the n-grams belonging to tokens appearing in similar contexts are close to each other in the continuous numerical space.
After the model has been generated, a numeric representation for a given token w can be simply computed by summing up the numeric representations of the n-grams that compose w.
Since out-of-vocabulary words are quite common in social media texts, we think that the subwords information contained in the n-grams is particularly useful in our scenario.
The word embedding model allows to generate a numeric representation for every token. Therefore, in order to produce a (constant length) numeric representation of the whole post, we need to aggregate the vectors corresponding to the tokens of the post. Six different aggregation functions have been considered: average (avg), standard deviation (std), minimum (min), maximum (max), median (med), and sum (sum). Any combination of these aggregators can be adopted, thus the features generator requires an experimental tuning (see Section 7).
Moreover, 20 additional extra-features have
been introduced:
number of hateful tokens, computed using
the Italian Lexicon of Hate Speech
As an illustrative example, let consider that: FastText has generated numeric vectors of size 300 for every single token w of a post p, and that the combination of the three aggregators sum, min, max has been chosen. Then, the numeric vector representing p has 300 3 + 20 = 920 dimensions and it is formed by concatenating the three vectors, each one of size 300, given by every chosen aggregator together with the 20 extrafeatures.
Finally, in the case the number of features is too large for the classifier, during the training phase we are able to reduce the dimensionality to a given number k by selecting the features having the largest mutual information with respect to the class labels. 6
After some preliminary experiments, we have
decided to adopt a Support Vector Machine (SVM)
classifier
Compared to the popular neural network model, the SVM technique has less parameters to be tuned, it is computationally more efficient, and it generally obtains comparable performances.
Finally, it is important to note that, before the training phase, all the training features have been standardized in such a way that their means and variances, across all the training instance, are, respectively, 0 and 1. 7 7.1
The parameters of the different software components of HSD4I PG have been tuned using a grid search approach and a 10-folds cross-validation scheme.
FastText parameters have been chosen in the following ranges: number of epochs epoch 2 f5; 20; 50; 100g, the initial learning rate lr 2 FastText SVM
f0:05; 0:1g, the negative sampling neg 2 f5; 20; 50g, the window size ws 2 f5; 10g. Moreover, the skipgram model has been considered, while other FastText parameters that have been set to constant values are: dim = 300, minCount = 1, minn = 3, and maxn = 6.
Regarding the features generator (see Section 5), a combination of the six aggregators has to be chosen. Importantly, for combinations resulting in more than 1,000 features, the filtering procedure described at the end of Section 5 is performed.
After some preliminary experiments, we have decided to use the following ranges in order to tune the SVM parameters: kernel 2 frbf; linearg, C 2 f1:8; 2; 2:2; 2:4g. Moreover, the gamma and class weight parameters have been set to, respectively, auto and balanced.
The best parameter setting resulting from the experimental tuning is provided in Table 1. 7.2 Table 2 provides the results obtained by HSD4I PG in the four proposed tasks. In particular, the Macro-Average F1 score for each subtask is shown, along with the difference from the best competitor in the subtask.
SubTask
HSD4I PG HaSpeeDe-FB
HaSpeeDe-TW Cross-HaSpeeDe-FB Cross-HaSpeeDe-TW In this paper we have introduced a system for the hate speech detection of social media texts in Italian language. The results we have obtained for the HaSpeeDe task of the Evalita 2018 campaign are provided.
It is worth to point out that the results of most participants are very similar and quite far from being fully accurate. The question is whether hate annotation is objective or subjective. Few of the posts in the datasets looks to be difficult to annotate even for a human being. Indeed, we think that different people can produce different annotations. Therefore, it can be interesting to model the subjective perception of hatefulness and exploit such information in the detection task, perhaps, taking inspiration by recommender system techniques. Parameter epoch lr ns ws kernel C
Value 50 0.05 50 5 sum min max rbf 2.2
This setting has been used to generate the results submitted as ”run 2” at the Haspeede task of Evalita 2018 by the team ”Perugia1”. For a mistake, we have submitted a wrong file as ”run 1”. Anyway, in the following section we also provide the results of three additional executions of HSD4I PG:
Execution A) It uses the same setting of Table 1 except that C = 2, Execution B) It uses the same setting of Table 1 except that the lighter variant of the tokenizer (see Section 3) has been adopted, Execution C) It uses the same setting of Table 1 except that C = 2 and the lighter variant of the tokenizer (see Section 3) has been adopted.
A B C A B C