The present paper describes two neural network systems used for Hate Speech Detection tasks that make use not only of the pre-processed text but also of its Partof-Speech (PoS) tag. The first system uses a Transformer Encoder block, a relatively novel neural network architecture that arises as a substitute for recurrent neural networks. The second system uses a Depth-wise Separable Convolutional Neural Network, a new type of CNN that has become known in the field of image processing thanks to its computational efficiency. These systems have been used for the participation to the HaSpeeDe 2 task of the EVALITA 2020 workshop with CHILab as the team name, where our best system, the one that uses Transformer, ranked first in two out of four tasks and ranked third in the other two tasks. The systems have also been tested on English, Spanish and German languages.
Hate speech is not unfortunately a new problem in
the society, but recently it has found fertile ground
in social media platforms that enable users to
express themselves freely and often anonymously.
While the ability to freely express oneself is a
human right, inducing and spreading hate towards
another group is an abuse of this liberty
As such, many online micro-blogs such as Facebook, YouTube, Reddit, and Twitter consider hate speech harmful, and have both policies and instruments to remove hate speech content, that are get
Copyright © 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). ting better over time. Due to the societal concern and how widespread hate speech is becoming on the Internet, there is strong motivation to study automatic detection of hate speech. By doing so, the spread of hateful content can be reduced, having a safer place to stay online for the community but also a more attractive place for advertising sponsors who do not want their brand to be associated with hateful content. Obviously, detecting hate speech is a challenging task. For example, in case of wrong classification, a content creator could suffer socio-economic consequences such as the demonetization of one of its contents or the ban from the platform used. Therefore, the goal of hate speech detection is not only to identify a text that contains words that at first sight could be negative, but also to be able to distinguish news headlines that talk about crime news from a text that contains an effective “attack” against a person or group on the basis of attributes such as race, religion, ethnic origin, national origin, sex, disability, sexual orientation, or gender identity.
The rest of the paper is arranged as follows.
Section 2 reports a description of our systems
developed for hate speech detection tasks. Section
3 shows the results obtained in the HaSpeeDe 2
In this section we present the implementation details of all the used architectures. Both the systems we implemented share the use of PoS Tagging technique that is applied to the pre-processed text, and passed as an additional input to the neural network. 2.1 Before training a model, it is common practice to clean the data, especially if they are retrieved from social media. For this reason we implemented a classic text pre-processing pipeline, that consists of: lower casing the text; removing HTML tags, mention and symbols; standardizing words by cutting the characters repeated more than two times in a row. We also made some keyword substitutions in all our data sets: • URLs and the “url” keyword of the HaSpeeDe 2 data set were replaced by the symbol LINKURL • Happy emoticons like “ :) ” or “ :D ” were replaced by the symbol HAPPYEMO • Angry or sad emoticons like “ :@ ” or “ :( ” were replaced by the symbol BADEMO It is important to note that we have not removed the emojis from the text as our word embedding takes into account emojis as plain words. 2.2
In this work we use the PoS Tagging technique to
provide our networks with more information about
the meaning of a sentence through an explicit
classification on the basis of its grammatical structure.
This is a crucial point with regards to hate
sentences. In fact they tend to have particular
structures. As an example, one of the most widespread
hate sentence is the verbless one, also known as
nominal utterance
As the HaSpeeDe 2 organizers required
explicitly to use the same system for both tasks A and
B, we set up a PoS Tagging model not too
biased towards either news headlines or tweets. As
a consequence, we enriched the PoS Tagger
provided by the Python’s spaCy library
@user useless people like
all Muslims
@user no more refugees in Italy
please no more
Four bicycles stolen from Milan-Sanremo
cyclists: found in a gypsy camp url
TRAGEDY IN PRISON - The nomad
Carlo Helt takes his own life url
1
0
0
URLs to the vocabulary. In this way we have
injected some parts of the speech of the social
media language into a standard PoS Tagging model.
We were definitely aware that tweet oriented
models such as UDPipe tool
For our application we chose fastText
For the French, English and German tweets we
used pre-trained models
leads to a significantly shorter training time than recurrent solutions. Attention is a means of selectively weighting different elements in input data, so that they will have an adjusted impact on the hidden states of downstream layers.
A Transformer was conceived as an encoderdecoder model, that is an ideal approach for machine translation tasks and language modeling. In this work we used the Transformer encoder architecture, as an alternative to recurrent or convolutional neural networks (CNN) (see Figure 2). We used just one Transformer encoder for the text input and one for the PoS input, then we averaged them thorough max pooling. Finally, we used dropout and a dense layer to get the output probabilities. After testing various combinations of parameters, we found that the most efficient for this task are: 12 heads in Multi-Head attention layer, 768 hidden units, embedding size equal to 300, dropout = 0:2 and batch size equal to 128. Training lasted 3 epochs, about 40 seconds each. 2.5
Transformers
Depth-wise Separable Convolution (DSC) is a well known technique in Computer Vision to lower dramatically the number of parameters in CNN. DSC consists in decomposing classical 3D convolution, performing at first a depth-wise spatial convolution for each channel, followed by a point-wise convolution which mixes together the resulting output channels. This computational trick achieves in mimicking the true convolution kernel operation, while reducing the size of the model, and speeding up the training with almost the same accuracy.
Our neural network architecture is reported in
Figure 3, and takes inspiration from Yoon Kim’s
well-known architecture
In this Section we describe the HaSpeeDe 2 tasks of the EVALITA 2020 competition, and we present our results obtained in each of them. To evaluate the degree of generality of our approach, we also tested it on hate speech detection tasks for languages other than Italian, that is English, Spanish and German. The official ranking reported for each run is given in terms of macro-average Fscore. 3.1
This is the main task, and it consists of a binary classification aimed at determining whether the message contains Hate Speech or not. We finetuned the parameters for this task and then we used the model as it is for the other tasks. We were provided with a labeled training set – made of tweets only – and two unlabeled test sets: one containing in-domain data, i.e. tweets, and the other out-ofdomain data, i.e. news headlines. Our results for both Task A test sets are reported in Table 2.
news news tweets tweets
Transformer 1/27
DSC 4/27
Transformer 3/27
DSC 5/27
Task B is a binary classification aimed at
determining whether the message contains stereotypes or
not. The task is motivated by the fact that
stereotypes constitute a common source of error in HS
identification
news news tweets tweets
Transformer 1/12 DSC 2/12 Transformer 3/12 DSC 5/12 We tested our systems also against data sets coming from either Hate Speech or Offensive Language detection tasks for other languages.
As it can be seen in the results, the Transformer model has always outperformed the DSC model: we expected this outcome due to the nature of the DSC model, designed to be as light as possible but still performing. Regarding the results obtained with the Italian language, we are satisfied with our implementations which have achieved excellent ranking positions in all tasks. In particular, the Transformer model outperformed all the systems that participated to the tasks ranking first with outof-domain data. This can be seen as a great ability of our model to generalize starting from a training data set different from that of the application. Regarding the results obtained with in-domain data we performed slightly worse, ranking third. This is probably due to the PoS Tagging model that we used in fact it is a model trained on Wikipedia and not on social language, even if slightly modified to manage hashtags, emoticons and URLs, it certainly does not perform well on social texts as if it were a purely PoS Tagging model trained on social media language.
As regards the results obtained with the other languages, we can see that with the Spanish language we get an excellent result, surpassing the first official ranked of the HatEval 2019 competition in Spanish. Our models do not achieve as good results as that of English and German even if the Transformer’s score is always above the median value. We think that this is caused by the nature of languages, because Germanic languages, such as English and German, probably benefit less than Latin ones from the additional use of the PoS Tagging, in the way we used it. We are still investigating how to get added value from PoS Tagging for the English and German languages. 5
In this paper we have introduced two systems for the hate speech detection of social media texts in Italian, Spanish, English and German language. The main feature of these models is to use as input to the neural network not only the pre-processed text, but also it’s PoS Tag. We are satisfied with the results obtained, because the systems implemented are light and performing. Furthermore we have shown that the use of models that include the additional use of the PoS Tagging, to give it more meaning, has given an added value, reached the top positions in the tasks ranking. Our future work will focus on injecting more and more the grammatical structure of a sentence into a model, in fact we are planning a language model that does not only have the purpose of predicting a word based on the given context but that it is also capable of predicting the PoS Tag of that word.