We describe our approach to address Task A of the EVALITA 2020 Hate Speech Detection (HaSpeeDe2) challenge. We submitted two runs that are both based on contextual embeddings - which we had chosen due to their effectiveness in solving a wide range of NLP problems. For our baseline run we use stacked embeddings that serve as features in a linear SVM. Our second run is a simple ensemble approach of three SVMs with majority voting. Both approaches outperform the official baselines by a large margin, and the ensemble classifier in particular demonstrates robust performance on different types of test data coming 6th (out of 27 runs) for news headlines and 10th (out of 27) for Twitter feeds.
Hate speech in social media (and its automatic
detection) has become a major problem in recent
years. It can be generically defined as “language
that is used to express hatred towards a targeted
group or is intended to be derogatory, to
humiliate, or to insult the members of the group”
Copyright c 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). ously provide rights to freedom of expression and prevent censorship and illegal discrimination. All this contributes to making automatically detecting hate speech a challenging task.
Nevertheless, social media platforms such as Twitter have defined clear guidelines prohibiting the use of hateful behaviour.1 Accounts with such contents can be reported and are subsequently deleted. The challenge is to be able to detect such content automatically with both high precision and high recall.
The EVALITA evaluation campaign introduced
a hate speech detection challenge applied to
Italian social media in 2018
We will provide some background that should motivate the system architectures we developed. There are several aspects to be mentioned here.
First of all, given the impressive advances in a
broad range of natural language processing tasks
using a transformer-based architecture
1https://help.twitter.com/en/rules-and-policies/hatefulconduct-policy
This leads us to the next design decision. The
winning team in the 2018 HaSpeeDe competition,
ItaliaNLP, submitted as one of their runs a SVM
with three different feature categories, namely raw
and lexical text, morpho-syntactic and lexicon
features, which performed extremely well in
particular when trained and tested on Twitter data
Ensemble methods have repeatedly been shown
to outperform individual classifiers for a variety
of tasks including hate speech detection. For
example, an ensemble of ten simple neural
classifiers proposed by
Finally, given the task definition in which the classifier was to be trained on social media data but then tested on both social media and news headlines we were aiming at an approach that would have a robust performance across domains rather than being tailored specifically to one type of data.
One additional motivation for our work is the intention to develop approaches that can be applied to different languages (we will get back to that point when we outline future directions).
We will now demonstrate how those motivating considerations lead to the system architecture we propose.
We submitted two runs of which the first one can be considered our own baseline approach. We first present both architectures at a conceptual level and will go into the technical details when we discuss the experimental setup in the next section. Our runs are:
Model 1: Stacked embeddings as features of a linear SVM Model 2: Ensemble of several SVMs with different text representations – both contextual embeddings and TF-IDF-based.
Both models can be realised in many different ways. The core idea, as motivated before, is to experiment with transformer-based contextual embeddings but to avoid fine-tuning and instead deploy a traditional, more transparent approach of SVM. The ensemble can consist of a variety of different systems that can be aggregated in many ways. In this paper (and as submitted) we treat each system as equally important and use a simple majority vote.
Stacked embeddings have been shown to be
effective in NLP applications, e.g.
Our own baseline model combines two different document embeddings: transformer document and document pool embeddings which are then fed into a linear SVM to train a classifier. We keep the architecture deliberately simple.
There is a wide range of transformer-based
language models. One of our motivations was to
train a classifier that will generalise beyond a
specific domain but also has the potential to
generalise beyond a specific language. We therefore
opted for XLM-RoBERTa (XLM-R) that has been
shown to outperform alternative multilingual
models such as mBERT in various NLP tasks
Our second system is an ensemble classifier consisting of three SVMs each trained on a different text representation, namely:
Transformer document embeddings using XLM-R
The first two of these are exactly the same as we have seen in Model 1 except that they are not stacked but fed into different classifiers. Again we observe that the general setup is kept simple to avoid overfitting for the specific problem at hand thereby allowing more scope for future experiments. 4
We applied our systems to Task A - Hate Speech Detection (Main Task). 4.1
Training and test data is briefly described here.
Training Data Set: the training data set consists of 6,839 tweets in total, 2,766 of them classified as hate speech. The corpus has three columns: tweet ID, text and the label (0 = no hate speech, 1 = hate speech). Table 1 summarises these numbers.
0 1 Total
4,073 2,766 6,839 Test Data Set: unlike training data which was all Twitter feeds, there were two sets of test data, the first one sampled from Twitter and the second one from news headlines. The Twitter test set has 1,263 entries in total, the news test set 500. The two columns in both sets are the ID and the text of the tweet and news headlines, respectively. The classes 0 and 1 in the Twitter test set include 641 and 622 tweets respectively. In the news headline test set 319 entries have the label 0, 181 the label 1 (see Table 2).
0 1 Total In line with our overall aim of simplicity and generalisibility (rather than tuning) we applied a simple pre-processing pipeline that would apply to both Twitter data as well as news headlines. There are only small variations in the different normalization steps as follows.
For any embedding-based processing the text
was lower-cased and punctuation was removed so
that any input, be it tweet or news headline, would
be represented as a string of unpunctuated tokens.
For the calculation of our (sparse) TF-IDF
representation the text was tokenized and in addition to
that stopwords were removed. After that each
token was vectorized using TF-IDF. Figure 1 shows
an overview of the preprocessing.
All implementation was done in Python. For all
text and document embeddings we used flairNLP2.
Our SVMs were developed using scikit-learn
use stacked embeddings composed of Transformer
Document and Document Pool Embeddings. The
Transformer Document Embeddings are obtained
using XLM-R. Document Pool Embeddings are
calculated using a mean-pooling over all word
embeddings. It consists of forward and backward
embeddings for the Italian language as provided by
flair
Flair allows for the easy combination of embeddings to create stacked embeddings – one for each input text. These vectors (together with the labels) are then used to train the SVM. Using grid-search on the training data the most suitable parameter settings were determined, and Table 3 specifies the settings which were then used in the submitted run.
C kernel degree gamma
1.0 ’linear’ 3 1 2https://github.com/flairNLP/flair 3https://www.nltk.org
Ensemble of SVMs: three different feature representations are used to train one SVM each as illustrated in Table 4. The first two incorporate the same representations as already seen in Figure 2.
SVM2.1 SVM2.2 SVM2.3
Transformer Document Embeddings Document Pool Embeddings
TF-IDF
Again we used grid-search for parameter tuning (see Table 5).
Input is run against each classifier, and through majority voting over these three predictions the final classification category is determined. 5
We first present detailed results and then discuss our findings and insights. We start with our baseline approach and then move on to the classifier ensemble. Macro-F1 is the official metric for this competition. In addition to that we look at Precision, Recall and F1 at category-level and also include confusion matrices for each approach (Model 1 and Model 2) and test set (Twitter data and news headlines). There were 27 runs submitted for each dataset and the official baseline was a linear SVM with TF-IDF of word and char-grams. 5.1
Twitter Data: Training and testing on Twitter data results in a Macro-F1 score of 0.7399 which makes it into position 16 (out of 27). The official task baseline is 0.7212. Details are displayed in Table 6 and Figure 3.
News Headlines: On the news headlines test data we get a Macro-F1 of 0.6684 with official baseline result of 0.6210 (rank 12). More details are in Table 7 and Figure 4. Twitter Data: Our ensemble approach gets a Macro-F1 of 0.7599 (rank 10). More details are included in Table 8 and Figure 5.
News Headlines: On the news headlines test data we get a Macro-F1 of 0.6984 with an official baseline result of 0.6210 (rank 6). More details can be found in Table 9 and Figure 6.
Precision
Recall
F1 Our first observation we derive from the results is that the ensemble approach we proposed for this task does provide a robust and solid performance – solid in that it scores well in the ranked list of systems and robust in that it also ranks highly when applied to out-of-domain data (coming 6th out of 27 submitted runs on data it had not been trained on). Given the simplicity of our system architecture and the composition of the official baseline system we also note the superiority of transformerbased contextual embeddings over bag-of-words approaches (while this comes as no surprise it is still worth pointing out). Moving from a featurebased to a pre-training plus fine-tuning approach will most certainly further push up the scores.
Looking at the balance between precision and recall, we find that both our approaches have a tendency to return a fair number of false positives for the Twitter data set. This could indicate that words and phrases used to express hateful content is quite common in social media even if it does not actually represent hate speech. On the other hand, we record a large proportion of false negatives when classifying news headlines. This could be an indicator of a more subtle way in which hate speech is expressed in traditional news outlets.
Generally speaking, both models perform better on Twitter data than on news headlines – again an insight that was to be expected due to the training data. However, the fact that our approach managed to score higher in the ranked list of systems for data it was not trained on is a result that confirms our initial assumptions – that using a corpus with a very broad range of topics, styles and languages as our core language model would help in making the system transfer more easily to unseen input.
This leads us to an area of future research.
While it would be possible to improve the
performance of our system by making the preprocessing,
the language model and any fine-tuning step match
more closely the expected test data – e.g. by using
AlBERTo, a BERT-based transformer trained on
Italian Twitter data
We presented a simple but effective architecture to detect hate speech in Italian social media and news headlines. Our ensemble-based architecture relies on contextual embeddings trained on a large multilingual corpus which we see as the basis for the robustness of the approach. There is plenty of room for further improvement and the results we report here will serve as a benchmark in this development.
This work was supported by the project COURAGE: A Social Media Companion Safeguarding and Educating Students funded by the Volkswagen Foundation, grant number 95564.