This paper presents an approach to the shared task HaSpeeDe within Evalita 2018. We followed a standard machine learning procedure with training, validation, and testing phases. We considered word embedding as features and deep learning for classification. We tested the effect of merging two datasets in the classification of messages from Facebook and Twitter. We concluded that using data for training and testing from the same social network was a requirement to achieve a good performance. Moreover, adding data from a different social network allowed to improve the results, indicating that more generalized models can be an advantage.
ll manoscritto presenta un approccio per la risoluzione dello shared task HaSpeeDe organizzato all’interno di Evalita 2018. La classificazione e` stata condotta con caratteristiche del testo estratte con word embedding e utilizzando algoritmi di deep learning. Abbiamo voluto sperimentare l’effetto dell’integrazione di messaggi di Facebook e Twitter ha e abbiamo ottenuto due risultati. 1) Addestrare modelli con un dataset integrato migliora le performance di classificazione in datasets provenienti dai singoli social network suggerendo una migliore capacita` di generalizzazione del modello. 2) Tuttavia, utilizzare modelli addestrati su datasets provenienti da un social network per classificare messaggi provenienti da un altro social network comporta un peggioramento delle performance indicando che e` indispensabile includere nel train set messaggi dello stesso social network che si e` interessati a classificare nel test set.
In the last few years, there is a growing attention to
the automatic detection of hate speech in text. This
appears as an answer to the increased spreading of
online abuse in social networks. Several
evaluation initiatives have been presenting different yet
related classification tasks, e.g. TRAC
The goal of this task is to improve the automatic classification of hate speech in Italian. More specifically, there were three sub-tasks, promoting the development of features that would work independently of social network. For the task HaSpeeDe-FB, only the Facebook dataset could be used to train the model and classify Facebook data; for HaSpeeDe-TW, only the Twitter dataset could be used to classify Twitter data; and for the Cross-HaSpeeDe, only the Facebook dataset could be used to classify the Twitter and vice versa.
In our approach, we focused on understanding the effects of merging the two provided datasets. As features, we used word embeddings and deep learning for classification with a simple dense neural network. In this paper, we present the details of our approach, our results and conclusions. 2
Previous research in the field of automatic
detection of hate speech can give us insight into how
to approach this problem. Two surveys
summarize previous research and conclude that the
approaches rely frequently on Machine Learning and
classification
Regarding the automatic classification of
messages, one first step is the gathering of training
data. Several studies published datasets
considering hate speech with different classification
systems
After data collection, one of the most important
steps when using classification is the process of
feature extraction
Different studies proved that deep learning
algorithms outperform previous approaches. This
was the case when using character or
tokenbased n-grams with Recurrent Neural Network
Language Model (RNN)
In this work, we propose an innovative
approach in hate speech detection by merging
different datasets, in the sequence of a previous
experiment
Regarding the specificities of our approach in this contest, the main research question of our work concerns the effects of merging new datasets on the performance of models for hate speech classification. Accordingly with the previous study, we hypothesize that merging datasets will lead to a better performance. Additionally, we want to investigate how models perform when only data from different sources was used in the training.
In the next sections, we present our methodology and approach to this problem. 3 3.1
The data proposed for this task results of
joining a collection of Facebook comments from
2016
As a first step, we load the messages, remove the retweet marker “RT” in case of the tweets, and also the URL links present in the text. 3.3
We follow a methodology of classification with
training, testing and validation. Keeping 30% of
the data for validation allows us to estimate the
results we would achieve in the contest. We use
word embeddings and deep learning as presented
in previous literature
In the procedure of feature extraction, we vectorize the text. We start by tokenizing the 1https://github.com/StopPropagHate/ experiment_evalita_HaSpeeDe data, considering only the top 10,000 words in the dataset. Additionally, we consider only the first 100 words of the tweets. We use the functions text tokenizer, fit text tokenizer, texts to sequences and pad sequences in our extraction.
For the classification we use 10 fold crossvalidation and apply a simple dense neural network. We use binary crossentropy for loss with the rmsprop optimizer, we define the custom metric F1, so that it would be in according to the contest used metric. Regarding the model, we instantiate an empty model and we customize it: First we add an embedding layer where we specify the input length (100, the maximum length of the messages) and give the dimensionality of the input data (dimensional space of 10,000). We add a dropout of 0.25.
We flatten the output and add a dense layer, specified with 256 unit, with “relu” as a parameter. We add a dropout of 0.25.
We add a dense layer with just a single neuron to serve as the output layer. Aiming for a single output, we use a sigmoid activation.
We use keras compile function to compile and fit the model. We use batch size 128 and we tune the number of epochs starting by using 10. We also feed the model with the classes weights, corresponding to the frequencies of the classes in the training set. We average the F1 and loss results of the 10 folds for each epoch. For the epoch number, we kept the maximum number before overfitting to happen (the results only improving in the training set, but not in the test set). We save the final model and apply it to the validation data, with the function keras predict. We conduct a permutation test in order to have a p-value associated to the F1. 4
We conduct three different experiments following the procedure described in Section 3.
Task HaSpeeDe-FB In the HaSpeeDe-FB run1, we train and test with Facebook data. In the HaSpeeDe-FB run2, we mix Facebook with the Twitter provided data and see the effect in predicting hate speech in Facebook.
Task HaSpeeDe-TW We follow a similar procedure, but we switched the roles of Facebook and Twitter data. For theHaSpeeDe-TW run1 only Twitter data is used. In a second run HaSpeeDeTW run2, we mix data for training and use Twitter for testing.
Task Cross-HaSpeeDe This is a proposed outof-domain task. In the Cross-HaSpeeDe-FB, only the Facebook dataset can be used to classify Twitter data. In the Cross-HaSpeeDe-TW, only the Twitter dataset is used to classify Facebook data. 5
We separated the conditions with testing data from Facebook from Twitter and we compared three different conditions: the training data is from the same social network (1), the training data is both from and not from the social network (2), and the training data is not from the social network (3). 5.1
For each of the runs in our experiment we tuned the epoch parameter and we analyzed the average of the 10 folds for each of the 10 epochs (Figure 1).
The decided number of epochs for each run is presented in the Table 1. We concluded that using mixed data for training (Condition 2) has a better performance (F1) than using data only from the social network (Condition 1). Additionally, using data only from other social network (Condition 3) provided poor results. Finally, classifying Facebook data was easier than Twitter data.
C. system epoch 1 HaSpeeDe-FB run1 7 2 HaSpeeDe-FB run2 3 3 Cross-HaSpeeDe-TW 4 1 HaSpeeDe-TW run1 6 2 HaSpeeDe-TW run2 4 3 Cross-HaSpeeDe-FB 6
F1 0.723 0.738 0.284 0.630 0.679 0.434 p-value 0.001 0.001 0.001 0.001 0.001 1 Regarding the contest results (Table 2), similarly to the validation results we verified again that using mixed data for training (Condition 2) is better.
Also in this case we verified that using only data from a different social network provided much worse results (Condition 3). Opposing to the validation results we found here that generally classifying Facebook data was more difficult than Twitter data. 1F0000....3468 ●●● ●●● ●●● ●●●● ●●●● ●●● ●● ●● ●● ●● da●●ta1Ftvra000al...ii446ndiantgion●● ● ●● ●● ● ● ● ● ● ● data 1F00..68 da●tatraining 0.3 ●● ●● ●● ● ● ● ● ● ● ● ●● tvraaliindia00nt..gi43on ●●●● ●●●● ●●●● ●●●● ●●●● ●● ●● ●● ●● ●● ● validation lsso00..21 2 4 epoch 6● ● 8● ● 1●0 lsso000...210 2 4● ● epoch 6● ● 8● ● 1●0 lsso00..21 2 4 epoch 6● ● 8● ● 1●0 (d) HaSpeeDe-TW run1 (e) HaSpeeDe-TW run2 (f) Cross-HaSpeeDe-FB
Regarding the main finding of this experiment, the results show that in this contest adding new data from a different social network brought improved performance. However, in the scope of this work it was not possible to investigate the reasons for this. One possibility may be the increased number of instances in the training when adding new datasets. Also using data from a different social network may bring less overfitting from training with only a dataset. 6
Throughout our approach to this shared task, our goal was to measure the effects of merging new datasets on hate speech classification. Supported by a previous experiment, we expected that adding data would help the classification. Indeed, we verified that merging datasets allowed us to have a small improvement of the results.
Complementary to this result, we tried the same approach following the same method and idea, in the Evalita 2018 AMI task. Merging datasets did not help for misoginy classification. In this case, we found that merging extra misogynistic or hate speech data kept the mysoginy classification with similar performance.
The reason why merging datasets worked in one case and not in the other remains unclear, and requires exploration in future studies. Possible variables interfering are the number of messages used for training and also the number of distinct words in the data.
This work was partially funded by the Google DNI grant Stop PropagHate.