English. This report describes an approach to face a task regarding the identification of hate content and stereotypes within tweets. Two models will be shown, both presented to the HaSpeeDe competition proposed by EVALITA 2020. They are based on a Logistic Regression model that takes different types of embedding as input. The best system shows interesting results.
The use of bad words and bad language has
always been a subject of debate. The spread of
social media platforms, such as Twitter and
Facebook, has fostered the growth of hate speech
online. These sites have been urged to treat and
remove offensive content, but the phenomenon is
so pervasive that the manual way of filtering out
hateful tweets is not enough. For that reason, the
development of automatic recognition systems is
increasingly important. To date, the use of
Natural Language Processing
Copyright © 2020 for this paper by its authors. Use
permitted under Creative Commons License Attribution 4.0
International (CC BY 4.0).
proposed so far are based on manual feature
extraction
The task proposed in the competition
The performances of the participating systems are evaluated on a corpus of Italian tweets as in the previous edition and also on a set of mixed text genres, such as newspapers, comments and headlines.
The dataset used is the one provided by the competition organizers. In particular, the entire dataset is split into one Training Set composed of tweets and two test sets: an in-domain (based on tweets) and a smaller out-of-domain (based on newspaper phrases) test set. Overall, the Training Set includes 6,839 Italian tweets distributed as in Tables 1 and 2.
TR Set
As we can see, the data are not well distributed. Regarding the Hate Speech Training Set, we have that sixty percent of the data are classified as hate speech. The Stereotype Training Set is also a little unbalanced, with fifty-five percent of the data classified as non-stereotype. 4
In this section, the proposed approaches will be described, focusing on what has been developed for the preprocessing of data, the used embeddings and models. Some decisions regarding the choice of models and the extraction of features were made based on the results obtained in other related works. 4.1
A Tweet is a text message with a maximum length of 280 characters. It may contain elements such as hashtags, mentions, links and emoticons. An example of a tweet extracted from the dataset is shown below:
”@user La societa` multirazziale... #migranti #profughi #rom URL” As we can see in the example, the dataset provided has already been preprocessed, censuring names and URLs, probably for privacy. The preprocessing phase that we faced implements a series of functions aimed at modifying a tweet to eliminate useless elements and to standardize it. Punctuation, emoji and any symbols are also eliminated. The tweet is also transformed into a lower case representation as shown: ”la societa` multirazziale migranti profughi rom”
Regarding this phase, the transformation of the single words from an inflected form to root or canonical form was also carried out, respectively, through stemming and lemmatization. We tried to consider these characteristics during the feature selection phase. However, these attempts will not be mentioned further, as they did not produce relevant results. 4.2
The preprocessed tweets were used to generate the feature useful for classification purposes. Both tasks were addressed with the same types of representation and the same models.
• TF-IDF Vector:
These three types of features were used both individually and in combination with each other by concatenation. To decrease the size of these vectors and to speed up the training phase, a features Selection phase is also performed using a Random Forest Classifier.
For both tasks, we tried the use of an SVM Classifier with kernel RBF, a Logistic Regression and a Random Forest. As already mentioned, each of these models has taken various concatenations of the previous feature vectors as input.
We tested each model using 3-fold crossvalidation and performed a grid-search to iterate over the models and all the parameters.
As a result of this search, the best final model
was undoubtedly the Logistic Regression that has
performed well also in previous papers
News TS NoHS HS 0.835 0.615 0.7256
Beyond the macro-F1 values obtained, it is interesting to note the behavior of the model with regard to the out-domain Test Set in both tasks. In particular, the F-scores show worse values in the classification of sentences that actually contain hate speech or stereotyping. This is actually due to low Recall values (about 0.51 for both tasks) which is probably due to the fact that the model is trained on a different type of data. 6
Observing the results on the in-domain Test Set, our best models obtained a ranking of 15/27 and 6/12 respectively for tasks A and B. Regarding the out-domain Test Set, they obtained the third-best score in both tasks. The result obtained with the first Test Set confirms that the proposed approach turned out to be too simplistic. However, it’s interesting to notice how such a simple system achieved a good placement in the out-of-domain test-set. An explanation of that could be the way the Training Set was preprocessed. In fact, each tweet has been transformed into a plain text, without taking into consideration any characteristic of a ’social’ language. This may have positively influenced the model in predicting the out-of-domain classification.
A further observation to be made about the dataset
concerns a lack of correlation between the use of
bad words and the presence of hateful contents
in a phrase. This fact shows how Offensive
Language Detection and Hate Speech Detection
are related topics, but they remain two distinct
tasks
The participation in the Hate Speech Detection 2020 competition proposed by Evalita is derived from purely academic purposes.
We focused on using different types and combinations of embeddings. Surprisingly, the best results were obtained with the use of Tfidf only instead of the use of a combination of more sophisticated embeddings such as GloVe and DistilBert. After a feature selection phase carried out through a Random Forest, the results obtained through a Linear SVM and a Logistic Regression were compared. The latter was the best.
We are aware that the presented system does not introduce new elements with respect to the state of the art of current technologies. Despite this, it was interesting to observe the different results obtained in relation to the composition of the Test Set.
The project was completely developed in python, and the code is publicly available at the following link:
https://github.com/eliabisconti/haspeede