In this paper we summarize our participation in the CLEF conference 2021 regarding the Profiling Hate Speech Spreaders on Twitter task. We suggested a Support Vector Machine classifier that uses as features word n-grams. Our final software achieved an accuracy of 72% in English, 82% in Spanish and therefore, an average accuracy of 77%. The evolution that social media have experienced, becoming an essential factor in the communication of today's society [1], has led to new data sources. That is why a lot of organizations use this data as a tool to analyze the feedback about some of their events, members, or products. However, organizations need to be able to discern between which opinions are written by users whose based solely on hating, and which are not to be able to do an objective analysis. The goal of Profiling Hate Speech Spreaders on Twitter task is to identify possible hate speech spreaders as a first step towards preventing hate speech from being propagated among online users. Thus, in order to distinguish between authors, we use character and word n-grams as a feature with a Support Vector Machine (SVM) classifier and we prove different preprocessing strategies to provide a prediction for each user. In Section 2 we expose various related works on this task. In section 3 we present our method and different models and preprocessing strategies that we have tested. In Section 4 we show our results and finally, in Section 5, we expose the conclusions we have reached.
Most social networks have imposed rules on users that prohibit hate speech. However, controlling that these standards are met requires a large amount of manual work to review user reports. Due to this fact many of these platforms have increased the number of people in charge of controlling the generated content. Therefore, developing systems that are capable of detecting hateful users streamlines the review process by helping moderators to dismiss false reports. In order to develop automated hate speech detection systems, it should be noted that there are different approaches to this task.
On the one hand, there are the approaches based on combining some traditional machine learning
model, such as Naive Bayes, SVM, Random Forest among others, with the extraction of features using
character and word n-grams calculated from the Term Frequency - Inverse Document Frequency
(TFIDF) [
This section presents the dataset and the models utilized in the experimentation. For this we have
used python and toolkits of emoji2, keras [
The corpus of this task is composed of two sub corpuses, one of them with tweets in English and the other with tweets in Spanish. In addition, each sub corpus contains 200 XML files, which correspond to the authors, and each file contains 200 tweets from an author. It should be noted that tweets have been pre-cleaned, and hashtags, URLs and user mention in tweets have been converted to regulated tags.
Firstly, we have grouped in a single chain all the tweets belong to the author, so there are 200 samples per corpus and then some preprocessing strategy applies to them.
Consequently, we have based on the preprocessing method of Pizarro [
In our experimentation we have developed different types of machine learning models such as support vector machines (SVM), random forest (RF), XGBoost classifiers (XGB) and a neuronal model based on pre-trained BERT transformer.
Relative to SVM, RF and XGB models, it should be noted that these models are being trained using features of character and word n-grams calculated from the TF-IDF of each author. In addition, we have run a grid search to find the best preprocessing and vectorization strategy and combination of hyperparameters for the models.
For the finetuning of the SVM model, we have applied different types of kernel and various values of hyperparameter C.
Regarding the random forest (RF) model, we have experimented with the quantity of trees in the forest, the criteria for measuring the quality of partitions and the minimum number of samples required to partition an internal node.
Relative to XGBoost classifier, we have tested with the number of estimators, the learning rate, and the maximum depth of a tree.
On the other hand, regarding the pre-trained BERT [
In the training phase, we have used a 10-fold cross validation strategy to finetune the parameters of models and to select the best of them. Therefore, in the table 7 is shown estimated accuracies of each model.
In this paper, we summarized the submitted models through the TIRA platform [
Firstly, it is worth noting the great influence that cleaning and tokenizing data has on the operation of classic classification models, we observed that for each language we have to tune specifically the preprocessing strategy used.
Relative to obtained results in the training phase with 10-fold cross validation strategy, we contemplate that SVM model gives the best accuracy in both languages. In addition, we see neural model BERT provides the worst performance probably due to the small quantity of data.
Finally, comparing the results obtained in the training phase with the estimation made in the training phase, we observed that with Spanish tweets we have made a good estimate of the accuracy whereas with English we find a small difference.