With the rise of social media over the last decade, the amount of content that is published every day on the internet has become huge. Unfortunately, as the amount of published content grows, the amount of hate speech that can be found on social media also grows. This fact motivates the creation of systems that could automatically detect this undesired behaviors in order to report them to the competent authorities. With this purpose, we have developed a system that detect users which could be considered as hate spreaders employing a TF-IDF vectorizer in combination with an SVM, achieving an accuracy of 81% over the Spanish dataset and 69% over the English dataset.
Hate speech is commonly defined as a propaganda of ideas based on the superiority of a group of people because of their race, color or ethnic origin. This problem is not novel, as it has been present in our society during centuries, but due to the rise of social media it has reached unprecedented levels. Given the huge amount of content generated by the users and the impossibility to manually check all the content, the automatic detection of hate speech has become a relevant task.
With this purpose, the aim of this competition is to automatically detect hate speech, but employing an author profiling perspective instead of a tweet perspective. Therefore, we are interested in detecting which users could be considered as hate spreaders.
This paper presents our participation in the Author Profiling task at PAN [
The dataset of this competition was composed by 200 authors for each language, where each author was composed by 200 tweets. From the 200 authors, 100 were hate spreaders and the other 100 were not. Moreover, we would like to remark that all the urls, links, hashtags and user mentions on the corpus were masked by a unique token for each type.
The objective of this step is to reduce the vocabulary size by merging different token occurrences that are referring to the same concept. For the preprocessing of the dataset we have followed these steps.
In first place, as we are interested in detecting hate speech at author level, we have concatenated all the tweets of each author into one single string. Then, we have converted the text to lowercase and we have replaced all the emojis and emoticons by the token “emoji” employing a regular expression. After that, we have applied a different linguistic preprocessing for each language: in the English dataset, we have replaced some contractions by their expanded form, (for example, the token: “you’d” has been replaced by “you would”, the token “it’s” has been replaced by “it is”, etc), meanwhile in the Spanish dataset we have replaced some words by their homologous colloquial tokens (for instance, the token “por” has been replaced by “x” and the token “que” has been replaced by “k”). Then, we have removed all the punctuation signs such as points, commas, exclamation signs, etc. Moreover, we have reduced different forms of expressing laugh such as “hahahah”, “ahahha”,“jajajaja”, “lol”,“lmao” to the token “haha”. Finally, we have removed the stopwords and performed stemming on both datasets.
In this task we have considered the feature extraction process and the machine learning model estimation as a combined optimization process. Therefore, we have performed an extensive grid search to choose the best combination of hyper-parameters for the tfidf vectorizer and the different machine learning models employed. To assess the performance of our classifier, we have performed a 10-fold cross validation over the training dataset.
For feature extraction, we have employed a TF-IDF [
Before starting to discuss the selected classifiers, we would like to remark that we have decided
to avoid employing deep learning for this task. This is due to the fact that we only have 200
samples for each author, and given that deep learning is usually data-hungry, it could easily turn
into overfit. Therefore, among all the possible machine learning models, we have chosen the
following: logistic regression (LR) [
Finally, we would like to remark that we have used scikit-learn [
If we take a look at Tab. 2, it can be seen that linear models such as SVM, logistic regression and SGD classifier have performed particularly well at this task. This is due to the fact that the number of features is much larger than the number of samples, making it feasible to separate the two classes linearly. From these models, the best performing one has been the SVM, achieving an 80.90% of accuracy for the Spanish dataset and a 70.50% of accuracy for the English dataset.
LR NB RF KNN SGD SVM
EN ES EN ES EN ES EN ES EN ES EN ES
C: 100
C: 100 alpha: 0.25, prior: False alpha: 0.25, prior: True criterion:“gini”, depth: 4, min_samples: 10 criterion:“gini”, depth: 4, min_samples: 8 weights:“distance”, metric: “euclidean”,
neighbors: 5, leaf_size=20 weights:“distance”, metric: “euclidean”, neighbors: 10, leaf_size=20 alpha:0.01, loss:“perceptron” alpha:0.001, loss:“hinge”
Tf-idfVect Hyper-params
word, unigram and bigrams, min_df: 11
word, unigram and bigrams, min_df: 8
word, unigram and bigrams, min_df: 8
word, unigram and bigrams, min_df: 8
word, bigrams, min_df: 9
word, unigram and bigrams, min_df: 9
word unigrams, min_df: 4
word unigrams, min_df: 10
Acc. (%) 68.50 80.50 65.55 79.00 67.00 80.50 65.00 77.00 70.00 80.00 70.50 80.90 Again, this is motivated by the fact that the SVM chooses the separating hyperplane with the maximal margin from among all the possible separating hyperplanes. Other techniques such as Random Forest, Naïve Bayes classifier and K-NN also performed well, but they did not achieve the results obtained by the linear models.
As final model, we have chosen an SVM for both datasets, given that it is the classifier which has achieved the highest estimated accuracy in both languages. Finally, we have trained an SVM for each language, employing the full train dataset and the hyper-parameters described in Tab. 2. The results obtained in the competition with these models are the following:
It can be seen that our estimation of the performance of the system has worked reasonably
well, matching the test accuracy very similar levels to the ones predicted during training. Test
accuracy results have been provided by the TIRA evaluation platform [
To sum up, we have described the methods employed for this task. We have detailed how our whole system works, from the preprocessing step to the estimation of the best hyper-parameters for the feature extractor and the machine learning models. We have also seen that our estimation of the accuracy employing a 10 fold CV is consistent with the test results. As future works, we would like to test an ensemble model of different classifiers to see if it can beat the performance achieved by our SVM.