Nowadays, social media is perhaps one of the most powerful channels of communication among people worldwide. Despite the physical constraints, people in a social network communicate efcfiiently and instantaneously without restriction. While this can promote the interchange of ideas and information, in this scenario, people with a dangerous idiosyncrasy can achieve more people with low restrictions. Automatic hate speech identification in social networks is a Natural Language Processing task dedicated to pointing out users that have this kind of misconduct among its publication's content. In this work, we tackled the PAN21 Hate Speech identification task through Semantic Emotion-based models in both Spanish and English languages. We implement several approaches, one of them is designed to output explainable results based on the user's emotional charge.
Social media is perhaps one of the most powerful channels of communication among people worldwide. Despite the physical constraints, people in social networks such as Twitter interchange efficiently and instantaneously ideas and information without restriction. In this scenario, hate speech emerges as a problem when communication denigrates a person or a group based on some characteristics such as race, color, gender, or sexual orientation.
Automatic identification of hate speech has been popular because of the nature of social
networks, and it has been tackled on several fronts. On the one hand, several competitions have
been run contests at the message level. For example, the HatEval [
Using lexicons of labeled words is a fundamental tool in many text classification approaches. The main idea behind this method is to use curated information that associates words with some helpful information to solve the task.
Despite that dictionary-based classifiers are among the first approaches in the community, its
simplicity and the inherent ability of explanation, these methods remain popular even today. For
instance, Nielsen [
However, one of the main drawbacks of lexicons is that they need to be created by experts and have an inherent dependence on language and domain. Another critical issue is the lack of exhaustiveness due to the explosion of terms (e.g., neologisms, inflections, synonymy, hyponymy, hypernymy). Finally, depending on the domain, the lexical variations and errors also negatively affect the performance of the models.
Our work is based on the EmoLex dictionary along with semantic representations of the vocabulary to cope with many of the issues regarding plain lexicons.
Several of these limiting issues can be solved using semantic word-embeddings, which are
semantic lexicons associating words with a vector in a semantic space. Semantic spaces are
1https://github.com/fnielsen/afinn
2https://www.cs.uic.edu/~liub/FBS/sentiment-analysis.html
3https://www.saifmohammad.com/WebPages/NRC-Emotion-Lexicon.htm
learned from a huge non-annotated text corpus, based on the distributional hypothesis of semantics,
i.e., words with similar meanings will tend to appear in similar contexts. Some examples of word
embeddings are fastText [
Semantic language models are sophisticated methods dedicated to understanding language, and
they are created to predict the semantic of a sequence of words through very large non-annotated
corpora. Our approach centers on word meanings instead of sentence meanings; therefore,
language models are beyond the scope of this contribution. The interested reader is referred to
the related literature [
FastText4 is a word-embedding with a fast construction. It learns word distributional semantic using small windows around words. In addition to other approaches, it is designed to tackle out-of-vocabulary words using subwords which are small substrings that compose words and are to compute word-vectors whenever a word is unknown. Using fastText models, we cope with the vocabulary diversity found in social networks.
A popular way to tackle author profiling problems is supervised learning; in this approach, a set of labeled examples is given to an algorithm to create a model that can label never-seen examples. The PAN @ CLEF21 asks for profiling of hate speech spreaders using user’s textual information retrieved from Twitter. Therefore, our dataset examples are a list of text messages and their associated label.
Classical supervised learning will receive a set of examples (, ) of the form = 1, 2, · · · ,
and = 1, 2, · · · , , where is a vector of some dimension , i.e., ∈ R . On the other
hand, is a categorical value that represents a valid class. For this matter, profiling methods
based on supervised learning need to transform input messages into a real-valued vector, is often
obtained by human annotators that assign a label to each example. From a general perspective,
the vectorization can be based on how authors write, the actual content, or the meaning of their
messages. More detailed, we can capture how authors write using stylometry features [
Once the dataset is in the (, ) form, we need to learn from these examples to obtain a
predicting model. There are different types of machine learning models used for the Author
Profiling task [
This notebook tackles the problem of profiling Hate Speech Spreaders, in Spanish and English languages, through its published messages in social media. In particular, we focus on the homonymous task in PAN @ CLEF21 for benchmarking our model. Each dataset, English, and Spanish, contains 200 cases for each of the two languages, each dataset contains 200 different authors, and each author is represented by two hundred messages. Furthermore, each author is labeled as a hate speech spreader or not.
Our approach is designed to be explainable through the projection of users into an Emotionspace induced by EmoLex. It can support variations of the same concepts and lexical variations of the same word based on semantic representations with the help of fastText. Multilingual support for Spanish and English languages is straightforward since translations of EmoLex, and existing pre-trained models of fastText for Spanish and English languages. We test our emotion-based encoding with seven different classifiers and provide a brief statistical analysis in the experimental results section. Section 2 details our modeling approach.
This section contextualizes our participation in PAN @ CLEF21. Section 2 details the construction and prediction stages of our model. The experimental setup and results are described in sections 3 and 4. Our final comments and conclusions are given in Section 5. 2. Emotion-based modeling of users Our model is pretty general and straightforward. However, its construction needs a set of Emotion-prototypes based on the emotion lexicon and word embeddings. We use the EmoLex and pre-trained FastText embeddings for both Spanish and English languages. Figure 1 illustrates the general flow of the prototype’s computation.
First, the procedure segments words per emotion; in the case of the EmoLex, it associates each word with ten emotions and sentiments: anger, anticipation, disgust, fear, joy, negative, positive, sadness, surprise, and trust. Note that words linked with several emotions will be linked with such emotions. The emotion sub-lexicons are then used to create a unique vector prototype that summarizes the emotion using word embeddings of individual words, using the companion fastText word-embedding model. These emotion-prototypes are stored as = {1, 2, · · · , 10}, and will be used to encode user’s messages and be able to train the models and predict new instances.
Once emotion-prototypes are created, we can train and predict. Figure 2 shows the flow that transforms an author into an emotion vector. Each author is represented as the sequence of its messages; these messages are plain text normalized and partitioned into a list of tokens. This procedure removes diacritic symbols, punctuation signs, duplicate letters, stop words, URLs, and emojis in this step. Along with text normalization, all user mentions are normalized to USER. Messages are then tokenized as unigrams. Finally, these unigrams are used to create a sequence that will be used to create a 300-dimensional vector.
Segmentation
* =‖* ‖ Emotions to vector
emotion-prototypes as follows:
* = ∈e∑mo︁tion ‖‖ where emotion is the set of words per emotion found in the emotion lexicon; and represents the embedding vector of term .
Output: Emotion prototypes
= {1, 2, · · · , 10} Computation for the following emotions: 1. Anger 2. Anticipation 3. Disgust 4. Fear 5. Joy
The emotion-prototypes are used to map user’s messages to an emotion-space, using the cosine similarity among prototypes and user vectors. First, we need to transform the sequence of words into a single vector using the companion word-embedding; then, we will use this vector and vector prototypes to create a 10-dimensional vector used by a classifier at training and prediction stages.
Our implementation of the emotion-space above detailed in §2 is based on a number of well-known
libraries in the Python language. NLTK [
We ran our tests on a computer with a 3.4 GHz Quad-core Intel Core i7, 32 GB RAM, and operating system macOS Catalina 10.15.7.
The data provided by PAN organizers were 200 XML documents for both English and Spanish, each document corresponds to a user and includes 200 tweets written by him.
Text preprocessing and normalization
Sequence to vector The normalized messages are represented as a vector as follows: ˇ = ∑︁
∈text ‖‖ ^ =‖ˇˇ‖ where text is a collection of words in user’s messages, and represents the 300-dimensional vector of term in the word embedding.
Emotion-based vectorization
= [cos(1, ^), cos(2, ^), · · · , cos(10, ^)] where is an Emotion-prototype and ^ the 300-dimensional user vector
uses the fastText’s get_sentence_vector to vectorize user’s messages.
cos(, ) = ∑‖︀‖‖·‖ User’s vector with the following attributes: 1. Anger 2. Anticipation 3. Disgust 4. Fear 5. Joy
We create the emotion-prototypes and project our datasets as described in §2. The preprocessing of texts is performed with the help of NLTK5 package. Therefore, we characterize every user with a 10-dimensional vector, and this representation is the input for the machine learning models at training and prediction stages.
In order to participate in the PAN contest, we develop several models using the training data provided by the organizers. We split the original training dataset into two subsets to test our models. We randomly assign 70%/30% from the training dataset to training/test splits, each of which forms a balanced subset.
We consider several classifiers for creating our emotion-based models. More precisely, we use Naive Bayes (NB), K-Nearest Neighbor (KNN), both linear and non-linear Support Vector Machine (SVM), Nearest Centroid (NC), Logistic Regression (LR), and Gradient Boosting (GB) from the Scikit-Learn package. We ran a hyperparameter optimization process to select those models that perform the best. Nonetheless, we used default parameters for Logistic Regression. Table 1 lists the parameter grid used for the model selection.
We used grid search on the mentioned space for the search process and weighted each model using a 5-fold, 3-repetition stratified k-fold cross-validation. We chose the parameter combination in each language that had the highest accuracy during the cross-validation for the final selection. Thus, the final models were fitted on the entire training set. The best hyperparameters are summarized in Table 2.
After every learning algorithm fit a model for each language, we evaluate them using the test subset to predict the class labels, obtaining an estimate of how well these models perform on unseen data. The accuracies achieved are shown later in Table 3. Finally, based on the test results, the ML models selected to participate in the Hate Speech Spreader task were chosen.
This section presents the experimental results of our approach for the PAN @ CLEF21 Hate Speech Spreader Profiling task. As commented, we divided the dataset to perform model selection, and therefore we show the results for this internal process and the results in the ofcfiial gold standard.
Figure 3 shows the performance of model evaluation using 5-fold with 3-repetition stratified k-fold cross-validation for both English and Spanish. The classifiers SVM for English and LSVM for Spanish evidence that their data points consistently hover around the center values; thus, the predictions will have less variation. Likewise, these two models have competitive medians comparing to the others.
0.8 0.7 ion0.6 t a u l a v E 0.5 0.4
KNN SVM NB NC GB KNN SVM NB NC
GB LinearSVM
The performance for the machine learning models is shown in Table 3. Note that the highest accuracy during the cross-validation is SVM with RBF kernel for English and LinearSVC for Spanish. Subsequently, we choose these two models based on the model evaluation (see Fig. 3) and the accuracy from Table 3.
Analysis of emotion-based hate speech spreader models
Gradient boosting is an ensemble of decision trees that, instead of accessing data through a kernel
function, accesses attributes directly. Decision trees require the computation of the importance of
each attribute as part of its construction algorithm; it is feasible to use it outside of the decision
tree context to obtain insight into the problem. In particular, we use the Gini importance [
Figure 4 shows the importance of each attribute, as seen by our models. For example, in the case
of English, Fig. 4a, there is some remarkable difference between the fourth most important and
the rest of them. Trust, Disgust, Anticipation, and Joy are the most critical predictors in English,
highlighting Trust and Disgust. On the other hand, Fig. 4b shows the attribute importance for the
Spanish language; Disgust dominates the other features, clearly standing out for determining hate
speech spreaders. Our Spanish modeling closely mimics Plutchik’s classification of emotions [
Figure 5 shows the emotion and sentiment distributions per class, computed on the PAN @ CLEF21 training set. The top row shows distributions for the English language. Here we observe that no hate speech spreaders (negative examples) have large variations in their emotions as
Trust
Disgust Anticipation
Joy Surprise
Fear
Anger Sadness Negative
Positive 0.975 0.950 0.925 0.900 0.875 0.850 0.825 0.800 0.90 0.85 0.80 0.75 0.70 0.975 0.950 compared with those in the positive class. Note that several median values (white point in the box inside the violin shape) also dramatically moves. This effect is easily noticeable by those features with the highest Gini importance, see Fig. 4a. Figures 5c and 5d illustrates how emotions distributes for the Spanish language writers for negative and positive classes. Again, we observe that mass concentrates around the median for positive class; we can also observe a noticeable difference, the median in some attributes like Anger, disgust, and fear; this could be associated with a higher emotional charge in the Spanish language Hate Speech spreaders.
This paper proposes Semantic Emotion-based models on both Spanish and English languages to cope with the Profiling Hate Speech Spreaders challenge at PAN @ CLEF21. Our approach was designed to be explainable through the projection of users into an Emotion-space induced by EmoLex, supporting lexical variations of the same word based on semantic representations with the help of word embeddings such as fastText.
We conducted a broad model selection study to get the best performing algorithms for our approach. In this sense, we selected SVM with RBF kernel for English and Linear SVC for Spanish as our emotional-based models. Unfortunately, the accuracy of our models was 5% lower on the test set compared to the cross-validation results.
There is still room to improve our work in the future; our next steps in the research include exploring the use of different word embeddings and lexicons with different emotion classifications to improve the overall effectiveness of the classification process. Due to our emotion-centered modeling, we noticed that the Spanish model closely mimics Plutchik’s classification of emotions for hatred from our experiments. This behavior requires a more profound exploration to describe its effect on a more fine scale.