This paper presents a comprehensive approach to detect humor that spreads prejudice on Twitter. Our methodology utilizes embedding extraction and fine-tuning techniques, employing 1D Convolutional Neural Networks (CNN) to capture relationships among embeddings and enhance model performance. Additionally, we leverage sentiment analysis, along with other extracted variables, to further improve the efectiveness of the models. We address three distinct tasks in our evaluation. The first task focuses on distinguishing tweets that express prejudice through humor from those that express prejudice without humor. In the second task, we perform multilabel classification to identify the targeted minority groups in prejudiced tweets, including women and feminists, the LGBTIQ community, immigrants and racially discriminated people, and over- weight individuals. The third task involves predicting the degree of prejudice on a continuous scale ranging from 1 to 5 for tweets targeting minority groups. Experimental results demonstrate the eficacy of our approach, highlighting the significance of capturing relationships among embeddings using 1D Convolutional Neural Networks. Additionally, the incorporation of sentiment analysis and other extracted variables further enhances model performance. Our findings contribute to advancing sentiment analysis and prejudice detection in social media, fostering a more inclusive online environment. The proposed methodology opens up avenues for future research and development in this domain
The expression of prejudice is a common strategy used to harm individuals from minority
groups [
The first subtask aims to determine whether a prejudicial tweet is intended to cause humor. Participants are required to distinguish between tweets that use humor to express prejudice and tweets that express prejudice without using humor. This task involves binary classification, where systems are evaluated and ranked based on the F1-measure over the positive class. The F1-measure provides a balanced evaluation of both precision and recall, capturing the system’s ability to correctly identify hurtful humor instances.
In the second subtask, participants are asked with identifying the targeted minority groups in the tweets. The specified minority groups include women and feminists, the LGBTIQ community, immigrants and racially discriminated people, and overweight people. This task is formulated as a multilabel classification problem, where systems need to assign relevant labels to each tweet. The evaluation metric employed for this task is the macro-F1 score, which considers the average performance across all labels.
The third subtask focuses on predicting the degree of prejudice expressed in the tweets on a continuous scale ranging from 1 to 5. Participants are required to assign a numerical value to indicate the level of prejudice exhibited towards the minority groups. The evaluation metric used for this task is the Root Mean Squared Error (RMSE), which measures the average diference between the predicted values and the ground truth values. A lower RMSE indicates better performance in accurately predicting the degree of prejudice.
The dataset used in this study consists of 2,671 tweets collected exclusively from the Twitter platform. These tweets were manually labeled and annotated regarding attributes such as humor and prejudice. The aim of this dataset is to provide information and analysis on the presence and manifestation of humor and prejudice in the context of Twitter.
Regarding the dataset characteristics, there is an uneven distribution in the classes of humor and prejudice targets. In terms of humor, approximately 32.5% of the tweets are classified as humorous, while the remaining 67.5% do not contain humorous elements.
In relation to prejudice, the following proportions are observed: 48.4% of the tweets show some form of prejudice towards women, 22.7% towards the LGBTQIA+ community, 24.9% towards immigrants or racial groups, and 8.0% specifically exhibit prejudice towards overweight individuals (fatphobia).
Lastly, the average value of ”mean_prejudice” in the dataset is approximately 3.05, with a standard deviation of 0.81. This indicates that, on average, the tweets exhibit a moderate level of prejudice in their content.
In this section, we explain the process of features extraction from the Twitter data. We employed the following resources and methods:
HurtLex [
We utilized pretrained transformer models to extract variables related to emotions, irony, and
cyberbullying from the tweets. Specifically, we used the twitter-xlm-roberta-emotion-es [
The extraction of these features enables us to capture the presence of ofensive language, stereotypes, emotions, irony, and cyberbullying in the Twitter data. This information contributes to enhance the performance of our models and provides valuable insights into the nature of the tweets.
Embeddings are obtained from the pretrained transformer model bertin-roberta-base-spanish
[
To optimize the performance of our models, a grid search with 10-fold cross-validation is performed. This allows us to explore diferent hyperparameter combinations and select the best configuration for each model.
Four models are compared in our experiments: Support Vector Machine (SVM), Multi-Layer Perceptron (MLP), XGBoost, and Random Forest. Each model is trained on the embedded tweet data to learn the patterns and relationships between the input features and the target labels.
In addition to the tweet embeddings, we also analyze the efect of using sentiment features. These features capture the sentiment expressed in the tweets, which can provide valuable insights for prejudice detection. To feed the machine learning models, we utilize the embedding of the special token [CLS], which represents the overall meaning of the entire tweet. Additionally, to incorporate sentiments, we concatenate seven emotion-associated components (sadness, joy, anger, surprise, disgust, fear, and others) to this embedding.
This process is applied to all three tasks: HUrtful HUmour Detection, Prejudice Target Detection, and Degree of Prejudice Prediction. The same set of models and hyperparameter optimization techniques are employed for each task, allowing us to evaluate their performance consistently across the diferent aspects of prejudice detection.
The results of the cross-validation for the proposed method are presented below.
For Task 1, the performance of SVM, MLP, and XGBoost models was similar, with F1 scores approaching 0.7. The inclusion of emotions and HurtLex variables had a noticeable efect, particularly on XGBoost and Random Forest models. Figure 1 shows the cross-validation results for Task 1.
For Task 2, which involved binary classifications for each target, the models achieved macro F1scores of over 0.85. Once again, the inclusion of emotions and HurtLex variables had a noticeable efect, especially on MLP and Random Forest models. Figure 2 presents the cross-validation results for Task 2a.
For Task 2b, the SVM model achieved the best performance with an RMSE of 0.7. However, there was no observable efect when including emotions and HurtLex variables in this task. Figure 3 presents the cross-validation results for Task 2b.
Fine-tuning Transformers enables task-specific adaptation and potentially better performance compared to using precalculated embeddings with classical classifiers. The idea is to feed the Transformer model with 80 tokens representing a tweet. Instead of using the embedding associated with the [CLS] token or taking the average, we propose the use of 1-dimensional convolutional neural networks (CNN) with varying numbers of filters followed by 1-dimensional Max Pooling layers to obtain a final representation, which is then flattened into a vector.
The goal of this approach is to capture relationships between the tokens and the embeddings, allowing us to improve the performance of the models in all three tasks. To the resulting vector from the convolutional operations, we add the variables explained earlier: HurtLex, emotions, irony, and cyberbullying extracted using pretrained transformers. We then apply dense layers, and the final layer depends on the specific task. For the first task, we use a neuron with sigmoid activation, and for the second task, we use four neurons with sigmoid activation. It’s worth noting that now, unlike before with independent binary classifications, we are modeling the possible relationships between the diferent targets. For example, if a tweet expresses prejudice against women, it might also be more likely to exhibit fatphobia.
In these two tasks, binary cross-entropy is used as the optimization function. For the third task, a neuron without activation is used. The batch size used was 32, and for the second task, oversampling was performed to balance the batches.
Attention matrices provide insights into the relative importance of each word in relation to others within a sentence. They ofer a representation of how tokens interact with each other during the model’s processing.
In our approach, we calculate attention matrices for each input sentence. Each matrix consists of values that indicate the level of attention or importance assigned to diferent word pairs within the sentence. By analyzing these matrices, we can identify the strongest interactions between tokens.
To highlight the most significant interactions, we extract the maximum value from all attention matrices. This allows us to identify the pairs of words that have the highest attention or influence on each other. By focusing on these key interactions, we gain valuable insights into the relationships between words and their impact on the overall meaning of the sentence.
Our experimentation on the test set reveals the significant impact of incorporating emotions
and HurtLex features in improving metrics compared to using solely pre-trained transformer
embeddings. This demonstrates the value of leveraging additional linguistic features to enhance
model performance. Furthermore, the power of fine-tuning a transformer for a specific task
is evident in our results. We conducted fine-tuning using xlm-roberta-large-spanish [
The models developed using the proposed architecture have allowed us to achieve a commendable position in the ranking. The incorporation of emotions and HurtLex variables, along with fine-tuning transformers, has proven to be efective in improving model performance across all three tasks. These results reflect the dedication and efort invested in building robust models.
• Rank: 5th • F1 score: 0.784
• Rank: 10th • Macro F1 score: 0.746
• Rank: 3rd • RMSE (Root Mean Squared Error): 0.881
In conclusion, our current models have demonstrated their efectiveness and enabled us to secure a strong position in the ranking. We are optimistic about the future optimization and experimentation possibilities, which we believe can yield even better results. With each iteration, we strive to push the boundaries of model performance and make meaningful contributions to the field of prejudicial language detection.
Our final approach involved fine-tuning the xlm-roberta-large-spanish transformer model for subtask 1. Similarly, for subtask 2a, we fine-tuned the bertin-roberta-base-spanish transformer model to efectively detect the targeted minority groups in the tweets.
Furthermore, we incorporated emotions and HurtLex features to enhance the models’ understanding of the linguistic context and improve their performance. This additional information proved valuable in capturing nuanced patterns and further improving F1 score.
Finally, we employed ensemble techniques, combining Support Vector Machine (SVM) and XGBoost models, to address the challenges of Subtask 2b and achieve superior results. This ensemble approach allowed us to leverage the strengths of each model and achieve better predictive performance. 10. Future work However, we believe that there is still room for further optimization and experimentation with the proposed architecture. By exploring diferent hyperparameter settings, conducting more extensive grid searches, and fine-tuning the model, we anticipate that even better results can be achieved.
With a deeper understanding of the task requirements and the potential of the proposed architecture, we are confident that future iterations of our models can surpass the current results. By continuously refining and iterating on our approach, we aim to contribute to the development of state-of-the-art models for prejudicial language detection.