People sometimes try to tell a joke to amuse others, but they can also hurt some other person or social group in the process, consciously or unconsciously. The HUrtful HUmour (HUHU) shared task tries to encourage the development of systems that detect and classify ofensive texts and whether or not they are intended to be humorous. In this paper, we detail the participation of the RETUYT-InCo team in the HUHU shared task, where we got the first place for task 1 with an F 1 score of 0.820.
Disparagement humor is how some authors refer to the amusement through hurting some group
of people. This concept is related to the phenomenon where a person attempts to make a joke
by humiliating a social group based on some characteristics, such as gender, sexual orientation,
appearance, beliefs, nationality, or other common backgrounds [
The HUrtful HUmour (HUHU) shared task [
In this paper, we describe the approaches followed by the RETUYT-InCo team for our
participation in the HUrtful HUmour (HUHU) shared task [
The rest of the paper is structured as follows: section 2 presents the data used for training or ifne-tuning the models; section 3 describes the systems developed for our submissions; section 4 shows the results for our submissions and the positions obtained in the ranking; finally section 5 includes the conclusions of our work.
The training set made available by the organizers contains 2671 tweets, all of them considered to be hurtful or conveying prejudice in some way. Each tweet is also labeled according to five dimensions: if it is considered humorous, and if it shows prejudice towards women, the LGBTIQ community, immigrants or people’s race, and overweight people.
Figure 1 shows the balance of humorous/non-humorous tweets and the proportion of tweets belonging or not to each class for each dimension. For example, it can be noticed the prejudice_overweight category is pretty unbalanced towards the negative class, while the prejudice_woman category is more balanced in general.
Figure 2 shows a histogram of the mean prejudice values broken down by group. Note that there seems to be a diference in the distribution of the prejudice values for humorous and non-humorous tweets, with the humorous class shifted toward higher values.
The previously mentioned data provided by the shared task includes a labeled training set and an unlabeled test, but no dev data. In order to compare our diferent experiments internally, Intended Humor?
Yes No 1 2
Mean Prejudice3 4 5 we decided to make our own internal 90%-10% split of the training data into train and dev sets. Throughout the rest of the document, whenever we refer to the “train set” and “dev set”, we will refer to our own internal splits. Our train set contains 2404 tweets, and our dev set contains 267 tweets, and we aimed to keep the partition as similarly balanced as the original set as possible, although this was dificult for the prejudice group 4 ( prejudice_overweight), as there were too few examples of the positive class.
The systems we proposed can be divided into two big groups: classic machine learning methods and fine-tuning pre-trained deep learning models. The code for these experiments can be found here: https://github.com/pln-fing-udelar/retuyt-inco-huhu-2023. We have applied these methods generally to all the tasks by only changing the part right before the output – changing the classification/regression head.
We employed a pipeline that consists of data augmentation, preprocessing the texts, extracting features, dimensionality reduction, and using classic machine learning methods. For data augmentation, we specifically considered the data balancing issue, as shown in Figure 1. We experimented with upsampling the data by employing back-translation (from Spanish to English and then back) on 20% of the training dataset. At the preprocessing time, we considered lowercasing, removing Spanish-specific accents, deleting numbers, and removing Twitterspecific devices such as mentions, URLs, and hashtags. All these preprocessing techniques were not always applied; we considered diferent combinations. After this, we tokenized the texts and also experimented with applying stemming and removing stop words. For dimensionality reduction, we considered applying Principal Component Analysis (PCA). The rationale behind employing dimensionality reduction is that, in some cases, we experimented with a large number of features yet the training set size is rather small. For the feature extraction, we explored multiple methods, including Bag of Words (BoW), tf–idf, and diferent pre-trained language models.
We explored features from several pre-trained language models, including RoBERTuito [11] (base uncased) and multiple models within SentenceTransformers [12]. We hypothesized that RoBERTuito would have a great performance in the context of the HUHU task since it was trained on a large dataset of tweets written in Spanish. However, the pretrained model does not provide or guarantee any fixed-length sentence representation (tweet representation, in our case). Still, to our surprise, we found that using the [CLS] token (without any fine-tuning; even when it was unused during training) presents great performance, as the next section of this paper shows. We also believed that SentenceTransformers models would be a suficient fit since they were trained to provide fixed-length sentence representation without any fine-tuning. The SentenceTransformers pre-trained models we experimented with are Sentence-T5 [13] (base and large), all-MiniLM-L6-v2 (based on MiniLM [14]), GTR [15] (base, large, and cohere-io/gtr-t5-large-1-epoch), XLM-RoBERTa [16] (symanto/sn-xlm-roberta-base-snli-mnli-anli-xnli, paraphrase-xlm-r-multilingual-v1), BETO-based models [17] (hiiamsid/sentence_similarity_spanish_es), GPT-Neo [18] variant 2.7B, BERTIN-RoBERTA [19], and a Spanish variant of RoBERTa (Maite89/Roberta_finetuning_semantic_similarity_stsb_multi_mt). Finally, we also tried obtaining several real-valued features from pysentimiento [20], including the hate-speech-specific features: aggressiveness ( ag), targeted (tr), and general hate-speech (hs). Note that we did not perform preprocessing when computing features from pre-trained language models.
To learn from the data using the features, the Machine Learning methods we explored were: Support Vector Machines (SVM) [21], k-Nearest Neighbors (kNN), Gradient Boosting, Decision Trees, Random Forests, Logistic/Linear Regression, SGD-optimized linear models, MLP, Neural Networks, and Naive Bayes. These methods were implemented using the scikit–learn library [22].
We want to note that we did not try all combinations of the mentioned methods with the pre-processing techniques, still, we experimented with combinations that we believe make sense. See the next section for details on the presented results.
Apart from leveraging frozen features coming from pre-trained language models (among other features) as input to shallow machine learning models, we experimented with fine-tuning pre-trained learning language models. Concretely, we conducted experiments with XLMRoberta [16], RoBERTuito [11], and BERT [30, 17]. When using BERT, we froze the backbone. For the other two models, all the parameters were trainable. In all cases, we tuned a new head on the training set. For XLM-RoBERTa, we average the backbone outputs to pass them to the new classifier head. In the case of BERT and RoBERTuito, we take the [CLS] token output. We explored using diferent probabilities of dropout as well as using one or two linear layers separated by ReLU activations. We trained the models using the Adam [31] optimizer with RoBERTuito Embedding PCA 100 kNN RoBERTuito Embedding PCA 200 kNN RoBERTuito Embedding kNN RoBERTuito Embedding PCA 100 MLP RoBERTuito Embedding PCA 200 MLP SentenceTransformers XLM-RoBERTa Linear SentenceTransformers BERTIN-RoBERTa Linear SentenceTransformers SimilaritySpanish Linear SentenceTransformers T5 kNN RoBERTuito fine-tuning BERT Multilingual fine-tuning 0.7349 0.7317 0.7317 0.7416 0.7473 0.5952 0.6383 0.6590 0.7636 0.7935 0.6627 diferent learning rates.
For RoBERTuito, which has the same architecture as RoBERTa, we introduced a classification head consisting of 592,130 additional parameters. This brings the total number of trainable parameters during the fine-tuning step to 108,196,608. We utilized the tokenizer provided by the pretrained model for consistency. During training, we employed a learning rate of 3 × 10− 5 with the Adam optimizer, and the training was conducted for two epochs.
The methods carried out for each task difer mostly in the last part of the modeling. For Task 1, we employed softmax for the classification and used the cross-entropy loss. For Task 2a, we treated the problem as a multi-label classification and thus computed probabilities for each class separately, dividing it into four binary classification tasks (and then proceeded similarly to Task 1). For both of these tasks, we also consider kNN applied only to the centroid as opposed to the whole training set. For each class, a positive and negative centroid was computed to then compare new instances. For Task 2b, we regressed the output variable and used the mean squared error as the loss.
tf–idf GradientBoosting BoW RandomForest SentenceTransformers XLM-RoBERTa Linear SentenceTransformers BERTIN-RoBERTa Linear SentenceTransformers SimilaritySpanish Linear SentenceTransformers T5 kNN RoBERTuito fine-tuning BERT Multilingual Fine-tuning Group 1 Group 2 Group 3 Group 4
We presented the submissions made by the RETUYT-InCo team for the HUHU shared task presented at IberLEF 2023. Our experiments include a variety of classical and neural machine learning models, trained with diverse features (from BoW to sentence embedding features), as well as some fine-tuning from pre-trained LLM experiments. The models performed well over the test set, obtaining the first place for Task 1, with an F 1 score of 0.820, and a good ranking in general for the other tasks.
In future work, we want to explore the possibility of adding more information to the dataset Task
1 to enrich the models. For example, we could add more tweets from the HAHA 2021 dataset [8], which includes information on humor target, a category that is closely related to the prejudice groups used in the HUHU shared task. IberLEF 2019: Humor analysis based on human annotation, in: IberLEF@SEPLN, 2019, pp. 132–144. URL: https://ceur-ws.org/Vol-2421/HAHA_overview.pdf. [8] L. Chiruzzo, S. Castro, S. Góngora, A. Rosá, J. Meaney, R. Mihalcea, Overview of HAHA at IberLEF 2021: Detecting, rating and analyzing humor in Spanish, Procesamiento del Lenguaje Natural 67 (2021) 257–268. URL: http://journal.sepln.org/sepln/ojs/ojs/index.php/ pln/article/view/6394. [9] J. Meaney, S. R. Wilson, L. Chiruzzo, W. Magdy, Don’t take it personally: Analyzing gender and age diferences in ratings of online humor, in: Social Informatics: 13th International Conference, SocInfo 2022, Glasgow, UK, October 19–21, 2022, Proceedings, Springer, 2022, pp. 20–33. URL: https://link.springer.com/chapter/10.1007/978-3-031-19097-1_2. [10] J. Meaney, S. Wilson, L. Chiruzzo, A. Lopez, W. Magdy, SemEval 2021 task 7: Hahackathon, detecting and rating humor and ofense, in: Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021), 2021, pp. 105–119. URL: https://aclanthology.org/ 2021.semeval-1.9/. [11] J. M. Pérez, D. A. Furman, L. Alonso Alemany, F. M. Luque, Robertuito: a pre-trained language model for social media text in spanish, in: Proceedings of the Language Resources and Evaluation Conference, European Language Resources Association, Marseille, France, 2022, pp. 7235–7243. URL: https://aclanthology.org/2022.lrec-1.785. [12] N. Reimers, I. Gurevych, Sentence-BERT: Sentence embeddings using Siamese BERTnetworks, in: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Association for Computational Linguistics, Hong Kong, China, 2019, pp. 3982–3992. URL: https://aclanthology.org/D19-1410. doi:10.18653/v1/ D19-1410. [13] J. Ni, G. Hernandez Abrego, N. Constant, J. Ma, K. Hall, D. Cer, Y. Yang, Sentence-t5: Scalable sentence encoders from pre-trained text-to-text models, in: Findings of the Association for Computational Linguistics: ACL 2022, Association for Computational Linguistics, Dublin, Ireland, 2022, pp. 1864–1874. URL: https://aclanthology.org/2022.findings-acl.146. doi:10.18653/v1/2022.findings-acl.146. [14] W. Wang, F. Wei, L. Dong, H. Bao, N. Yang, M. Zhou, MiniLM: Deep self-attention distillation for task-agnostic compression of pre-trained transformers, in: H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, H. Lin (Eds.), Advances in Neural Information Processing Systems, volume 33, Curran Associates, Inc., 2020, pp. 5776–5788. URL: https://proceedings. neurips.cc/paper_files/paper/2020/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf. [15] J. Ni, C. Qu, J. Lu, Z. Dai, G. H. Ábrego, J. Ma, V. Y. Zhao, Y. Luan, K. B. Hall, M.-W. Chang, et al., Large dual encoders are generalizable retrievers, arXiv preprint arXiv:2112.07899 (2021). URL: https://arxiv.org/abs/2112.07899. [16] A. Conneau, K. Khandelwal, N. Goyal, V. Chaudhary, G. Wenzek, F. Guzmán, E. Grave, M. Ott, L. Zettlemoyer, V. Stoyanov, Unsupervised cross-lingual representation learning at scale, in: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Association for Computational Linguistics, Online, 2020, pp. 8440–8451. URL: https://aclanthology.org/2020.acl-main.747. doi:10.18653/v1/2020.acl-main.747. [17] J. Cañete, G. Chaperon, R. Fuentes, J.-H. Ho, H. Kang, J. Pérez, Spanish pre-trained BERT model and evaluation data, in: PML4DC at ICLR 2020, 2020. URL: https://pml4dc.github. io/iclr2020/papers/PML4DC2020_10.pdf. [18] S. Black, L. Gao, P. Wang, C. Leahy, S. Biderman, GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow, 2021. URL: https://doi.org/10.5281/zenodo. 5297715. doi:10.5281/zenodo.5297715, If you use this software, please cite it using these metadata. [19] J. De la Rosa, E. G. Ponferrada, M. Romero, P. Villegas, P. G. de Prado Salas, M. Grandury, BERTIN: Eficient pre-training of a spanish language model using perplexity sampling, Procesamiento del Lenguaje Natural 68 (2022) 13–23. URL: http://journal.sepln.org/sepln/ ojs/ojs/index.php/pln/article/view/6403. [20] J. M. Pérez, J. C. Giudici, F. Luque, pysentimiento: A Python Toolkit for Sentiment Analysis and SocialNLP tasks, 2021. URL: https://arxiv.org/abs/2106.09462. arXiv:2106.09462. [21] C. Cortes, V. Vapnik, Support-vector networks, Machine learning 20 (1995) 273–297. URL: https://link.springer.com/article/10.1007/BF00994018. doi:10.1007/BF00994018. [22] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, E. Duchesnay, Scikit-learn: Machine learning in Python, Journal of Machine Learning Research 12 (2011) 2825–2830. URL: https://scikit-learn.org/. [23] J. D. Hunter, Matplotlib: A 2D graphics environment, Computing in science & engineering 9 (2007) 90–95. URL: https://matplotlib.org/. [24] T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz, J. Davison, S. Shleifer, P. von Platen, C. Ma, Y. Jernite, J. Plu, C. Xu, T. Le Scao, S. Gugger, M. Drame, Q. Lhoest, A. Rush, Transformers: State-of-the-art natural language processing, in: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, Association for Computational Linguistics, Online, 2020, pp. 38–45. URL: https://aclanthology.org/2020.emnlp-demos.6. doi:10.18653/v1/2020.emnlp-demos.6. [25] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, S. Chintala, PyTorch: An Imperative Style, High-Performance Deep Learning Library, in: H. Wallach, H. Larochelle, A. Beygelzimer, F. d’Alché Buc, E. Fox, R. Garnett (Eds.), Advances in Neural Information Processing Systems 32, Curran Associates, Inc., 2019, pp. 8024–8035. URL: http://papers.neurips.cc/ paper/9015-pytorch-an-imperative-style-high-performance-deep-learning-library.pdf. [26] C. R. Harris, K. J. Millman, S. J. Van Der Walt, R. Gommers, P. Virtanen, D. Cournapeau, E. Wieser, J. Taylor, S. Berg, N. J. Smith, et al., Array programming with NumPy, Nature 585 (2020) 357–362. URL: https://numpy.org/. [27] S. Bird, E. Klein, E. Loper, Natural Language Processing with Python: Analyzing Text with the Natural Language Toolkit, O’Reilly Media, Inc., 2009. URL: https://www.nltk.org/. [28] M. Honnibal, I. Montani, S. Van Landeghem, A. Boyd, spaCy: Industrial-strength Natural Language Processing in Python, 2020. URL: https://spacy.io/. doi:10.5281/zenodo. 1212303. [29] The pandas development team, pandas-dev/pandas: Pandas, 2023. URL: https://github.
com/pandas-dev/pandas. [30] J. Devlin, M.-W. Chang, K. Lee, K. Toutanova, BERT: Pre-training of deep bidirectional transformers for language understanding, in: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Association for Computational Linguistics, Minneapolis, Minnesota, 2019, pp. 4171–4186. URL: https://aclanthology.org/ N19-1423. doi:10.18653/v1/N19-1423. [31] D. P. Kingma, J. Ba, Adam: A method for stochastic optimization, in: Proceedings of the 3rd International Conference on Learning Representations (ICLR), 2014. URL: https: //arxiv.org/abs/1412.6980.