This work describes the participation of the UMUTeam in the HOMO-MEX shared task at IberLEF 2023, on Hate speech detection in Online Messages directed tOwards the MEXican Spanish speaking LGBTQ+ population. We have addressed the two proposed tasks: Task 1, consisting of identifying the category of hate speech and, Task 2, on determining the types of phobia from a given set of tweets. For both tasks, we have evaluated diferent approaches based on the combination of sentence embeddings using ensemble learning and knowledge integration. Specifically, the sentence embeddings have been extracted from several Spanish and multilingual Large Language Models after fine-tuning them for each task separately. In total, 11 teams participated in Task 1 and 9 teams in Task 2. The best run sent by our team placed in position 3rd for Task1 and position 8th for Task 2 with an F1-score of 0.842 and a macro-average F1-score of 0.669, respectively, with 0.885 and 0.696 being the results obtained by the teams ranked in 1st position.
Spanish and other Iberian languages. The competition was organized through CodaLab and can be accessed at the following link: https://codalab.lisn.upsaclay.fr/competitions/10019.
The objective of the HOMO-MEX shared task is to improve automatic detection systems designed for the classification of hate speech directed towards the LGBT+ community. Specifically, it is proposed two tasks: • Task 1: Hate speech detection. It is a multi-class classification task and consists of, given a tweet, classifying it in one of the following categories: – LGBT+ phobic, if the tweet contains hate speech directed anyone whose sexual orientation and/or gender identity difers from cis-heterosexuality. – not LGBT+ phobic, if the tweet does not include hate speech towards the LGBT+ population, but mentions this community.
– not LGTB+ related, if the tweet is not related to the LGBT+ community. • Task 2: Fine-grained hate speech detection. It is a multi-label classification task and consists of, given a tweet that contains LGBT+ phobia, identifying one or more types of phobia present in it: – Lesbophobia: homophobia explicitly directed at homosexuals who identify as female. – Gayphobia: homophobia explicitly directed at homosexuals who identify as male. – Biphobia: hate speech directed against people who are attracted to more than one gender. – Transphobia: hate speech directed against non-cis-gendered people. – Other LGBT+phobia: hate speech against other sexual and gender minorities not included in any of the previous categories (e.g “aphobia”: hatred received by people who do not feel sexual attraction).
Our team has participated in both tasks, in which we sent a total of 5 runs, based on the
combination of sentence embeddings extracted from several Large Language Models (LLMs)
after fine-tuning them for each task separately. These LLMs include Spanish models such as
BETO [
The rest of the paper is organized as follows. Section 2 presents the details of the dataset provided by the organizers to the participating teams. Subsequently, in Section 3, the methodology followed to carry out the experimentation is described. Next, Section 4 shows the results obtained during the validation and evaluation phases. In addition, a discussion of the results is presented. Finally, Section 5 concludes the paper with the main insights and future directions.
The dataset of this shared task is composed of 12,416 tweets written in Mexican Spanish that have been extracted between 2012 and 2022, out of which 11,000 corresponds to Task 1 and 1,416 to Task 2. At a first stage, training was made available in order to the participants develop their systems. We select a subset of these tweets for custom validation in a ratio of 80-20. Later, test sets were released to participate in both tasks. The distribution of the datasets for Task 1: Hate speech detection and Task 2: Fine-grained hate speech detection are presented in Table 1 and Table 2, respectively. We can observe that, for Task 1, the majority of the tweets do not include hate speech towards the LGTB+ community, but mention it and, for Task 2, most of the tweets have content about gayphobia. Finally mention that the organizers decided not to make the test set public after the end of the competition, so it is not possible to provide statistics on the distribution of the test data, beyond the total of tweets, nor to analyze it.
Next, we examined the corpus and its correlation with the labels using the UMUTextStats
tool [
P ) (REGech iiifttrscLaeenuug morpahpforoilxocegesycs-o-osafreudppsefjru-enoinosxcsce-fetiwevgiesvosceo-aneora-rtstessrddihivp-psvsjqoee-ue-enguw-cse-g((erats-((ciMM(((((aileRgtEplvLeSSPPhihpnOOafEReEinSSTT-y-zhetsGRRRX4sgYYYYeiiiornvelc)))))))))tratfhgexssrtl0.0
In a nutshell, our methodology can be described as follows. First, we apply some basic datacleaning to the dataset. Second, we fine-tuned each evaluated LLM separately before extracting their sentence embeddings. This fine-tuning process involve the training of 10 diferent models. Third, we evaluate several neural networks using these sentence embeddings together in a knowledge integration strategy with the objective of finding the best hyperparameters. Besides, we conduct extract hyperparameter optimization stages for each LLM separately to use their outputs in the ensemble learning strategies.
We conduct a basic data cleaning process to obtain a more generic model. We remove for the texts argot used in social networks, such as hyperlinks, hashtags, mentions and extra white spaces. We also expand some abbreviations typically used in social networks and the language used in short texts, expanded acronyms, and replace numbers with the token [NUMBER].
Once the dataset is cleaned, we fine-tune several LLMs for both tasks separately. We evaluate 10 models per LLM evaluating the following hyperparameters: the learning rate (between 1e-5 and 5e-5 following a uniform distribution), the number of epochs (between 1 and 5), the batch size (8 or 16), the warm-up steps (0, 250, 500 or 1000) and the weight decay (between 0.0 and .3 following a uniform distribution). The models are selected using HyperOptSearch with Tree of Parzen Estimators (TPE) and the ASHA Scheduler with the objective of maximizing the macro weighted f1-score. All this process is conducted using RayTune.
Table 3 depicts the results achieved in this process for Tasks 1 (left) and Task 2 (right). It can be observed than LLMs for Task 1 have much more lower warm-up steps and weight decay except in the case of BERTIN that have the same number of warm-up steps and lower weight decay. Besides, both tasks have achieved better results with smaller batch size (8 vs 16). Finally, we did not find relevant information concerning the number of training epochs nor the learning rate.
After this step, we extract the sentence embeddings for the best model of each LLM. It is worth mentioning that we extract the embeddings at sentence level because it allows us to combine them more easily in a new multi-input neural network taking profit of the strengths of each LLM. The sentence embeddings are obtained from the encoding of the classification token, as suggested in [12]. These embeddings are a fixed-length vector of 768.
Once the sentence embeddings are obtained, we evaluate to combine them using a knowledge integration strategy by feeding them in a multi-input neural network. The best configuration of this new neural network is also determined by a hyper optimization stage conducted in Keras. Now that the input are fixed sentence embeddings, we evaluate traditional neural network architectures, in which we assess the number of hidden layers and the number of neurons per layer, the learning rate, the batch size, the dropout mechanism for regularization and the activation function between layers. For Task 1, the best neural network consists of a deep neural network with 8 hidden layers and 16 neurons per layer stacked in a rhombus shape. The network uses no dropout and a learning rate of 0.01. The batch size is 512 and it uses tanh as activation function. For Task 2, however, the best results are achieved with a shallow neural network with 2 hidden layers but with 512 neurons per layer and no activation function between the hidden layers. The batch size is 64, the learning 0.01 and a strong dropout mechanism of 512.
In this section we report and discuss the results obtained during the validation phase and the oficial results achieved in the evaluation phase for Task 1: Hate speech detection and Task 2: Fine-grained hate speech detection.
We tested diferent Spanish and multilingual LLMs. Specifically, diferent approaches based on
sentence embeddings extracted from the LLMs were evaluated after fine-tuning them for each
task separately. In addition, diferent sentence embeddings combination strategies were also
evaluated by using knowledge integration and ensemble learning. The Spanish LLMs evaluated
were BETO [
Table 4 and Table 5 present the results obtained with the validation set for Task 1 and Task 2, respectively. As can be seen in Table 4, the best performing strategy in the hate speech detection task was knowledge integration and the best individual model was TWHIN. If we take a look at Table 5, for the fine-grained hate speech detection task, knowledge integration and TWHIN were also the best performing approach and best individual model, respectively. model KI EL (HIGHEST) EL (MEAN) EL (MODE) EL (WEIGHTED) precision recall f1-score
This subsection presents the results obtained in the evaluation phase. The organizers selected F1-score to rank the systems performance for Task 1 and they chose the macro-average F1-score for Task 2. Each team could submit a maximum of 5 runs, selecting the best one for ranking. We defined our 5 runs to evaluate the diferent feature integration strategies implemented. The results for each of the runs are depicted in Table 6 as well as the strategy followed in each of the run. The best result for Task 1 was obtained with the ensemble learning on a weighted mode. For Task 2, the best result was also reached with ensemble learning, but this time with the ensemble based on the mode of the labels. In general, it is observed that the combination strategies evaluated provide similar results, except for the approach of ensemble learning on the highest probability of each class in Task 2, where a notable diference is observed.
For the competition, we selected run 5 for Task 1 and run 2 for Task 2, as they were the ones that provided the best results.
Table 7 shows the oficial leader-board for Task 1, in which we achieved the 3rd position with a score of 84.21%. The results of our team are highlighted with a gray background. model KI 01. Knowledge Integration 02. Ensemble learning (mode) 03. Ensemble learning (highest probability) 04. Ensemble learning (average probabilities) 05. Ensemble learning (weighted mode)
Task 1 Task 2
In Task 2, we achieved more limited results, as it can be observed in Table 8 reaching to position 8 in the ranking with a score of 66.87%. In this case, the results among all participants are more similar, achieving a average results of 67.69 with a standard deviation of 1.17.
In this working notes we have described our participation in the HOMO-MEX shared task concerning hate-speech identification and categorization in Mexican-Spanish. We are very proud with our participation as we achieved competitive results, reaching the third position in the first task concerning hate-speech identification. In the second task, however, we achieved only the 8th position, but our results are only about a 3% less than the winner. To participate in both tasks, we fine-tuned several Spanish and Multi-lingual LLMs, extracted their sentence embedddings and combined their strengths into a multi-input neural network Knowledge Integration fashion. Besides, we evaluated other integration techniques such as ensemble learning and analyzed the data set using linguistic features.
There is space for improvement in our proposal. First, we observed that most of the errors performed by our systems using a custom validation split are related to words that very tied to Mexican Spanish. In this sense, we need to analyze if these words are recognized in the LLMs and how their embeddings are similar to more generic Spanish words. Second, our research group have already evaluated diferent datasets in Spanish concerning hate-speech [ 13]. We will use these models to validate this dataset in order to understand diferences between Spanish from Spain and Spanish from Mexico. Third, we did not evaluate data augmentation techniques for solving data-imbalanced nor the integration of the linguistic features in the ensemble or the knowledge integration model.
This work is part of the research projects AIInFunds (PDC2021-121112-I00) and LT-SWM (TED2021-131167B-I00) funded by MCIN/AEI/10.13039/501100011033 and by the European Union NextGenerationEU/PRTR. This work is also part of the research project LaTe4PSP (PID2019-107652RB-I00/AEI/ 10.13039/501100011033) funded by MCIN/AEI/10.13039/501100011033. It also has been partially supported by Project CONSENSO (PID2021-122263OB-C21), Project MODERATES (TED2021-130145B-I00) and Project SocialTox (PDC2022-133146-C21) funded by MCIN/AEI/10.13039/501100011033 and by the European Union NextGenerationEU/PRTR, Project PRECOM (SUBV-00016) funded by the Ministry of Consumer Afairs of the Spanish Government, Project FedDAP (PID2020-116118GA-I00) supported by MICINN/AEI/10.13039/501100011033 and WeLee project (1380939, FEDER Andalucía 2014-2020) funded by the Andalusian Regional Government. Salud María Jiménez-Zafra has been partially supported by a grant from Fondo Social Europeo and the Administration of the Junta de Andalucía (DOC_01073). [12] N. Reimers, I. Gurevych, Sentence-bert: Sentence embeddings using siamese bert-networks, in: K. Inui, J. Jiang, V. Ng, X. Wan (Eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019, Association for Computational Linguistics, 2019, pp. 3980–3990. URL: https: //doi.org/10.18653/v1/D19-1410. doi:10.18653/v1/D19-1410. [13] J. A. García-Díaz, S. M. Jiménez-Zafra, M. A. García-Cumbreras, R. Valencia-García, Evaluating feature combination strategies for hate-speech detection in spanish using linguistic features and transformers, Complex & Intelligent Systems (2022) 1–22.