We present in this paper our participation in the share task Multilingual Hope Speech detection (HOPE) at IberLEF-2023. It consists of two binary subtasks with notable differences in terms of language and balance of labels. For the first task we have used BERTuit, a transformer model proposed for Spanish language; and for the second one we employed DistilBERT, a light transformer model trained by distilling BERT base. After several submissions for these tasks, we achieved an average macro F1 value of 0.744 in the first task, at the second position of the leaderboard; in the second task we finally ranked at first position, with an average macro F1 value of 0.501.
Hope Speech (HS) is the type of speech that can relax a hostile environment [
By detecting it automatically, positive comments can be more widely disseminated, and can have a
very significant effect when it comes to combating sexual or racial discrimination or when we seek to
foster less bellicose environments [
Offensive messages on social media are posted towards people because of their race, color, ethnicity,
gender, sexual orientation, nationality, or religion. As [
The shared task “HOPE. Multilingual Hope Speech detection” is part of IberLEF-2023, an
evaluation campaign for Natural Language Processing (NLP) systems where several challenges are run
with large international participation from research groups in academia and industry [
After our participation in this task, we present a summary of the work and the results obtained. The next section shows a description of the subtasks and datasets provided by the organizers. The experimental methodology and evaluation results are laid out in Sections 3 and 4. And some conclusions are presented in the last section.
Next, we will briefly describe the subtasks and datasets provided for the task “HOPE. Multilingual Hope Speech detection”, part of IberLEF-2023. 2.1.
This subtask consists of, given a Spanish tweet, identifying whether it contains hope speech or not. The possible categories for each text are: • HS: Hope Speech.
• NHS: Non-Hope Speech.
The data provided for this subtask consists of a set of LGBT-related tweets annotated as HS (Hope Speech) or NHS (Non-Hope Speech) [9].
A tweet is considered as HS if the text: i) explicitly supports the social integration of minorities; ii) is a positive inspiration for the LGTBI community; iii) explicitly encourages LGTBI people who might find themselves in a situation; or iv) unconditionally promotes tolerance. On the contrary, a tweet is marked as NHS if the text: i) expresses negative sentiment towards the LGTBI community; ii) explicitly seeks violence; or iii) uses gender-based insults.
The number of samples and distribution for each category in the training dataset provided by the organization is shown in Table 1. As it can be seen, the categories were almost equally distributed. 2.2.
This subtask consists of, given an English Youtube comment, identifying whether it contains hope speech or not. The possible categories for each text are: • HS: Hope Speech.
• NHS: Non-Hope Speech.
The English corpus provided for this subtask is an extension of the English part of the HopeEDI dataset [10]. It consists of comments posted on YouTube videos on a wide range of socially relevant topics such as Equality, Diversity, and Inclusion, including LGBTIQ issues, COVID-19, women in STEM, Black Lives Matter, etc.
The number of samples and distribution for each category in the training dataset provided by the organization is shown in Table 2. As it can be seen, there exist a clear unbalance in the dataset.
Category NHS (Non-Hope Speech)
HS (Hope Speech)
To evaluate the results at both subtasks, precision, recall, and F1-score were measured per category and averaged using the macro-average method. Models were ranked using the macro-F1 score.
A Transformer is a deep learning model that adopts the self-attention mechanism, differentially weighting the importance of each part of the input data. It is frequently used in the fields of NLP and Computer Vision [11].
Simple Transformers is an NLP library designed to simplify the usage of transformer models without having to compromise on utility. It is built on the work of Hugging Face and their Transformers library [12, 13].
At the highest level, Simple Transformers is branched into common NLP tasks such as text classification, question answering, and language modeling. Each of these tasks have their own taskspecific Simple Transformers model. It has built-in support for: • Text Classification • Token Classification • Question Answering • Language Modeling • Language Generation • Multi-Modal Classification • Conversational AI • Text Representation Generation
While all the task-specific models maintain a consistent usage pattern (initialize, train, evaluate, predict), this separation allows the freedom to adapt the models to their specific use case. Figure 1 shows the initial code for an example of a classification model. 3.1.
For the “Hope Speech detection in Spanish” we have employed BERTuit, a transformer proposed for Spanish language, and pre-trained using RoBERTa optimization [14]. The transformer had been trained from scratch with text created by native speakers from Twitter, by using more than 230 million Tweets from the Archive Twitter Stream Grab [15], from 2021 to 2018.
The training dataset provided by the organizers consisted of 1612 records, of which 821 (51%) were of the NHS (Non-Hope Speech) category.
A basic pre-processing was applied to the text, consisting of: • change text to lowercase • remove http/https links • remove the hash sign (#) • strip whitespace (including newlines) Apart from that, all the digits were eliminated from the text in some versions of the BERTuit model.
We have mainly used the default hyperparameters, some of which are shown in the Table 3.
Apart from that, some extra preprocessing was applied in some versions of the DistilBERT model. The additional preprocessing consisted in removing Unicode characters, and also removing single letters and numbers surrounded by space.
We have mainly used the default hyperparameters, some of which are shown in the Table 4.
Several versions of the BERTuit model were trained and tested using the test dataset provided by the organizers. The configuration options of the best ones are shown in Table 5.
Several versions of the DistilBERT model were trained and tested using the test dataset provided by the organizers. The configuration options of the best ones are shown in Table 7.
The corresponding evaluation metrics for these models are shown in Table 8. As it can be seen, the late submission improved the macro F1, due to the increase in both the precision and recall of the HS category; so it can be deduced that the extra preprocessing worked well in this case.
In this paper we present the methods and results for the shared task Multilingual Hope Speech detection (HOPE) at IberLEF-2023. We employed BERTuit and DistilBERT transformer models for the two binary subtasks, respectively. We have used the Simple Transformers library for the python language.
After some preprocessing and light tuning of hyperparameters, we achieved acceptable results in both subtasks after several submissions for testing. We ranked at the second position of the leaderboard in the Spanish language subtask, and at first position for the English language.
As future work that could improve the classification results, oversampling/undersampling techniques could be applied for Subtask 2, which presents a clear unbalance in the distribution of categories. Also, it might be worth further tuning of the hyperparameters.
This paper is part of the I+D+i Project titled “Conspiracy Theories and hate speech online: Comparison of patterns in narratives and social networks about COVID-19, immigrants, refugees and LGBTI people [NON-CONSPIRA-HATE!]”, PID2021-123983OB-I00, funded by MCIN/AEI/10.13039/501100011033/ and by “ERDF A way of making Europe”.
We want to thank Juan M. Calvo Flores for his suggestion on the use of the Simple Transformers library. [9] García-Baena, D., García-Cumbreras, M. Á., Jiménez-Zafra, S. M., García-Díaz, J. A., & Valencia-García, R. (2023). Hope speech detection in Spanish: The LGBT case. Language Resources and Evaluation, 1-28. [10] Chakravarthi, B. R., Muralidaran, V., Priyadharshini, R., Cn, S., McCrae, J. P., GarcíaCumbreras, M. Á., Jiménez-Zafra, S. M., Valencia-García, R., Kumar Kumaresan, P., Ponnusamy, R., García-Baena, D. & García-Díaz, J. (2022). Overview of the Shared Task on Hope Speech Detection for Equality, Diversity, and Inclusion. In Proceedings of the Second Workshop on Language Technology for Equality, Diversity and Inclusion (pp. 378-388). https://aclanthology.org/2022.ltedi-1.58. [11] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, I.
Polosukhin, Attention is all you need, in: Proceedings of the 31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA, 2017, pp. 5998-6008.
URL: http://arxiv.org/abs/1706.03762. [12] Simple Transformers. URL: https://simpletransformers.ai. [13] Hugging Face. URL: https://huggingface.co. [14] J. Huertas-Tato, A. Martin, D. Camacho, BERTuit: Understanding Spanish language in
Twitter through a native transformer, 2022. URL: https://arxiv.org/abs/2204.03465. [15] Twitter Stream Grab. URL: https://archive.org/details/twitterstream. [16] DistilBERT. URL: https://huggingface.co/docs/transformers/model_doc/distilbert. [17] V. Sanh, L. Debut, J. Chaumond, T. Wolf. DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter, 2019. URL: https://arxiv.org/abs/1910.01108.