=Paper=
{{Paper
|id=Vol-3756/HOPE2024_paper12
|storemode=property
|title=Ometeotl at HOPE2024@IberLEF: Custom BERT Models for Hope Speech Detection
|pdfUrl=https://ceur-ws.org/Vol-3756/HOPE2024_paper12.pdf
|volume=Vol-3756
|authors=Jesús Armenta-Segura,Grigori Sidorov
|dblpUrl=https://dblp.org/rec/conf/sepln/Armenta-SeguraS24
}}
==Ometeotl at HOPE2024@IberLEF: Custom BERT Models for Hope Speech Detection==
https://ceur-ws.org/Vol-3756/HOPE2024_paper12.pdf
Ometeotl at HOPE2024@IberLEF: Custom BERT
Models for Hope Speech Detection
Jesús Armenta-Segura, Grigori Sidorov
Instituto Politécnico Nacional, Centro de Investigación en Computación (CIC IPN), Juan de Dios Bátiz Av., Gustavo A.
Madero, 07738 Ciudad de México, México.
Abstract
Hope speech has the potential to mitigate hostile environments and alleviate illnesses and depression,
making it crucial to detect automatically. In this paper, we present our submissions for the shared task
HOPE at IberLEF 2024. This shared task encompasses two datasets: HopeEDI, consisting of tweets in
Spanish, and PolyHope, which includes tweets in both English and Spanish. We proposed the use of
custom BERT models, specifically pretrained on multilingual datasets and tailored for sentiment analysis.
We achieved fourth, sixth and eighth place in the HopeEDI, PolyHope in Spanish, and PolyHope in
English datasets, respectively, with respect of the Averaged F1 score. The code to reproduce our results
can be found in https://github.com/JesusASmx/ometeotl-Hope2024
Keywords
Transformers, Hope Speech Detection, Natural Language Processing, CEUR-WS
1. Introduction
Hope, an exceptional human trait, empowers individuals to anticipate forthcoming events and
potential outcomes with adaptability. These anticipations wield significant sway over emotions,
behavior, and mood [1]. Those with high levels of hope demonstrate distinctive responses
to obstacles in comparison to their low-hope counterparts; they perceive impediments as
challenges to surmount and utilize their cognitive pathways to devise alternative strategies
toward achieving their objectives [2, 3]. Furthermore, elevated levels of hope have been linked
to numerous beneficial outcomes such as academic achievement [4] and decreased levels of
depression [5]. Conversely, low levels of hope are associated with adverse effects, including
diminished well-being [6].
In the past two years, there has been a notable surge of interest in hope speech within the
domain of Natural Language Processing (NLP). Shared initiatives focused on hope speech detec-
tion were convened at the second workshop on Language Technology for Equality, Diversity,
and Inclusion (LT-EDI-2022), a part of ACL 2022 [7], LT-EDI-2023 at RANLP 2023 [8], and the
HOPE shared task at IberLEF 2023 [9]. Recently, another collaborative endeavor has emerged
[10], proposed at IberLEF 2024 [11], concentrating on exploring hope from two vantage points:
IberLEF 2024, September 2024, Valladolid, Spain
$ jarmentas2022@cic.ipn.mx (J. Armenta-Segura); sidorov@cic.ipn.mx (G. Sidorov)
https://jesusasmx.github.io/ (J. Armenta-Segura); https://www.cic.ipn.mx/~sidorov (G. Sidorov)
0009-0003-8729-7096 (J. Armenta-Segura); 0000-0003-3901-3522 (G. Sidorov)
© 2024 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
CEUR
Workshop
Proceedings
http://ceur-ws.org
ISSN 1613-0073
CEUR Workshop Proceedings (CEUR-WS.org)
CEUR
ceur-ws.org
Workshop ISSN 1613-0073
Proceedings
• Hope for equality, diversity, and inclusion [12].
• Hope as expectations [13].
The former perspective, previously investigated in the prior edition of IberLEF 2023, is further
enriched in this new iteration by expanding and refining the Spanish dataset to encompass
hope speech directed towards the LGTBI community. Conversely, the latter perspective remains
unexplored in any prior shared task.
Detecting hope speech presents several challenges that need to be addressed to develop
effective models. Firstly, there’s a need to explore various machine learning approaches tailored
for multilingual hope detection, considering the diverse linguistic nuances across different
languages. Secondly, techniques must be devised to handle the imbalanced distribution of labels
within datasets, ensuring fair representation of different hope expressions. Additionally, the
noisy nature of social media data poses a challenge, requiring robust methods to filter out
irrelevant information and identify genuine instances of hope speech. Moreover, extending
hope detection beyond the domains in which systems are trained necessitates adapting models
to recognize hope in diverse contexts. Linguistic disparities between English and Spanish
texts further complicate the task, requiring specialized approaches to capture hope expressions
effectively.
In order to overcome these challenges, we proposed a model based on custom BERT
architectures [16] pretrained on multilingual datasets, specifically tailored for sentiment
analysis. Our selection of transformers stems from their revolutionary impact on natural
language processing tasks, notably through the introduction of the self-attention mechanism
[17]. This mechanism empowers models to capture global dependencies and contextual
relationships within sequences, making transformers highly effective in tasks such as sentiment
analysis and, notably, hate speech detection [18]. Additionally, to mitigate the challenges
posed by imbalanced data distribution, we adopted a strategy of selecting predictions based on
higher logits relative to the less represented class. This approach significantly improved results,
underscoring the importance of the challenges highlighted by the organizers of the shared
task. For instance, while submissions in the HopeEDI dataset [12] using argmax predictions
achieved a best average F1-score of 0.51, our latest submission achieved a score of 0.64, only
three points below the top-ranked submission. The code to reproduce these results can be
found in https://github.com/JesusASmx/ometeotl-Hope2024
The structure of this paper is as follows: in Section 2, we detail our methodology. In Section 3,
we outline our experimental workflow. In Section 4, we discuss the results of our experiments.
Finally, in Section 5, we conclude the paper.
2. Methodology
Our main aim was to create a technique that could embed the hope short texts into a
two-dimensional framework, delineating between Hope and Not Hope. This was accomplished
by utilizing a loss function and a backpropagation algorithm to adjust these values towards
the provided golden labels. As explained in the introduction, we employed custom BERT
architectures pretrained on multilingual datasets tailored for sentiment analysis. Specifically,
we leveraged the Huggingface model nlptown/bert-base-multilingual-uncased-sentiment [19].
We also employed k-fold cross-validation in order to ensure the robustness of our predictions.
We stated 𝑘 = 5 and we splitted the training set into 5 parts. For each epoch, we trained the
model 𝑘 times by taking as validation set each one of the 𝑘 parts. In all cases, the learning curves
were reasonable fitted by considering the high complexity of the datasets. As a consequence,
this election was correct.
3. Experimental Setup
3.1. Data
For the three datasets [12, 13], two were in Spanish and one in English. The HopeEDI dataset
contains 2, 550 tweets, while the PolyHope in Spanish contains 28, 424 tweets. The PolyHope
dataset in English contains 28, 424 tweets. See Table 3.1 for the full statistics of each datasets.
Language Split Hope Speech Not Hope Speech Total
Train data 700 700 1, 400
HopeEDI (Spanish) Validation data 100 100 200
Test data 200 200 400
Train data 2, 202 4, 701 6, 903
PolyHope (Spanish) Validation data 351 799 1, 150
Test data 379 773 1, 152
Train data 3, 088 3, 104 6, 192
PolyHope (English) Validation data 502 530 1, 032
Test data 491 541 1, 032
Table 1
Statistics of the datasets. The HopeEDI dataset was balanced, while both PolyHope datasets were not.
3.2. Experimental workflow
In the first phase of the contest, we experimented with several custom BERT models over the
validation set. Since the model nlptown/bert-base-multilingual-uncased-sentiment presented the
best results, we used it for the final phase of the contest. For all experiments, the employed
hiperparameters were:
• Random Seed: 42.
• Folds for Cross Validation: 5.
• Epochs: 5.
• Batch Size: 32.
• Max. tokenizer length: 128.
• Token Padding system: Left.
• Padding Token: Eos Token.
• Label encoding:
– PolyHope: Not Hope = 0 and Hope = 1.
– HopEDI: NHS = 0 and HS = 1.
• Optimizer: Adam.
• Learning Rate: 5𝑒 − 5.
• Epsilon value: 1𝑒 − 8.
Figures 1 and 2 depicts the evolution of the performance through the epochs, in the three
experiments.
Figure 1: Evolution of the experiments over the HopeEDI dataset through the 5 epochs. For each epoch,
𝑘 = 5 and it is determined with the gradient of the green. Underfitting is presented in epoch 1, fold 3
and in epoch 5, fold 2. The rest of the training had a reasonable fitting by considering the monotonicity
and the complexity of the data.
4. Results and Discussion
In terms of the average F1 score, we obtained 0.64 in the HopeEDI dataset, only 0.03 points
under the first place. We obtained 0.84 in the PolyHope Spanish dataset, only 0.03 points under
the first place. Finally, we obtained 0.82 in the PolyHope Spanish dataset, 0.05 points under the
first place. The full leaderboards can be consulted in Table 2 for the HopeEDI dataset, Table 3
for the PolyHope Spanish dataset and Table 4 for the PolyHope English dataset.
All the issues warned by the organizers revealed to have a deep impact in any trained model,
and our decisions to rely in multilingual models with cross-validation training demonstrated to
Figure 2: Evolution of the experiments over the PolyHope (EN, left) and PolyHope (ES, right) datasets
through the 5 epochs. For each epoch, 𝑘 = 5 and it is determined with the gradient of the green. For
the English dataset, underfitting is presented in epoch 3, fold 4 and in epoch 5, fold 4. The rest of the
training had a reasonable fitting by considering the monotonicity and the complexity of the data. For
the Spanish dataset, underfitting is presented in epoch 3, fold 2 and in epoch 4, fold 5. The rest of the
training had a reasonable fitting by considering the monotonicity and the complexity of the data.
be a strong proposition, by considering how close we were from the first place. Moreover, our
custom prediction algorithm explained in the introduction to perform the predictions, allowed
us to overcome the problem of data unbalance.
Pos. Team Macro F1 Weighted Macro F1
1 thindang 0.67 0.67
2 ChauPhamQuocHung 0.66 0.66
3 hongson04 0.65 0.65
4 Us 0.64 0.64
5 michaelibrahim 0.64 0.63
··· ··· ···
15 Arunraj_Subburaj 0.54 0.54
16 MIKHAIL 0.51 0.51
Table 2
Evaluation results for the HopeEDI dataset. We achieved fourth place out of 16 participants.
With respect of the learning curves depicted in Figures 1 and 2, both cases shows an overall
good fit along with a adecuate decreasing rate. However, there are instances indicating issues
with model performance. In the case of the HopeEDI dataset, underfitting is observed in epoch
1, fold 3, where the model fails to capture the underlying patterns in the training data, but
overfitting is noted in epoch 5, fold 2, where the model may lacks of generalization capabilities
with respect of the training data.
In the case of the PolyHope datasets, the english case presents underfitting in epoch 3, fold 4,
and in epoch 5, fold 4. In these instances, the model fails to adequately learn from the training
Pos. Team Macro F1 Weighted Macro F1
1 olp 0.85 0.87
2 hongson04 0.84 0.86
3 ChauPhamQuocHung 0.83 0.85
4 ronghao 0.83 0.85
5 KavyaG 0.82 0.84
6 Us 0.81 0.84
7 MIKHAIL 0.81 0.83
··· ··· ···
13 thindang 0.73 0.76
14 Fida 0.71 0.74
Table 3
Evaluation results for the PolyHope Spanish dataset. We achieved sixth place out of 14 participants.
Pos. Team Macro F1 Weighted Macro F1
1 hongson04 0.87 0.87
2 olp 0.86 0.86
3 ronghao 0.86 0.86
4 ChauPhamQuocHung 0.85 0.85
5 MIKHAIL 0.85 0.85
6 zahraahani 0.85 0.85
7 hamadanayel 0.83 0.83
8 Us 0.82 0.82
9 KavyaG 0.82 0.81
··· ··· ···
16 AmnaNaseeb 0.23 0.31
17 JuanCalderon 0.21 0.21
Table 4
Evaluation results for the PolyHope English dataset. We achieved eigth place out of 17 participants.
data, resulting in high training and validation errors, indicating that the model is too simplistic
and does not capture the underlying patterns in the data. For the spanish case, underfitting is
evident in epoch 3, fold 2, and in epoch 4, fold 5.
5. Conclusions and Further Work
Detecting hope speech presents several challenges that must be addressed to develop effective
models capable of operating across diverse linguistic landscapes and social media multilingual
contexts. Additionally, handling imbalanced label distributions within datasets is essential to
ensure fair representation of diverse hope expressions. Furthermore, the noisy nature of social
media data necessitates the development of robust methods to filter out irrelevant information
and identify genuine instances of hope speech amidst the clutter.
In this paper, custom BERT models were proposed to address these challenges, pretrained on
multilingual datasets for sentiment analysis. We performed the training with 5-cross-validation
and we adopted a strategy of selecting predictions based on higher logits relative to the less
represented class to mitigate imbalanced data distribution issues.
Our experimental results, very close to the first place of each category, demonstrate the
effectiveness of our proposed approach. Concretely, we achieved weighted macro F1-scores
of 0.64 in the HopeEDI dataset (only 0.03 points behind the top place); 0.84 in the PolyHope
Spanish dataset (only 0.05 points behind the top place) and 0.82 in the PolyHope Spanish
dataset (only 0.05 points behind the top place).
Further work may consists in exploring more robust multilingual custom BERT models. For
instance, this same nlptown/bert-base-multilingual-uncased-sentiment model can be pretrained
again with more datasets in more languages in order to enhance its ability to generalize
across different linguistic contexts. Concretely, it can be expanded with other spanish or
latin-language datasets. Another aspect with room for enhancement is the manipulation of
the datasets itself. For instance, it may be possible to explore data augmentation in order to
evaluate its viability to enhance the results. Conversely, data reduction aiming the elimination
of noise can be useful: Developing more advanced filtering algorithms to clean social media
data will be crucial. Employing noise reduction strategies such as anomaly detection or cluster-
ing can help in isolating genuine instances of hope speech from irrelevant or misleading content.
By addressing these areas, future research can build on the foundation laid by this study
to develop more robust, accurate, and generalizable models for detecting hope speech across
diverse and noisy social media landscapes.
References
[1] P. Bruininks, B. F. Malle, Distinguishing hope from optimism and related affective states,
Motivation and Emotion 29 (2005) 324–352.
[2] C. R. Snyder, The Psychology of Hope: You Can Get There from Here, Simon and Schuster,
1994.
[3] C. R. Snyder, Hypothesis: There is hope, in: Handbook of Hope, Elsevier, 2000, pp. 3–21.
[4] C. R. Snyder, Hope theory: Rainbows in the mind, Psychological Inquiry 13 (2002) 249–275.
[5] C. R. Snyder, B. Hoza, W. E. Pelham, M. Rapoff, L. Ware, M. Danovsky, L. Highberger,
H. Ribinstein, K. J. Stahl, The development and validation of the children’s hope scale,
Journal of Pediatric Psychology 22 (1997) 399–421.
[6] E. Diener, Subjective well-being, in: The Science of Well-Being, 2009, pp. 11–58.
[7] B. R. Chakravarthi, V. Muralidaran, R. Priyadharshini, S. C. Navaneethakrishnan, J. P.
McCrae, M. Á. García-Cumbreras, S. M. Jiménez-Zafra, R. Valencia-García, P. K. Kumaresan,
R. Ponnusamy, D. García-Baena, J. A. García-Díaz, Overview of the shared task on hope
speech detection for equality, diversity, and inclusion, in: Association for Computational
Linguistics, 2022, pp. 378–388.
[8] P. Kumaresan, B. R. Chakravarthi, S. Cn, M. Á. García, S. M. Jiménez-Zafra, J. A. García-Díaz,
R. Valencia-García, M. Hardalov, I. Koychev, P. Nakov, D. García-Baena, K. K. Ponnusamy,
B. Preston, Overview of the shared task on hope speech detection for equality, diversity,
and inclusion, in: Proceedings of the Third Workshop on Language Technology for
Equality, Diversity and Inclusion, Association for Computational Linguistics, 2023, pp.
47–53.
[9] S. M. Jiménez-Zafra, M. Á. Garcia-Cumbreras, D. García-Baena, J. A. Garcia-Díaz, B. R.
Chakravarthi, R. Valencia-García, L. A. Ureña-López, Overview of hope at iberlef 2023:
Multilingual hope speech detection, Procesamiento del Lenguaje Natural 71 (2023) 371–381.
[10] D. García-Baena, F. Balouchzahi, S. Butt, M. Á. García-Cumbreras, A. Lambebo Tonja,
J. A. García-Díaz, S. Bozkurt, B. R. Chakravarthi, H. G. Ceballos, V.-G. Rafael, G. Sidorov,
L. A. Ureña-López, A. Gelbukh, S. M. Jiménez-Zafra, Overview of HOPE at IberLEF 2024:
Approaching Hope Speech Detection in Social Media from Two Perspectives, for Equality,
Diversity and Inclusion and as Expectations, Procesamiento del Lenguaje Natural 73
(2024).
[11] L. Chiruzzo, S. M. Jiménez-Zafra, F. Rangel, Overview of IberLEF 2024: Natural Language
Processing Challenges for Spanish and other Iberian Languages, in: Proceedings of the
Iberian Languages Evaluation Forum (IberLEF 2024), co-located with the 40th Conference
of the Spanish Society for Natural Language Processing (SEPLN 2024), CEUR-WS.org,
2024.
[12] D. García-Baena, M. Á. García-Cumbreras, S. M. Jiménez-Zafra, J. A. García-Díaz,
R. Valencia-García, Hope speech detection in spanish: The lgbt case, Language Resources
and Evaluation 57 (2023) 1487–1514.
[13] F. Balouchzahi, G. Sidorov, A. Gelbukh, Polyhope: Two-level hope speech detection from
tweets, Expert Systems with Applications 225 (2023) 120078. doi:10.1016/j.eswa.2023.
120078.
[14] J. Armenta-Segura, G. Sidorov, Anime Success Prediction Based on Synopsis Using Tradi-
tional Classifiers, in: Proceedings of Congreso Mexicano de Inteligencia Artificial, COMIA,
2023.
[15] J. Armenta-Segura, G. Sidorov, Anime popularity prediction before huge investments: a
multimodal approach using deep learning, 2024. URL: https://arxiv.org/abs/2406.16961.
arXiv:2406.16961.
[16] J. Devlin, M.-W. Chang, K. Lee, K. Toutanova, Bert: Pre-training of deep bidirectional
transformers for language understanding, arXiv preprint arXiv:1810.04805 (2018).
[17] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, I. Polo-
sukhin, Attention is all you need, Advances in neural information processing systems 30
(2017).
[18] J. Armenta-Segura, C. J. Núñez-Prado, G. O. Sidorov, A. Gelbukh, R. F. Román-Godínez,
Ometeotl@multimodal hate speech event detection 2023: Hate speech and text-image
correlation detection in real life memes using pre-trained BERT models over text, in:
A. Hürriyetoğlu, H. Tanev, V. Zavarella, R. Yeniterzi, E. Yörük, M. Slavcheva (Eds.), Pro-
ceedings of the 6th Workshop on Challenges and Applications of Automated Extraction of
Socio-political Events from Text, INCOMA Ltd., Shoumen, Bulgaria, Varna, Bulgaria, 2023,
pp. 53–59. URL: https://aclanthology.org/2023.case-1.7.
[19] Nlptown bert model for multilingual sentiment analysis, Available at:, .
https://huggingface.co/nlptown/bert-base-multilingual-uncased-sentiment.