Hope, an exceptional human trait, empowers individuals to anticipate forthcoming events and potential outcomes with adaptability. These anticipations wield significant sway over emotions, behavior, and mood [1]. Those with high levels of hope demonstrate distinctive responses to obstacles in comparison to their low-hope counterparts; they perceive impediments as challenges to surmount and utilize their cognitive pathways to devise alternative strategies toward achieving their objectives [2,3]. Furthermore, elevated levels of hope have been linked to numerous beneficial outcomes such as academic achievement [4] and decreased levels of depression [5]. Conversely, low levels of hope are associated with adverse effects, including diminished well-being [6].
In the past two years, there has been a notable surge of interest in hope speech within the domain of Natural Language Processing (NLP). Shared initiatives focused on hope speech detection were convened at the second workshop on Language Technology for Equality, Diversity, and Inclusion (LT-EDI-2022), a part of ACL 2022 [7], LT-EDI-2023 at RANLP 2023 [8], and the HOPE shared task at IberLEF 2023 [9]. Recently, another collaborative endeavor has emerged [10], proposed at IberLEF 2024 [11], concentrating on exploring hope from two vantage points:
• Hope for equality, diversity, and inclusion [12].
• Hope as expectations [13].
The former perspective, previously investigated in the prior edition of IberLEF 2023, is further enriched in this new iteration by expanding and refining the Spanish dataset to encompass hope speech directed towards the LGTBI community. Conversely, the latter perspective remains unexplored in any prior shared task.
Detecting hope speech presents several challenges that need to be addressed to develop effective models. Firstly, there's a need to explore various machine learning approaches tailored for multilingual hope detection, considering the diverse linguistic nuances across different languages. Secondly, techniques must be devised to handle the imbalanced distribution of labels within datasets, ensuring fair representation of different hope expressions. Additionally, the noisy nature of social media data poses a challenge, requiring robust methods to filter out irrelevant information and identify genuine instances of hope speech. Moreover, extending hope detection beyond the domains in which systems are trained necessitates adapting models to recognize hope in diverse contexts. Linguistic disparities between English and Spanish texts further complicate the task, requiring specialized approaches to capture hope expressions effectively.
In order to overcome these challenges, we proposed a model based on custom BERT architectures [16] pretrained on multilingual datasets, specifically tailored for sentiment analysis. Our selection of transformers stems from their revolutionary impact on natural language processing tasks, notably through the introduction of the self-attention mechanism [17]. This mechanism empowers models to capture global dependencies and contextual relationships within sequences, making transformers highly effective in tasks such as sentiment analysis and, notably, hate speech detection [18]. Additionally, to mitigate the challenges posed by imbalanced data distribution, we adopted a strategy of selecting predictions based on higher logits relative to the less represented class. This approach significantly improved results, underscoring the importance of the challenges highlighted by the organizers of the shared task. For instance, while submissions in the HopeEDI dataset [12] using argmax predictions achieved a best average F1-score of 0.51, our latest submission achieved a score of 0.64, only three points below the top-ranked submission. The code to reproduce these results can be found in https://github.com/JesusASmx/ometeotl-Hope2024
The structure of this paper is as follows: in Section 2, we detail our methodology. In Section 3, we outline our experimental workflow. In Section 4, we discuss the results of our experiments. Finally, in Section 5, we conclude the paper.
Our main aim was to create a technique that could embed the hope short texts into a two-dimensional framework, delineating between Hope and Not Hope. This was accomplished by utilizing a loss function and a backpropagation algorithm to adjust these values towards the provided golden labels. As explained in the introduction, we employed custom BERT architectures pretrained on multilingual datasets tailored for sentiment analysis. Specifically, we leveraged the Huggingface model nlptown/bert-base-multilingual-uncased-sentiment [19].
We also employed k-fold cross-validation in order to ensure the robustness of our predictions. We stated 𝑘 = 5 and we splitted the training set into 5 parts. For each epoch, we trained the model 𝑘 times by taking as validation set each one of the 𝑘 parts. In all cases, the learning curves were reasonable fitted by considering the high complexity of the datasets. As a consequence, this election was correct.
For the three datasets [12,13], two were in Spanish and one in English. The HopeEDI dataset contains 2, 550 tweets, while the PolyHope in Spanish contains 28, 424 tweets. The PolyHope dataset in English contains 28, 424 tweets. See Table 3 Statistics of the datasets. The HopeEDI dataset was balanced, while both PolyHope datasets were not.
In the first phase of the contest, we experimented with several custom BERT models over the validation set. Since the model nlptown/bert-base-multilingual-uncased-sentiment presented the best results, we used it for the final phase of the contest. For all experiments, the employed hiperparameters were:
• Random Seed: 42.
• Folds for Cross Validation: 5.
• Epochs: 5.
• Batch Size: 32.
• Max. tokenizer length: 128.
• Token Padding system: Left.
• Padding Token: Eos Token.
• Label encoding:
-PolyHope: Not Hope = 0 and Hope = 1.
-HopEDI: NHS = 0 and HS = 1.
• Optimizer: Adam.
• Learning Rate: 5𝑒 − 5.
• Epsilon value: 1𝑒 − 8.
In terms of the average F1 score, we obtained 0.64 in the HopeEDI dataset, only 0.03 points under the first place. We obtained 0.84 in the PolyHope Spanish dataset, only 0.03 points under the first place. Finally, we obtained 0.82 in the PolyHope Spanish dataset, 0.05 points under the first place. The full leaderboards can be consulted in Table 2 for the HopeEDI dataset, Table 3 for the PolyHope Spanish dataset and Table 4 for the PolyHope English dataset.
All the issues warned by the organizers revealed to have a deep impact in any trained model, and our decisions to rely in multilingual models with cross-validation training demonstrated to be a strong proposition, by considering how close we were from the first place. Moreover, our custom prediction algorithm explained in the introduction to perform the predictions, allowed us to overcome the problem of data unbalance.
Macro F1 Weighted Macro F1 With respect of the learning curves depicted in Figures 1 and 2, both cases shows an overall good fit along with a adecuate decreasing rate. However, there are instances indicating issues with model performance. In the case of the HopeEDI dataset, underfitting is observed in epoch 1, fold 3, where the model fails to capture the underlying patterns in the training data, but overfitting is noted in epoch 5, fold 2, where the model may lacks of generalization capabilities with respect of the training data.
In the case of the PolyHope datasets, the english case presents underfitting in epoch 3, fold 4, and in epoch 5, fold 4. In these instances, the model fails to adequately learn from the training
Detecting hope speech presents several challenges that must be addressed to develop effective models capable of operating across diverse linguistic landscapes and social media multilingual contexts. Additionally, handling imbalanced label distributions within datasets is essential to ensure fair representation of diverse hope expressions. Furthermore, the noisy nature of social media data necessitates the development of robust methods to filter out irrelevant information and identify genuine instances of hope speech amidst the clutter. In this paper, custom BERT models were proposed to address these challenges, pretrained on multilingual datasets for sentiment analysis. We performed the training with 5-cross-validation and we adopted a strategy of selecting predictions based on higher logits relative to the less represented class to mitigate imbalanced data distribution issues.
Our experimental results, very close to the first place of each category, demonstrate the effectiveness of our proposed approach. Concretely, we achieved weighted macro F1-scores of 0.64 in the HopeEDI dataset (only 0.03 points behind the top place); 0.84 in the PolyHope Spanish dataset (only 0.05 points behind the top place) and 0.82 in the PolyHope Spanish dataset (only 0.05 points behind the top place).
Further work may consists in exploring more robust multilingual custom BERT models. For instance, this same nlptown/bert-base-multilingual-uncased-sentiment model can be pretrained again with more datasets in more languages in order to enhance its ability to generalize across different linguistic contexts. Concretely, it can be expanded with other spanish or latin-language datasets. Another aspect with room for enhancement is the manipulation of the datasets itself. For instance, it may be possible to explore data augmentation in order to evaluate its viability to enhance the results. Conversely, data reduction aiming the elimination of noise can be useful: Developing more advanced filtering algorithms to clean social media data will be crucial. Employing noise reduction strategies such as anomaly detection or clustering can help in isolating genuine instances of hope speech from irrelevant or misleading content.
By addressing these areas, future research can build on the foundation laid by this study to develop more robust, accurate, and generalizable models for detecting hope speech across diverse and noisy social media landscapes.