Multiclass Hope Speech Detection Through Transformer Methods⋆ Zahra Ahani, Moein Shahiki Tash , Majid Tash, Alexander Gelbukh and Irna Gelbukh Instituto Politécnico Nacional (IPN), Centro de Investigación en Computación (CIC), Mexico city, Mexico Abstract Hope includes the belief and expectations that we have for the realization of desired events. It is a hopeful prospect or expectation that positive circumstances will unfold or conditions will improve. These emotions are a core human emotion and mindset that drives people to persevere in the face of obstacles, pursue their ambitions, and believe in the potential for better outcomes even in the midst of challenges. Through attention to binary classification in hope research, we can categorize hopeful states as either "Hope" or "Not Hope." Additionally, the "Hope" classification is segmented into three specific types: "Generalized Hope," "Realistic Hope," and "Unrealistic Hope. Our impressive outcomes, driven by analysis and training data, were achieved through transformer methods showcased in the HOPE track of the IberLEF 2024 competition. Our proposed method achieved very competitive results in all subtasks, however, the best-performing result secured an average macro F1 score of 0.85 in the binary hope speech detection subtask in the English language. Keywords Hope, Linguistic, Psycholinguistic, Transformer 1. Introduction The increase in social media platform users has significantly enhanced information sharing, allowing instant access to the latest updates with just a click. These platforms are used not only for social interaction but also for entertainment and information retrieval [1]. The surge in social media usage has revolutionized not only how people interact but also how they consume and disseminate information. Consequently, researchers have increasingly focused on analyzing social media comments. One particularly important area of study is the detection and analysis of hope speech. Hope can be described as a receptive attitude toward what lies ahead, encompassing desires, expectations, and aspirations for certain outcomes. It significantly influences the human psyche, emotions, actions, and choices. Hope is commonly linked with notions of anticipated aspirations and potential outcomes in the future [2]. It serves as a source of comfort, motivation, and perseverance, guiding people through periods of uncertainty and hardship. It is a powerful motivator that encourages individuals to continue striving for a brighter future, even when IberLEF 2024, September 2024, Valladolid, Spain $ z.ahani2023@cic.ipn.mx (Z. Ahani); mshahikit2022@cic.ipn.mx (M. S. T. ); ziatash@yahoo.com (M. Tash); gelbukh@cic.ipn.mx (A. Gelbukh); i.gelbukh@nlp.cic.ipn.mx (I. Gelbukh)  0000-0002-1307-1647 (M. Tash); 0000-0002-1307-1647 (A. Gelbukh) © 2024 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). CEUR Workshop Proceedings http://ceur-ws.org ISSN 1613-0073 CEUR Workshop Proceedings (CEUR-WS.org) CEUR ceur-ws.org Workshop ISSN 1613-0073 Proceedings faced with adversity. However, crafting a sufficiently precise definition of hope that facilitates systematic examination presents a challenge. In everyday language, hope can carry varied implications; for instance, "I hope it will be sunny tomorrow" differs from "I hope to finish my work by tomorrow." Similarly, for individuals facing serious illnesses, such as cancer patients, "hoping that my cancer will not return" holds distinct significance compared to "hoping to see my children before I die." Given the diverse nuances associated with the concept of hope, it is crucial to delineate the term with care and precision [3]. As a result, hope speech has prompted many researchers to investigate its presence in social media. Chakravati’s seminal research highlights the importance of this discourse, thereby sustaining ongoing debates around hope [4]. Online social platforms have a significant impact on human interaction and create an environment where people can express their opinions freely. The distinctive features of social media, including rapid dissemination, affordability, accessibility, and anonymity, have fueled their widespread adoption. These platforms not only facilitate global communication but also serve as repositories of vast amounts of data, which are invaluable for analyzing human behavior and sentiment. Consequently, social media has emerged as a pivotal arena for investigating natural language processing (NLP) issues [5, 6]. The next section provides an overview of the methods employed in collaborative efforts over the years. Section 3 provides an overview of the dataset statistics and outlines the methodology employed to achieve the reported results. Section 4 delves into the findings of the research, while Section 5 offers a conclusion. 2. Related work When a person hopes for a certain result, it shows that he has a goal to achieve it. During this journey, factors such as directional thinking and agentic thinking play pivotal roles [3], so it is important to address the issue of "hope". Then we go to a series of research in this field. In an exploration of hope Palakodety et al. [7] demonstrated the relevance of hope in wartime contexts based on analysis of YouTube comments, spanning both Hindi and English languages, and presented in both Devanagari and Roman scripts. Employing logistic regression with l2 regularization, an 80/10 train-test split, N-grams (1, 3), sentiment score, and 100-dimensional polyglot FastText embeddings as features, they achieved an F-1 score of 78.51 (±2.24%). Chakravarthi et al. [8] provides an overview of collaborative research efforts focused on hope speech recognition in various languages including Tamil, Malayalam, Kannada, English and Spanish with 14 teams. The research is based on the HopeEDI dataset provided by [8], including an extensive collection of 63,883 social media comments. Notably, the ARGUABLY team achieved the highest performance in speech recognition for Hope, achieving a W-F1 score of 0.810. In their study presented at IberLEF 2023 Shahiki-Tash et al. [10] explored Multilingual Hope Speech Recognition. They investigated two specific tasks: identifying hope speech in Spanish tweets and English YouTube comments. The research introduced a word-based tokenization technique for training convolutional neural networks (CNNs). Leveraging CNNs from prior research in speech recognition, they anticipated favorable outcomes with this approach. Notably, this strategy achieved fourth place in both subtasks, with an average F1 macro rank of 0.72 for Spanish data and 0.49 for English data. Ahani et al. [11] delineates results from a joint endeavor in multilingual hope speech recog- nition, with the objective of classifying texts into hope and non-hope categories. The research covered two datasets: one in English and another in Spanish. Utilizing the SVM algorithm for English data and the KNN algorithm for Spanish data, the research secured third place. Specifically, the SVM-based approach yielded an F1 score of 0.49, while the KNN-based approach achieved an F1 score of 0.74. These outcomes underscore the efficacy of SVM and KNN algo- rithms in this context, emphasizing the significance of algorithm selection tailored to different languages. Balouchzahi et al. [12] introduces a two-level dataset for Hope’s work, consisting of more than 100,000 English tweets covering topics such as women’s abortion rights, black rights, religion, and politics. After annotation, the dataset The final ones consisted of 8,256 tweets and 4,175 "hope" tweets, and 4,081 "no hope" tweets were further categorized into generalized hope, realistic hope, and unrealistic hope. This study uses different machine learning models [13], including single-gram word TF-IDF, CNN [14], BiLSTM trained with GloVe and FastText em- beddings, and several transformations. Notably, these models together with simple n-grams have shown strong performance in binary hope speech. By referring to Table 1, the detailed results of applying the models of this article are presented. Table 1 Best performing models in each learning approach. Model Learning approach Averaged-weighted F1 Averaged-macro F1 Category BERT, RoBERTa, and XLNet Transformers 0.85 0.85 Binary hope speech detection BERT Transformers 0.77 0.72 Multiclass hope speech detection 3. Methodology In this research, we explored the efficiency of two transformer models, namely ’albert-base-v2’ and ’bert-base-multilingual-cased’, in handling text classification assignments in English and Spanish correspondingly. Our primary objective was to devise a technique capable of identifying hope-related attributes within texts and discerning between instances of "Hope" and "Not hope", subsequently categorizing them into three refined hope classifications: "generalized hope", "realistic hope", and "unrealistic hope". For further elucidation on this process, we can refer to flowchart 2 to enhance our comprehension of the task. 3.1. Task Description The proposed collaborative task encompasses two objectives aimed at uncovering hope within social media texts. Task 1: Targeting Spanish tweets, the aim is to discern instances of hopeful discourse, partic- ularly emphasizing themes of equality, diversity, and inclusion. Subtasks involve identifying hopeful discourse within the LGTBI domain and in unidentified domains [15, 16]. Task 2: This task centers on identifying hope speech associated with expectations within both English and Spanish texts. Subtasks include binary and multi-class predictive speech Figure 1: The framework of hope recognition. The outcomes of this study are pertinent to Task 2 [17]. 3.2. Dataset The dataset includes tweet comments in both English and Spanish. Table 2 provides statistics for the training data showing the distribution between two binary classes and multiple classes in the dataset. In addition, Table 3 displays the test dataset, comprising 1032 tweets in English and 1152 tweets in Spanish [18]. Table 2 Data metrics pertaining to the training set Category English Spanish Binary-Train Hope 3634 2553 Not Hope 3590 5500 Multiclass-Train Not Hope 3509 5500 Generalized Hope 2026 1337 Realistic Hope 858 579 Unrealistic Hope 750 637 3.3. Model Construction The hope speech task was addressed using the Simple Transformers library, with the ’albert-base- v2’ model utilized for English binary classification and the ’bert-base-multilingual-cased’ model for Spanish binary classification. Initially, the training and evaluation datasets were prepared, comprising text samples paired with corresponding labels indicating hope or non-hope speech. Table 3 Data metrics pertaining to the test set Category English Spanish Binary-Tast Hope 541 379 Not Hope 491 773 Multiclass-Test Not Hope 491 773 Generalized Hope 309 206 Realistic Hope 124 77 Unrealistic Hope 108 96 Subsequently, an instance of a Classification Model from the Simple Transformers library was instantiated, with the desired model architecture and training configuration parameters specified. The model was then trained on the training dataset and evaluated for its performance on the evaluation dataset. Furthermore, predictions were made on new text samples using the trained model to identify instances of hope speech. Throughout the process, an emphasis was placed on leveraging the capabilities of the Simple Transformers library to streamline the implementation of the hope speech detection task. At the second level, we also utilized the Simple Transformers library, employing the ’albert- base-v2’ model for the English multiclass task and the ’bert-base-multilingual-cased’ model for the Spanish multiclass task in hope speech classification. The training and evaluation datasets were meticulously curated, with each example paired with corresponding labels indicating categories like "generalized hope," "realistic hope," and "unrealistic hope." We instantiated a ClassificationModel with customized parameters to facilitate the training process. Following this, the model underwent training on the prepared training dataset and evaluation on a separate dataset to gauge its performance. Finally, the trained model was employed to make predictions on unseen data, showcasing its ability to classify hope speech across various languages and categories. The parameters for both ALBERT (albert-base-v2) and BERT (bert-base-multilingual- cased) are selected based on standard NLP practices to achieve a balance between performance and computational efficiency. These parameters are presented in Table 3. Table 4 Hyperparameters for ALBERT (albert-base-v2) and BERT (bert-base-multilingual-cased) Hyperparameter ALBERT (albert-base-v2) BERT (bert-base-multilingual-cased) Attention Heads 12 12 Hidden Size 768 768 Feedforward Size 3072 3072 Dropout Rate 0.1 0.1 Learning Rate 2e-5 2e-5 Batch Size 32 32 Epochs 15 15 4. Results In this study, the ’albert-base-v2’ and ’bert-base-multilingual-cased’ models were employed for analyzing English and Spanish, respectively. Table 4 displays the computational outcomes, showcasing various tasks assessed at different stages, with measures including precision (Pr), recall (Re), and F1 score (F1) for both micro (M_) and weighted (W_) averages alongside precision (acc). The results in Table 4 demonstrate the exceptional performance of the ’lbert-base-v2’ model, achieving an M_F1 score of 0.85 for English binary data and 0.79 for Spanish binary data using the ’bert-base-multilingual-cased’ model. Additionally, among 17 participants, we secured the 3rd position in the PolyHope Binary (English) category and ranked 3rd out of 13 participants in the PolyHope Multiclass (English) category. Moreover, we achieved the 7th position in the PolyHope Binary (Spanish) category out of 14 teams and the 8th position in the PolyHope Multiclass (Spanish) category out of 11 teams. Table 5 Results of PolyHope Model Performance for Binary and Multiclass Classification in English and Spanish Tasks M_Pr M_Re M_F1 W_Pr W_Re W_F1 acc acc PolyHope Binary (English) 0.85 0.85 0.85 0.85 0.85 0.85 0.85 PolyHope Multiclass (English) 0.67 0.68 0.67 0.74 0.74 0.74 0.74 PolyHope Binary (Spanish) 0.82 0.77 0.79 0.82 0.83 0.82 0.83 PolyHope Multiclass (Spanish) 0.51 0.44 0.44 0.67 0.69 0.67 0.69 5. Conclusion The exploration of hope speech within social media comments in both English and Spanish through transformer-based models has yielded significant insights and competitive results. Utilizing the ’albert-base-v2’ model for English and the ’bert-base-multilingual-cased’ model for Spanish, we were able to achieve commendable performance in both binary and multiclass classification tasks. The binary classification tasks demonstrated strong performance, with M_F1 scores of 0.85 for English and 0.79 for Spanish. The multiclass classification tasks, although more challenging, also showed reasonable performance, with M_F1 scores of 0.67 for English and 0.44 for Spanish. Our method’s success in the HOPE track of the IberLEF 2024 competition, securing 3rd place in both PolyHope Binary (English) and PolyHope Multiclass (English) categories, underscores the effectiveness of transformer models in capturing and categorizing hope-related attributes within text. 6. Acknowledgements The work was done with partial support from the Mexican Government through the grant A1- S-47854 of CONACYT, Mexico, grants 20241816, 20241819, and 20240951 of the Secretaría de Investigación y Posgrado of the Instituto Politécnico Nacional, Mexico. The authors thank the CONACYT for the computing resources brought to them through the Plataforma de Aprendizaje Profundo para Tecnologías del Lenguaje of the Laboratorio de Supercómputo of the INAOE, Mexico and acknowledge the support of Microsoft through the Microsoft Latin America PhD Award References [1] M. Tash, Z. Ahani, M. Zamir, O. Kolesnikova, G. Sidorov, Lidoma@ lt-edi 2024: Tamil hate speech detection in migration discourse, in: Proceedings of the Fourth Workshop on Language Technology for Equality, Diversity, Inclusion, 2024, pp. 184–189. [2] M. Arif, M. S. Tash, A. Jamshidi, I. Ameer, F. Ullah, J. Kalita, A. Gelbukh, F. Balouchzahi, Exploring multidimensional aspects of hope speech computationally: A psycholinguistic and emotional perspective (2024). [3] B. W. Corn, D. B. Feldman, I. Wexler, The science of hope, The Lancet Oncology 21 (2020) e452–e459. [4] B. R. Chakravarthi, Hope speech detection in youtube comments, Social Network Analysis and Mining 12 (2022) 75. [5] A. L. Tonja, M. G. Yigezu, O. Kolesnikova, M. S. Tash, G. Sidorov, A. Gelbuk, Transformer- based model for word level language identification in code-mixed kannada-english texts, arXiv preprint arXiv:2211.14459 (2022). [6] M. G. Yigezu, A. L. Tonja, O. Kolesnikova, M. S. Tash, G. Sidorov, A. Gelbukh, Word level language identification in code-mixed kannada-english texts using deep learning approach, in: Proceedings of the 19th International Conference on Natural Language Processing (ICON): Shared Task on Word Level Language Identification in Code-mixed Kannada-English Texts, 2022, pp. 29–33. [7] S. Palakodety, A. R. KhudaBukhsh, J. G. Carbonell, Hope speech detection: A computational analysis of the voice of peace, arXiv preprint arXiv:1909.12940 (2019). [8] B. R. Chakravarthi, V. Muralidaran, R. Priyadharshini, S. Cn, J. P. McCrae, M. Á. García, S. M. Jiménez-Zafra, R. Valencia-García, P. Kumaresan, R. Ponnusamy, et al., Overview of the shared task on hope speech detection for equality, diversity, and inclusion, in: Proceedings of the second workshop on language technology for equality, diversity and inclusion, 2022, pp. 378–388. [9] M. Shahiki-Tash, J. Armenta-Segura, O. Kolesnikova, G. Sidorov, A. Gelbukh, Lidoma at hope2023iberlef: Hope speech detection using lexical features and convolutional neural networks, in: Proceedings of the Iberian Languages Evaluation Forum (IberLEF 2023), co- located with the 39th Conference of the Spanish Society for Natural Language Processing (SEPLN 2023), CEUR-WS. org, 2023. [10] M. Shahiki-Tash, J. Armenta-Segura, Z. Ahani, O. Kolesnikova, G. Sidorov, A. Gelbukh, Lidoma at homomex2023@ iberlef: Hate speech detection towards the mexican spanish- speaking lgbt+ population. the importance of preprocessing before using bert-based models, in: Proceedings of the Iberian Languages Evaluation Forum (IberLEF 2023), 2023. [11] Z. Ahani, G. Sidorov, O. Kolesnikova, A. Gelbukh, Zavira at hope2023@ iberlef: Hope speech detection from text using tf-idf features and machine learning algorithms (2023). [12] F. Balouchzahi, G. Sidorov, A. Gelbukh, Polyhope: Two-level hope speech detection from tweets, Expert Systems with Applications 225 (2023) 120078. [13] M. S. Tash, Z. Ahani, A. Tonja, M. Gemeda, N. Hussain, O. Kolesnikova, Word level language identification in code-mixed kannada-english texts using traditional machine learning algorithms, in: Proceedings of the 19th International Conference on Natural Language Processing (ICON): Shared Task on Word Level Language Identification in Code-mixed Kannada-English Texts, 2022, pp. 25–28. [14] Z. Ahani, M. Shahiki Tash, Y. Ledo Mezquita, J. Angel, Utilizing deep learning models for the identification of enhancers and super-enhancers based on genomic and epigenomic features, Journal of Intelligent & Fuzzy Systems (????) 1–11. [15] D. García-Baena, M. Á. García-Cumbreras, S. M. Jiménez-Zafra, J. A. García-Díaz, R. Valencia-García, Hope speech detection in spanish: The lgbt case, Language Resources and Evaluation 57 (2023) 1487–1514. [16] L. Chiruzzo, S. M. Jiménez-Zafra, F. Rangel, Overview of IberLEF 2024: Natural Language Processing Challenges for Spanish and other Iberian Languages, in: Proceedings of the Iberian Languages Evaluation Forum (IberLEF 2024), co-located with the 40th Conference of the Spanish Society for Natural Language Processing (SEPLN 2024), CEUR-WS.org, 2024. [17] F. Balouchzahi, G. Sidorov, A. Gelbukh, Polyhope: Two-level hope speech detection from tweets, Expert Systems with Applications 225 (2023) 120078. doi:10.1016/j.eswa.2023. 120078. [18] D. García-Baena, F. Balouchzahi, S. Butt, M. Á. García-Cumbreras, A. Lambebo Tonja, J. A. García-Díaz, S. Bozkurt, B. R. Chakravarthi, H. G. Ceballos, V.-G. Rafael, G. Sidorov, L. A. Ureña-López, A. Gelbukh, S. M. Jiménez-Zafra, Overview of HOPE at IberLEF 2024: Approaching Hope Speech Detection in Social Media from Two Perspectives, for Equality, Diversity and Inclusion and as Expectations, Procesamiento del Lenguaje Natural 73 (2024).