=Paper=
{{Paper
|id=Vol-3756/HOPE2024_paper11
|storemode=property
|title=Multiclass Hope Speech Detection Through Transformer Methods
|pdfUrl=https://ceur-ws.org/Vol-3756/HOPE2024_paper11.pdf
|volume=Vol-3756
|authors=Zahra Ahani,Moein Shahiki Tash,Majid Tash,Alexander Gelbukh,Irna Gelbukh
|dblpUrl=https://dblp.org/rec/conf/sepln/AhaniTTGG24
}}
==Multiclass Hope Speech Detection Through Transformer Methods==
https://ceur-ws.org/Vol-3756/HOPE2024_paper11.pdf
Multiclass Hope Speech Detection Through
Transformer Methods⋆
Zahra Ahani, Moein Shahiki Tash , Majid Tash, Alexander Gelbukh and
Irna Gelbukh
Instituto Politécnico Nacional (IPN), Centro de Investigación en Computación (CIC), Mexico city, Mexico
Abstract
Hope includes the belief and expectations that we have for the realization of desired events. It is a
hopeful prospect or expectation that positive circumstances will unfold or conditions will improve.
These emotions are a core human emotion and mindset that drives people to persevere in the face of
obstacles, pursue their ambitions, and believe in the potential for better outcomes even in the midst of
challenges. Through attention to binary classification in hope research, we can categorize hopeful states
as either "Hope" or "Not Hope." Additionally, the "Hope" classification is segmented into three specific
types: "Generalized Hope," "Realistic Hope," and "Unrealistic Hope. Our impressive outcomes, driven by
analysis and training data, were achieved through transformer methods showcased in the HOPE track of
the IberLEF 2024 competition. Our proposed method achieved very competitive results in all subtasks,
however, the best-performing result secured an average macro F1 score of 0.85 in the binary hope speech
detection subtask in the English language.
Keywords
Hope, Linguistic, Psycholinguistic, Transformer
1. Introduction
The increase in social media platform users has significantly enhanced information sharing,
allowing instant access to the latest updates with just a click. These platforms are used not
only for social interaction but also for entertainment and information retrieval [1]. The surge in
social media usage has revolutionized not only how people interact but also how they consume
and disseminate information. Consequently, researchers have increasingly focused on analyzing
social media comments. One particularly important area of study is the detection and analysis
of hope speech.
Hope can be described as a receptive attitude toward what lies ahead, encompassing desires,
expectations, and aspirations for certain outcomes. It significantly influences the human psyche,
emotions, actions, and choices. Hope is commonly linked with notions of anticipated aspirations
and potential outcomes in the future [2]. It serves as a source of comfort, motivation, and
perseverance, guiding people through periods of uncertainty and hardship. It is a powerful
motivator that encourages individuals to continue striving for a brighter future, even when
IberLEF 2024, September 2024, Valladolid, Spain
$ z.ahani2023@cic.ipn.mx (Z. Ahani); mshahikit2022@cic.ipn.mx (M. S. T. ); ziatash@yahoo.com (M. Tash);
gelbukh@cic.ipn.mx (A. Gelbukh); i.gelbukh@nlp.cic.ipn.mx (I. Gelbukh)
0000-0002-1307-1647 (M. Tash); 0000-0002-1307-1647 (A. Gelbukh)
© 2024 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
CEUR
Workshop
Proceedings
http://ceur-ws.org
ISSN 1613-0073
CEUR Workshop Proceedings (CEUR-WS.org)
CEUR
ceur-ws.org
Workshop ISSN 1613-0073
Proceedings
faced with adversity. However, crafting a sufficiently precise definition of hope that facilitates
systematic examination presents a challenge. In everyday language, hope can carry varied
implications; for instance, "I hope it will be sunny tomorrow" differs from "I hope to finish my
work by tomorrow." Similarly, for individuals facing serious illnesses, such as cancer patients,
"hoping that my cancer will not return" holds distinct significance compared to "hoping to see
my children before I die." Given the diverse nuances associated with the concept of hope, it is
crucial to delineate the term with care and precision [3].
As a result, hope speech has prompted many researchers to investigate its presence in social
media. Chakravati’s seminal research highlights the importance of this discourse, thereby
sustaining ongoing debates around hope [4]. Online social platforms have a significant impact
on human interaction and create an environment where people can express their opinions
freely. The distinctive features of social media, including rapid dissemination, affordability,
accessibility, and anonymity, have fueled their widespread adoption. These platforms not only
facilitate global communication but also serve as repositories of vast amounts of data, which
are invaluable for analyzing human behavior and sentiment. Consequently, social media has
emerged as a pivotal arena for investigating natural language processing (NLP) issues [5, 6].
The next section provides an overview of the methods employed in collaborative efforts over
the years. Section 3 provides an overview of the dataset statistics and outlines the methodology
employed to achieve the reported results. Section 4 delves into the findings of the research,
while Section 5 offers a conclusion.
2. Related work
When a person hopes for a certain result, it shows that he has a goal to achieve it. During this
journey, factors such as directional thinking and agentic thinking play pivotal roles [3], so it is
important to address the issue of "hope". Then we go to a series of research in this field.
In an exploration of hope Palakodety et al. [7] demonstrated the relevance of hope in wartime
contexts based on analysis of YouTube comments, spanning both Hindi and English languages,
and presented in both Devanagari and Roman scripts. Employing logistic regression with l2
regularization, an 80/10 train-test split, N-grams (1, 3), sentiment score, and 100-dimensional
polyglot FastText embeddings as features, they achieved an F-1 score of 78.51 (±2.24%).
Chakravarthi et al. [8] provides an overview of collaborative research efforts focused on
hope speech recognition in various languages including Tamil, Malayalam, Kannada, English
and Spanish with 14 teams. The research is based on the HopeEDI dataset provided by [8],
including an extensive collection of 63,883 social media comments. Notably, the ARGUABLY
team achieved the highest performance in speech recognition for Hope, achieving a W-F1 score
of 0.810.
In their study presented at IberLEF 2023 Shahiki-Tash et al. [10] explored Multilingual Hope
Speech Recognition. They investigated two specific tasks: identifying hope speech in Spanish
tweets and English YouTube comments. The research introduced a word-based tokenization
technique for training convolutional neural networks (CNNs). Leveraging CNNs from prior
research in speech recognition, they anticipated favorable outcomes with this approach. Notably,
this strategy achieved fourth place in both subtasks, with an average F1 macro rank of 0.72 for
Spanish data and 0.49 for English data.
Ahani et al. [11] delineates results from a joint endeavor in multilingual hope speech recog-
nition, with the objective of classifying texts into hope and non-hope categories. The research
covered two datasets: one in English and another in Spanish. Utilizing the SVM algorithm
for English data and the KNN algorithm for Spanish data, the research secured third place.
Specifically, the SVM-based approach yielded an F1 score of 0.49, while the KNN-based approach
achieved an F1 score of 0.74. These outcomes underscore the efficacy of SVM and KNN algo-
rithms in this context, emphasizing the significance of algorithm selection tailored to different
languages.
Balouchzahi et al. [12] introduces a two-level dataset for Hope’s work, consisting of more
than 100,000 English tweets covering topics such as women’s abortion rights, black rights,
religion, and politics. After annotation, the dataset The final ones consisted of 8,256 tweets and
4,175 "hope" tweets, and 4,081 "no hope" tweets were further categorized into generalized hope,
realistic hope, and unrealistic hope. This study uses different machine learning models [13],
including single-gram word TF-IDF, CNN [14], BiLSTM trained with GloVe and FastText em-
beddings, and several transformations. Notably, these models together with simple n-grams
have shown strong performance in binary hope speech. By referring to Table 1, the detailed
results of applying the models of this article are presented.
Table 1
Best performing models in each learning approach.
Model Learning approach Averaged-weighted F1 Averaged-macro F1 Category
BERT, RoBERTa, and XLNet Transformers 0.85 0.85 Binary hope speech detection
BERT Transformers 0.77 0.72 Multiclass hope speech detection
3. Methodology
In this research, we explored the efficiency of two transformer models, namely ’albert-base-v2’
and ’bert-base-multilingual-cased’, in handling text classification assignments in English and
Spanish correspondingly. Our primary objective was to devise a technique capable of identifying
hope-related attributes within texts and discerning between instances of "Hope" and "Not hope",
subsequently categorizing them into three refined hope classifications: "generalized hope",
"realistic hope", and "unrealistic hope". For further elucidation on this process, we can refer to
flowchart 2 to enhance our comprehension of the task.
3.1. Task Description
The proposed collaborative task encompasses two objectives aimed at uncovering hope within
social media texts.
Task 1: Targeting Spanish tweets, the aim is to discern instances of hopeful discourse, partic-
ularly emphasizing themes of equality, diversity, and inclusion. Subtasks involve identifying
hopeful discourse within the LGTBI domain and in unidentified domains [15, 16].
Task 2: This task centers on identifying hope speech associated with expectations within
both English and Spanish texts. Subtasks include binary and multi-class predictive speech
Figure 1: The framework of hope
recognition. The outcomes of this study are pertinent to Task 2 [17].
3.2. Dataset
The dataset includes tweet comments in both English and Spanish. Table 2 provides statistics
for the training data showing the distribution between two binary classes and multiple classes
in the dataset. In addition, Table 3 displays the test dataset, comprising 1032 tweets in English
and 1152 tweets in Spanish [18].
Table 2
Data metrics pertaining to the training set
Category English Spanish
Binary-Train
Hope 3634 2553
Not Hope 3590 5500
Multiclass-Train
Not Hope 3509 5500
Generalized Hope 2026 1337
Realistic Hope 858 579
Unrealistic Hope 750 637
3.3. Model Construction
The hope speech task was addressed using the Simple Transformers library, with the ’albert-base-
v2’ model utilized for English binary classification and the ’bert-base-multilingual-cased’ model
for Spanish binary classification. Initially, the training and evaluation datasets were prepared,
comprising text samples paired with corresponding labels indicating hope or non-hope speech.
Table 3
Data metrics pertaining to the test set
Category English Spanish
Binary-Tast
Hope 541 379
Not Hope 491 773
Multiclass-Test
Not Hope 491 773
Generalized Hope 309 206
Realistic Hope 124 77
Unrealistic Hope 108 96
Subsequently, an instance of a Classification Model from the Simple Transformers library
was instantiated, with the desired model architecture and training configuration parameters
specified. The model was then trained on the training dataset and evaluated for its performance
on the evaluation dataset. Furthermore, predictions were made on new text samples using
the trained model to identify instances of hope speech. Throughout the process, an emphasis
was placed on leveraging the capabilities of the Simple Transformers library to streamline the
implementation of the hope speech detection task.
At the second level, we also utilized the Simple Transformers library, employing the ’albert-
base-v2’ model for the English multiclass task and the ’bert-base-multilingual-cased’ model for
the Spanish multiclass task in hope speech classification. The training and evaluation datasets
were meticulously curated, with each example paired with corresponding labels indicating
categories like "generalized hope," "realistic hope," and "unrealistic hope." We instantiated a
ClassificationModel with customized parameters to facilitate the training process. Following
this, the model underwent training on the prepared training dataset and evaluation on a separate
dataset to gauge its performance. Finally, the trained model was employed to make predictions
on unseen data, showcasing its ability to classify hope speech across various languages and
categories. The parameters for both ALBERT (albert-base-v2) and BERT (bert-base-multilingual-
cased) are selected based on standard NLP practices to achieve a balance between performance
and computational efficiency. These parameters are presented in Table 3.
Table 4
Hyperparameters for ALBERT (albert-base-v2) and BERT (bert-base-multilingual-cased)
Hyperparameter ALBERT (albert-base-v2) BERT (bert-base-multilingual-cased)
Attention Heads 12 12
Hidden Size 768 768
Feedforward Size 3072 3072
Dropout Rate 0.1 0.1
Learning Rate 2e-5 2e-5
Batch Size 32 32
Epochs 15 15
4. Results
In this study, the ’albert-base-v2’ and ’bert-base-multilingual-cased’ models were employed
for analyzing English and Spanish, respectively. Table 4 displays the computational outcomes,
showcasing various tasks assessed at different stages, with measures including precision (Pr),
recall (Re), and F1 score (F1) for both micro (M_) and weighted (W_) averages alongside precision
(acc). The results in Table 4 demonstrate the exceptional performance of the ’lbert-base-v2’
model, achieving an M_F1 score of 0.85 for English binary data and 0.79 for Spanish binary
data using the ’bert-base-multilingual-cased’ model. Additionally, among 17 participants, we
secured the 3rd position in the PolyHope Binary (English) category and ranked 3rd out of 13
participants in the PolyHope Multiclass (English) category. Moreover, we achieved the 7th
position in the PolyHope Binary (Spanish) category out of 14 teams and the 8th position in the
PolyHope Multiclass (Spanish) category out of 11 teams.
Table 5
Results of PolyHope Model Performance for Binary and Multiclass Classification in English and Spanish
Tasks M_Pr M_Re M_F1 W_Pr W_Re W_F1 acc acc
PolyHope Binary (English) 0.85 0.85 0.85 0.85 0.85 0.85 0.85
PolyHope Multiclass (English) 0.67 0.68 0.67 0.74 0.74 0.74 0.74
PolyHope Binary (Spanish) 0.82 0.77 0.79 0.82 0.83 0.82 0.83
PolyHope Multiclass (Spanish) 0.51 0.44 0.44 0.67 0.69 0.67 0.69
5. Conclusion
The exploration of hope speech within social media comments in both English and Spanish
through transformer-based models has yielded significant insights and competitive results.
Utilizing the ’albert-base-v2’ model for English and the ’bert-base-multilingual-cased’ model
for Spanish, we were able to achieve commendable performance in both binary and multiclass
classification tasks. The binary classification tasks demonstrated strong performance, with M_F1
scores of 0.85 for English and 0.79 for Spanish. The multiclass classification tasks, although
more challenging, also showed reasonable performance, with M_F1 scores of 0.67 for English
and 0.44 for Spanish.
Our method’s success in the HOPE track of the IberLEF 2024 competition, securing 3rd place
in both PolyHope Binary (English) and PolyHope Multiclass (English) categories, underscores
the effectiveness of transformer models in capturing and categorizing hope-related attributes
within text.
6. Acknowledgements
The work was done with partial support from the Mexican Government through the grant A1-
S-47854 of CONACYT, Mexico, grants 20241816, 20241819, and 20240951 of the Secretaría de
Investigación y Posgrado of the Instituto Politécnico Nacional, Mexico. The authors thank the
CONACYT for the computing resources brought to them through the Plataforma de Aprendizaje
Profundo para Tecnologías del Lenguaje of the Laboratorio de Supercómputo of the INAOE,
Mexico and acknowledge the support of Microsoft through the Microsoft Latin America PhD
Award
References
[1] M. Tash, Z. Ahani, M. Zamir, O. Kolesnikova, G. Sidorov, Lidoma@ lt-edi 2024: Tamil
hate speech detection in migration discourse, in: Proceedings of the Fourth Workshop on
Language Technology for Equality, Diversity, Inclusion, 2024, pp. 184–189.
[2] M. Arif, M. S. Tash, A. Jamshidi, I. Ameer, F. Ullah, J. Kalita, A. Gelbukh, F. Balouchzahi,
Exploring multidimensional aspects of hope speech computationally: A psycholinguistic
and emotional perspective (2024).
[3] B. W. Corn, D. B. Feldman, I. Wexler, The science of hope, The Lancet Oncology 21 (2020)
e452–e459.
[4] B. R. Chakravarthi, Hope speech detection in youtube comments, Social Network Analysis
and Mining 12 (2022) 75.
[5] A. L. Tonja, M. G. Yigezu, O. Kolesnikova, M. S. Tash, G. Sidorov, A. Gelbuk, Transformer-
based model for word level language identification in code-mixed kannada-english texts,
arXiv preprint arXiv:2211.14459 (2022).
[6] M. G. Yigezu, A. L. Tonja, O. Kolesnikova, M. S. Tash, G. Sidorov, A. Gelbukh, Word
level language identification in code-mixed kannada-english texts using deep learning
approach, in: Proceedings of the 19th International Conference on Natural Language
Processing (ICON): Shared Task on Word Level Language Identification in Code-mixed
Kannada-English Texts, 2022, pp. 29–33.
[7] S. Palakodety, A. R. KhudaBukhsh, J. G. Carbonell, Hope speech detection: A computational
analysis of the voice of peace, arXiv preprint arXiv:1909.12940 (2019).
[8] B. R. Chakravarthi, V. Muralidaran, R. Priyadharshini, S. Cn, J. P. McCrae, M. Á. García,
S. M. Jiménez-Zafra, R. Valencia-García, P. Kumaresan, R. Ponnusamy, et al., Overview
of the shared task on hope speech detection for equality, diversity, and inclusion, in:
Proceedings of the second workshop on language technology for equality, diversity and
inclusion, 2022, pp. 378–388.
[9] M. Shahiki-Tash, J. Armenta-Segura, O. Kolesnikova, G. Sidorov, A. Gelbukh, Lidoma at
hope2023iberlef: Hope speech detection using lexical features and convolutional neural
networks, in: Proceedings of the Iberian Languages Evaluation Forum (IberLEF 2023), co-
located with the 39th Conference of the Spanish Society for Natural Language Processing
(SEPLN 2023), CEUR-WS. org, 2023.
[10] M. Shahiki-Tash, J. Armenta-Segura, Z. Ahani, O. Kolesnikova, G. Sidorov, A. Gelbukh,
Lidoma at homomex2023@ iberlef: Hate speech detection towards the mexican spanish-
speaking lgbt+ population. the importance of preprocessing before using bert-based models,
in: Proceedings of the Iberian Languages Evaluation Forum (IberLEF 2023), 2023.
[11] Z. Ahani, G. Sidorov, O. Kolesnikova, A. Gelbukh, Zavira at hope2023@ iberlef: Hope
speech detection from text using tf-idf features and machine learning algorithms (2023).
[12] F. Balouchzahi, G. Sidorov, A. Gelbukh, Polyhope: Two-level hope speech detection from
tweets, Expert Systems with Applications 225 (2023) 120078.
[13] M. S. Tash, Z. Ahani, A. Tonja, M. Gemeda, N. Hussain, O. Kolesnikova, Word level
language identification in code-mixed kannada-english texts using traditional machine
learning algorithms, in: Proceedings of the 19th International Conference on Natural
Language Processing (ICON): Shared Task on Word Level Language Identification in
Code-mixed Kannada-English Texts, 2022, pp. 25–28.
[14] Z. Ahani, M. Shahiki Tash, Y. Ledo Mezquita, J. Angel, Utilizing deep learning models for
the identification of enhancers and super-enhancers based on genomic and epigenomic
features, Journal of Intelligent & Fuzzy Systems (????) 1–11.
[15] D. García-Baena, M. Á. García-Cumbreras, S. M. Jiménez-Zafra, J. A. García-Díaz,
R. Valencia-García, Hope speech detection in spanish: The lgbt case, Language Resources
and Evaluation 57 (2023) 1487–1514.
[16] L. Chiruzzo, S. M. Jiménez-Zafra, F. Rangel, Overview of IberLEF 2024: Natural Language
Processing Challenges for Spanish and other Iberian Languages, in: Proceedings of the
Iberian Languages Evaluation Forum (IberLEF 2024), co-located with the 40th Conference
of the Spanish Society for Natural Language Processing (SEPLN 2024), CEUR-WS.org,
2024.
[17] F. Balouchzahi, G. Sidorov, A. Gelbukh, Polyhope: Two-level hope speech detection from
tweets, Expert Systems with Applications 225 (2023) 120078. doi:10.1016/j.eswa.2023.
120078.
[18] D. García-Baena, F. Balouchzahi, S. Butt, M. Á. García-Cumbreras, A. Lambebo Tonja,
J. A. García-Díaz, S. Bozkurt, B. R. Chakravarthi, H. G. Ceballos, V.-G. Rafael, G. Sidorov,
L. A. Ureña-López, A. Gelbukh, S. M. Jiménez-Zafra, Overview of HOPE at IberLEF 2024:
Approaching Hope Speech Detection in Social Media from Two Perspectives, for Equality,
Diversity and Inclusion and as Expectations, Procesamiento del Lenguaje Natural 73
(2024).