Hope is one of the exceptional human capacities that allows for flexible anticipation of future events and possible expected outcomes. Hope speech detection is a crucial task in understanding and fostering positive discourse within online communities. Shared task HOPE at IberLEF 2024 encompasses two distinct but complementary subtasks, each addressing diferent facets of hope speech. Task 1 is called as “Hope for Equality, Diversity, and Inclusion” and aims to detect hope speech related to Equality, Diversity, and Inclusion. Task 2, “Hope as Expectations” focuses on expectations and desirable and undesirable facts. In this paper, we present our unsupervised approach to solve both tasks. Our main idea is to leverage the power of a large language model, ChatGPT 3.5, and the prompting technique to solve two tasks. We used ChatGPT 3.5 with three prompting techniques: zero-shot, few-shot (one-shot, three-shot), and chain of thought, combined with six diferent information strategies. Our best prompts achieved top 1 in task 1 with a Macro F1-score of 0.7161. However, the performance did not meet expectations in the sub-tasks of task 2.
Hope is one of the exceptional human capacities that allows for flexible anticipation of future
events and possible expected outcomes. These visions significantly influence emotions,
behaviour, and mood (Bruininks and Malle [
Inspired by applying Large language models (LLMs) or ChatGPT to solve Sentiment analysis
tasks (Fatouros et al. [
Prompt Engineering helps to efectively design and improve prompts to get better results
on diferent tasks with LLMs. There are many techniques have been proven to be efective
when working with LLMs, in this study we only focus on three techniques Zero-shot,
Fewshot, and chain of thought promptings. In the prompting context, the term “shot” refers to
the demonstration or example of what exactly the user wants LLMs to do. Zero-shot means
no examples are provided for LLMs, and instruction tuning has been shown to improve
zeroshot learning (Wei et al. [
The shared task on hope speech detection was held many times in the past at the second
workshop on Language Technology for Equality, Diversity and Inclusion (LT-EDI-2022), as a
part of ACL 2022 (Chakravarthi et al. [
This task is related to the inclusion of vulnerable groups and focuses on the study of the detection of hope speech in pursuit of Equality, Diversity, and Inclusion. Given a tweet written in Spanish, participants must identify whether it contains hope speech or not. Specifically, this task is divided into two subtasks, but participants will participate in both of them at the same time.
With given training data on LGTBI tweets from García-Baena et al. [
– nhs: if the tweet does not contain hope speech. where the test is set in subtask 1.a only contains LGTBI-related tweets while subtask 1.b contains tweets on unknown domains. • Task 2: Hope as expectations.
Hope is characterized as “openness of spirit toward the future, a desire, expectation, and
wish for something to happen or to be true” that remarkably afects a human’s state of
mind, emotions, behaviours, and decisions (Bruininks and Malle [
– Subtask 2a: Binary hope speech detection from English and Spanish Texts Given training data, the participant will classify the text into two categories. In this problem, each text will be assigned one of the following labels : ∗ Hope: tweets that convey a mention of hope.
∗ Not Hope: tweets that do not convey hope, expectation, or desire.
– Subtask 2b: Multiclass hope speech detection from English and Spanish Texts
(Balouchzahi et al. [
∗ Not Hope: tweets that do not convey hope.
We designed prompts to solve this shared task and experimented to research two aspects of the prompt design technique are Providing information strategies and Prompting techniques with ChatGPT 3.5.
This aspect aims to research which information is essential and eficient for Large language models to solve this shared task. We propose four kinds of information and combine them to create six providing information strategies.
• Information 1 - Request: This information simply points out what we want Large language models to do. • Information 2 - Concept of problem: This information provides the concept or definition of the key problem LMMs must solve. • Information 3 - Meaningful of classes: In this study, we apply ChatGPT for classification tasks, so the meaning of classes might be the essential information. • Information 4 - Role defining: In most situations, it is better to clearly understand that Someone must do something with a specific role. In this study, we defined ChatGPT as an NLP engineer expert and a language specialist, and it is labelled to create a dataset for this shared task. We use this role throughout subtasks in this shared task. Fig 1 illustrates the way we use four kinds of information above to create six information strategies. In which: • Strategy 1 - Only request: This strategy simply provides directly what we want Large language models to do without any additional information. • Strategy 2 - Concept of problem and Request: Except for the request information, this strategy also provides the concept of key problem LMMs must do. • Strategy 3 - Meaningful of classes and Request: Except for the request information, this strategy also provides the meaning of each class LLMs must classify. • Strategy 4 - Role defining and Request: Before giving the request, this strategy provides information about the role of LLMs with the hope that LLMs will understand more about their context and tasks. • Strategy 5 - Combination 1: Simply, this strategy is the combination of Information 2, 3 and 1. • Strategy 6 - Combination 2: Simply, this strategy is the combination of Information 2, 3, 4, and 1.
This aspect focuses on how to give the request and instructions for Large language models to generate the answer. In this study, we experimented with three prompting techniques: zero-shot, few-shot, and chain of thought (CoT). Additionally, we experimented with One-shot and Three-shot for the few-shot prompt setting. Specifically, each technique is as follows: Normal answer format: Tweet: “<given tweet>” Label: CoT answer format: Tweet: “<given tweet>” Explanation: Label:
Zero-shot prompting: <Strategy>\n<normal answer format> One-shot prompting: <Strategy>\n<One-shot demontration>\n<normal answer format> Three-shot prompting: <Strategy>\n<Three-shot demontration>\n<normal answer format> CoT prompting: <CoT Strategy>\n<CoT answer format>\n<CoT answer format> • Zero-shot: With the Zero-shot technique, the final prompts are the strategies of the
Providing information strategies aspect without providing additional information. • Few-shot: Except for the instructions information of the Providing information strategies, this technique will provide some examples to demonstrate what exactly the LMMs must do and what our expected output is. In this study, with the One-shot technique, we will provide one demonstration corresponding to each class. Similar to the Three-shot technique, we will provide three demonstrations corresponding to each class. • Chain of thought: Applying the chain of thought technique for general classification tasks is a form that requires LLMs to provide explanations for their classification results. Because of the diferences in the content of the expected answers in the Chain of thought and other techniques. We need to define two separate answering formats: the normal answer format and the CoT answer format. The harmony between Strategy, Prompting techniques, and answering format to create the final promptings is presented in Figure 2.
We solve this shared task by using an unsupervised method, and during the process of carrying out the experiment, we nearly don’t need the training data except to take some samples of training to serve for the Few-shot technique. We join all subtasks in this shared task and evaluate each subtask according to the metrics of the organizer.
Because we join all subtasks in this shared task, each subtask has a diferent goal, concept, or label, so to be convenient for us to present full prompting for each subtask, we will pre-define some variables to store some part of the text. In this report, We only present design prompting results for task 1; pre-define and complete prompting for all subtasks can be found in this request: request for the Zero-shot and Few-shot technique for task 1 Please classify whether a given text conveys hope or not. Only return the label from [Hope Speech, Non Hope Speech] without any other text.
request_CoT: request for the Chain of thought technique for task 1 Please classify whether a given tweet conveys hope or not. Only return one label from [Hope Speech, Non Hope Speech] and explain your classification results.
concept: the concept of Hope in task 1 Hope speech is the type of speech that is able to relax a hostile environment and that helps, gives suggestions and inspires for good to a number of people when they are in times of illness, stress, loneliness or depression.
class: Meaningful of classes in task 1 Describe the sentiment of the given text using one of these two attributes: “Hope Speech”, “Non Hope Speech”. Knowing that a text is considered as “Hope Speech“ if the text: (1) explicitly supports the social integration of minorities or otherwise distinct groups in society; (2) promotes tolerance, equality, positivity, and healing for all. And knowing that a text is considered “Non Hope Speech” if it: expresses the negative sentiment, discrimination, violence; or uses insults toward people.
one-shot: one demonstration for task 1 Tweet: “Hay que tratar de aislar democráticamente los argumentos polÃticos que justifican a machistas, xenófobos, racistas y las expresiones de odio a los colectivos LGTBI. El odio genera violencia y la violencia no tiene medida en sus consecuencias” Label: Hope Speech Tweet: “Los fundadores del socialismo,-Karl Marx: “Los homosexuales son peores que pederastas”.- Friedrich Engels: â€oeLa homosexualidad es una aberración de la burguesÃa degeneradaâ€.- Iósif Stalin: “La homosexualidad es un vicio burguÃs”” Label: Non Hope Speech three-shot: three demonstrations for task 1. It is similar to one-shot_1, however, the number of demonstrations in each class is three repository1. Detailed pre-variable to create prompting for task 1 as Fig 3. In Table 1, we show our detail syntax for final prompting in 6 strategies with three techniques: Zero-shot, One-shot, and Chain of thought for task 1. Because of paper length limitations, other subtasks are not presented in this table. But it’s easy to deduce based on the pre-defined variables and syntax we’ve provided at this repository1. Additionally, in Figure 4 we present full prompting according to One-shot prompting technique and providing information Strategy 5 - prompting helps us achieve top 1 on the test set of task1.
Table 2 shows the Macro Recision, Macro Recall, and Macro F1-score of all prompting we designed on the validation set for task 1. Through this table, we can see that all prompts achieved positive performance. Although there are small diferences between classifications, on average, strategy 3 and strategy 6 generally perform better than the others, especially as they often significantly improve over strategy 1. Strategy 1 is better than Strategy 2 in almost all techniques, although Strategy 2 includes the concept of Hope. This unfortunate situation may be because the concept we have defined is somewhat non-standard compared to the data set, causing ChatGPT to be confused in the process of providing the final classification results.
Concept of
problem Meaningful of classes
Request One-shot
Hope speech is the type of speech that is able to relax a hostile environment and that helps, gives suggestions and inspires for good to a number of people when they are in times of illness, stress, loneliness or depression.
Describe the sentiment of the given text using one of these two attributes: “Hope Speech”, “Non Hope Speech”. Knowing that a text is considered as “Hope Speech“ if the text: (1) explicitly supports the social integration of minorities or otherwise distinct groups in society; (2) promotes tolerance, equality, positivity, and healing for all. And knowing that a text is considered “Non Hope Speech” if it: expresses the negative sentiment, discrimination, violence; or uses insults toward people.
Please classify whether a given text conveys hope or not. Only return the label from [Hope Speech, Non Hope Speech] without any other text.
Tweet: “Hay que tratar de aislar democráticamente los argumentos polÃticos que justifican a machistas, xenófobos, racistas y las expresiones de odio a los colectivos LGTBI. El odio genera violencia y la violencia no tiene medida en sus consecuencias” Label: Hope Speech Tweet: “Los fundadores del socialismo,-Karl Marx: “Los homosexuales son peores que pederastas”.Friedrich Engels: â€oeLa homosexualidad es una aberración de la burguesÃa degeneradaâ€.- Iósif Stalin: “La homosexualidad es un vicio burguÃs””
Label: Non Hope Speech
Normal answer format
Tweet: <given tweet>
Label: The average on strategies of each prompting technique, the performances are similar. However, there is a special in the Chain of thought technique, Strategy 3, 4, 5, and 6 have significant improvements compared to Strategy 1, and 2. Besides, Strategies 4, 5, and 6 in this technique also give stable performance compared to the remaining techniques. This may be because the Chain of thought technique requires more information to make inferences. In this report result, the performance of the Few-shot (One-shot and Three-shot) technique does not improve performance compared to Zero-shot but this is completely within our expectations. Because the task in task 1 is described quite clearly, along with the meaning of the classes fed by strategy 3, it is enough for LLMs to understand its task. Providing additional demonstration with a few shots increases the size of the input text, which can afect the accuracy of LLMs during processing.
Table 3 reports the Macro F1-score on the validation set for subtask 2a and subtask 2b for both English and Spanish. In task 2, Performance seems more stable across techniques and strategies. Three-shot is usually better in all cases, and among strategies, Strategy 3 and Strategy 5 often give the most stable and highest results. If in binary classification tasks (task 1 and subtask 2. a), the Few-shot technique and strategy 3 have not made a diference from Zero-shot and strategy 1. Then, in multi-classification tasks 2. b, Few-shot provides some demonstrations, and strategy 3 has additional information on the meaning of classes compared to strategy 1, which has shown significantly improved results. This demonstrates the efectiveness of providing the meaning of classes and illustrative examples for LLMs in complex tasks.
According to the performance report of both task 1 and task 2, if there is only concept and request in the prompt (strategy 2), then concept information is not really efective in this task. However, when combining the other information, such as the meaning of classes or the role of LLMs (strategy 5, strategy 6), it is somewhat improved and more stable. With role definition (strategy 4) for LLMs, in English tasks, it works quite poorly, actually reducing accuracy compared to strategy 1 (just giving requests). However, we have an intuition about the efectiveness of defining roles for LLMs in Spanish tasks. It cannot be denied that when role information stands alone in strategy 4, it still produces quite bad results, but when combined with other information in strategy 6 when working with Spanish, it can be efective use. We are not saying that defining the role helps LLMs produce good results with Spanish. We would like to suggest that defining the role as “an NLP engineer expert as a language specialist” when working with Spanish helps Strategy 4 be more competitive when working with English.
Because of the limitation of submission, we can’t conduct all the prompting that we designed on the testing set. Table 4 presents the best result and our rank on the test set on all of the subtasks in promptings we submitted on CodaLab. For all subtasks, one-shot techniques are the best, and providing an information strategy is like what we observed on the validation set. We achieved top 1 in task 1, while the results achieved in task 2 were not satisfactory. It is possible that achieving top 1 in task 1 is due to the nature of the test set containing out-of-domain data compared to training data. While applying LLMs can be efective in classification tasks such Performance of all prompting on the validation set for task 1 on Macro Recision, Macro Recall, and Precision
F1-score
Precision
F1-score Performance of all prompting on the validation set for task 2 on Macro F1-score Macro F1-score Final result and rank on the test set on all of the subtasks public on CodaLab
Subtask
Task 1 Subtask 2a - English Subtask 2a - Spanish Subtask 2b - English Subtask 2b - Spanish
Technique One-shot One-shot One-shot One-shot One-shot
Strategy Strategy 5 Strategy 5 Strategy 6 Strategy 5 Strategy 6
Rank 1 9 9 7 11
Precision 0.64 0.75 0.73 0.46 0.47 as saving time, training costs, and the need to worry about training data. However, It is still inferior to supervised learning methods when data are in a specific domain. In particular, LLMs work based on the information we provide in the prompts, so a close description of the concept and goal of the test set is a must. Describe information of test set non-standard will directly afect the results of the method.
In this study, we designed and carried out experiments on various promptings. With each technique and strategy, we evaluate and explain the results. We achieved top 1 by using prompting which combined of Zero-shot prompting technique and providing information strategy 5, however, the results achieved in task 2 are quite modest. Applying LLMs like ChatGPT 3.5 to solve hope speech detection can bring many benefits, but we need to focus on describing the data. The concepts and problem objectives must be compatible with the data. At the same time, it is necessary to evaluate the complexity of the problem to choose the appropriate technique and strategy to take full advantage of the power of LLMs in tasks like this. and despair in chronic pain, Culture, Medicine, and Psychiatry 40 (2016) 35–58. [27] E. Garrard, A. Wrigley, Hope and terminal illness: False hope versus absolute hope, Clinical
Ethics 4 (2009) 38–43. [28] J. Eliott, I. Olver, The discursive properties of “hope”: A qualitative analysis of cancer patients’ speech, Qualitative health research 12 (2002) 173–193. [29] M. Links, J. Kramer, Breaking bad news: realistic versus unrealistic hopes, Supportive Care in Cancer 2 (1994) 91–93.