Implicit Sentiment Analysis (ISA) aims to capture emotions that are subtly expressed, often obscured by ambiguity and literary devices. This requires a shift in thinking based on memorized prior knowledge (past cases) and multi-stage logical reasoning to uncover the hidden emotions of the speaker. Inspired by human memory and reasoning, we propose a What, Why, and How three-question (3Q) framework that incorporates a memory mechanism for past cases. We design a three-step prompt principle for it: based on domain priors, first identify what the most crucial entity is, then infer why the speaker mentioned it, and finally uncover how the hidden emotions are. During the reasoning process, historical queries and responses are stored in memory as past case pairs. These pairs can be used to retrieve and generate improved prompts for any new queries, thereby enhancing the implicit sentiment analysis capabilities of LLMs. In addition, we compile a more practical and complex benchmark for ISA tasks. It spans multiple domains and includes bilingual corpora in both English and Chinese. Our framework is universal and minimalist, and achieves a new state-of-the-art by significantly outperforming previous methods in both zero-shot and fine-tuning settings. It also significantly reduces the tendency to over-interpret emotions.
Sentiment analysis (SA) aims at detecting the sentiment polarity of a given text. SA can be divided into
explicit SA (ESA) and implicit SA (ISA), with the former being the mainstream task where sentiment
expressions are explicitly present in the text [
Traditional research on implicit sentiment analysis has leaned on paradigms such as attention
mechanisms and feature extraction [
The advent of large language models (LLMs) such as ChatGPT [
While THOR has made progress in ISA tasks, there are cases where its performance is not efective. This problem motivates us to rethink the ISA task from the perspective of simulating human memory and reasoning. First and foremost, domain knowledge is essential for judging emotional polarity. Consider the sentence ‘No one can stop him from rampaging like a bull.’ again, its interpretation and associated emotions vary significantly between contexts. In a competitive setting, this might praise an athlete’s impressive performance and convey a positive sentiment. Conversely, in daily life, it might criticize someone’s careless behavior, reflecting a negative sentiment. Without the relevant domain or context, humans would also experience cognitive ambiguity in decision making. Furthermore, people will subconsciously recall past cases and use them to identify the most crucial entity that evokes emotion in a sentence (e.g., ‘him, which may refer to an athlete’). They then use empathy to understand why the speaker mentioned that entity by probing for its true intent (e.g., ‘The speaker wants to highlight this entity’s determined and unstoppable nature in relation to competition.’). Finally, based on the previous inferences and past cases, we can discern the underlying emotions (e.g., ‘which can be seen as positive attributes’).
Based on our findings, we propose a universal What, Why, and How three-question (3Q) framework with a memory function for past cases. It is structured in three steps: based on domain priors, 1) identify what is the most crucial entity in the sentence, 2) infer why the speaker wants to mention it, 3) uncover how the hidden emotion is like. During the above process, historical queries and responses are stored in memory as past case pairs. These pairs allow the retrieval and creation of improved prompts for any new query. This minimalist approach, which mirrors human memory and reasoning, efectively captures the essence of sentence emotions and reveals hidden meanings, thereby simplifying the judgment of sentiment polarity.
Moreover, to evaluate the efectiveness of the proposed framework, we enable a Chinese-English bilingual dataset derived from several oficial datasets. Experimental results show that the 3Q framework sets a new state-of-the-art (SOTA) benchmark in both supervised fine-tuning and zero-shot settings. Furthermore, it significantly outperforms THOR and Direct on F1 and neutral F1 metrics, while mitigating the negative efects of excessive emotion interpretation.
To sum up, our contributions are: • We propose a What, Why, and How three-question (3Q) framework with a memory mechanism for past cases. It mimics human memory and reasoning and puts LLMs in the shoes of speakers for ISA tasks. It refines ISA tasks into three simple steps, each coupled with a memory function: based on domain priors, 1) identify what the most crucial entity is in the sentence, 2) infer why the speaker wants to mention it, 3) uncover how the implied emotions are like. • We enable a more practical and complex benchmark for ISA tasks, where it includes both English and Chinese languages, covering 28 domains or scenes. • Extensive evaluations on the benchmark reporting SOTA results in both zero-shot and fine-tuning settings. Experimental results show that the 3Q framework can efectively handle various complex situations, such as over-interpretation of emotions.
Current research on ISA task predominantly centers on deep learning based methods. For example,
the SCAPT model [
We observe that many implicit sentiment analysis models, including THOR, are based on traditional ifne-grained sentiment analysis, such as Aspect-Based Sentiment Analysis (ABSA) tasks. We find that by referencing these fine-grained sentiment analysis tasks, decomposing the sentiment analysis of the corpus into key emotional elements such as ‘aspect words’ can improve the accuracy of sentiment analysis. However, ABSA requires manual annotation of aspect words, which is dificult to achieve in large datasets. Inspired by human memory and reasoning, we propose 3Q, a What, Why, and How threequestion framework that integrates a memory function for past cases. It is minimalist and universal, facilitating eficient implicit sentiment analysis.
There are several common methods of case-based reasoning with LLMs. Few-shot prompting involves
incorporating case demonstrations into prompts to guide the model toward better performance [
The rise of the Chain-of-thought (CoT) has been efective in alleviating these problems. It enhances
the multi-step reasoning ability of LLMs by inducing models that simulate human step-by-step reasoning
processes to reach conclusions [
The task of Implicit Sentiment Analysis (ISA) is to determine the sentiment polarity of a given sentence, categorizing it as positive, neutral, or negative. A standard approach without memory is to use a Direct Prompt template as input for LLMs, typically in the following form:
Given the sentence , what is the sentiment polarity towards it?
LLMs should return the answer via: ˆ = argmax ( | ).
In the 3Q framework, users sequentially ask three questions ( = 1, 2, 3) to query the model for sentiment analysis of the input sentence . At each step, the questions and the corresponding model-generated answers are stored in memory as past case pairs. If there are past case pairs in memory, LLMs will automatically retrieve them from memory and integrate them into the current question. The memory implementation consists of the following components:
Memory : Memory is a dynamically expanding table managed by the LLM itself that contains past case pairs. The -th case pair is denoted as (, ). The memory contains all these past case pairs for future reference. It supports the store operation .() and the fetch operation () itself. These operations can be achieved using simple key-value lookups and prompt concatenation. Figure 2 omits the mathematical formulas of these two operations for simplicity.
Past Case Pairs : All existing cases stored in memory can be concatenated to construct , which represents the totality of past information. It can be represented as: = (1) + (2) + ... + (). can also be combined with the next question +1 as a new query.
Combiner (1, 2): It supports combining past case pairs and questions as a new query.
The prompt-based memory can be structured as follows: <1> [INST] {1} [/INST] {1} </1> <2> [INST] {2} [/INST] {2} </2> <3> [INST] {3} [/INST] {3} </3>
It is worth noting that the structure of the prompt-based memory will vary slightly to accommodate the diferent prompt formats required by diferent LLMs.
In our novel 3Q framework, we aim to have the LLM identify the most crucial entity ˆ that influences sentence sentiment, then ask it to put itself in the speaker’s shoes to uncover the reasons ˆ. Finally, the LLM should determine the sentiment polarity ˆ based on these prior inferences. We break this process down into three steps as follows:
What. We first provide the sentence along with its corresponding field , and then we ask the LLM to identify the most critical entity within the sentence:
Given the sentence , the field of it is . What is the most crucial entity described in it? This step is represented as ˆ = argmax (|1(, )), where ˆ is the model’s inference of the entity , explicitly stating what the entity is and its possible reasons. Note that the entity needn’t be explicitly stated in the sentence; instead, it can be generalized and abstracted based on the semantic content of the sentence. After this step, the query and answer are stored in memory in the form of case pairs: = . append (1 (1, ˆ)).
Why. Now, based on 1, we position the model from the speaker’s perspective. In this step, we ask the LLMs to uncover the reasons why the speaker mentioned this particular entity ˆ: 2. Why is this entity mentioned in this sentence?
This step can be formulated as: ˆ = argmax (| 1, 2), where ˆ represents the model’s inference regarding the reason for mentioning the entity. After this step, the query and response are stored in the form of case pairs in the memory: = . append (2 (2, ˆ)).
How. With the 2 as the premise, we prompt LLMs to infer the sentiment polarity as the final outcome: 3. Based on the above reasons, how would you describe the sentiment polarity towards the sentence?
This step can be formulated as: ˆ = argmax (| 2, 3), where ˆ is the sentiment polarity ultimately predicted by the model. After this step, the query and response are stored in the form of case pairs in the memory: = . append (3 (3, ˆ)).
The 3Q framework combines the memory mechanism with CoT [
We construct a bilingual dataset in both Chinese and English for ISA tasks. It spans multiple domains
and includes high-quality explicit and implicit corpora. We select 6,000 samples from a number of
publicly accessible datasets: SemEval-2014 Task 4 [
As we analyzed before, one sentence can convey diferent meanings or emotions in diferent fields or contexts. It is unrealistic to perform sentiment analysis without specifying the related field. None of the existing oficial datasets meet this requirement. To address this issue, we annotate each sample with its corresponding field. Specifically, samples from SemEval-2014 Task 4 are labeled as either ‘laptops’ or ‘restaurants’, while those from Twitter US Airline Sentiment are marked as ‘aviation’. Chinese datasets represented by SMP-ECISA 2021 have already specified the field or topic in its oficial description. Therefore, samples from these datasets are annotated accordingly, including fields such as ‘daily’, ‘travel’, ‘politics’, etc. Finally, we obtain a bilingual benchmark dataset 4 with 28 diferent domains.
We choose the Llama 2-CHAT model from Hugging Face as our backbone LLM [
In both zero-shot and fine-tuning settings, we compare our methods with two baseline methods: THOR
and Direct. THOR is the only method that combines LLMs and CoT. The prompt used for it is the
same as the corresponding research [
In the zero-shot scenario, we also use Zero-Shot CoT, which has no memory mechanism, as the
baseline method. We adopt the standard prompt from this study [
For the Llama 2-CHAT model, we use a temperature of 1 and top-p of 1. This configuration is consistent across models with 7B, 13B, and 70B parameters. For gpt-3.5-turbo-1106 API, we set temperature to 1, top-p to 1, and leave other parameters (such as frequency-penalty) unchanged.
We use cloud A100x1 GPU (40G) instances, cloud A100x1 GPU (80GB) instances, and cloud A100x4 GPU (40GB) instances to fine tune the Llama2 model with model sizes of 7B, 13B, and 70B. The average tuning time is 1 hour. After fine tuning, it takes an average of 17 hours to perform inference on the 2000-sample test set. For the gpt-3.5-turbo-1106 model, we use the OpenAI API for fine tuning, which takes about 2 hours on average. After tuning, the inference process on the entire test set takes about 5 hours.
To train the BERT baseline, we use the AdamW optimizer with a learning rate of 5e-5. The batch size is set to 64 and the dropout probability is set to p=0.1. We use NVIDIA 3090x1 GPU instances in the cloud. ISA
All
We take the F1 as one of the evaluation metrics. During the experiments, we observe a significant diference in the neutral F1 compared to the positive and negative F1s. Therefore, we also include the neutral F1 in our evaluation metrics, recognizing its importance in assessing the interpretation of excessive emotion.
Table 2 presents the comparison results under zero-shot settings. It is evident that four methods
that combine LLMs with prompt engineering significantly outperform current state-of-the-art (SoTA)
baselines. Among these, 3Q stands out for its impressive performance. Specifically, when equipped with
the 7B Llama 2-CHAT model, it leads the best performing baseline (BERT-SPC) by 21.66%. As the model
scale increases, the gap in F1 between the two also widens, peaking at a 28.88% diference with the 70B
parameter model. This is consistent with the study’s conclusion [
On all three scales of the Llama2 model, 3Q outperforms the other three prompt methods and achieves a new SOTA performance. Specifically, in the ISA setting, its F1 score is approximately 7.86%, 11.86%, and 20.55% higher than Zero-shot CoT, Direct, and THOR, respectively. This suggests that the combination of both memory mechanisms and multi-step reasoning can provide a greater improvement in implicit sentiment analysis than neither memory mechanisms nor multi-step reasoning. Interestingly, the diference in F1 score between 3Q and THOR is the most significant. This is because a sentence usually contains multiple targets or entities. Sometimes sentence-level sentiment may contradict aspect-level sentiment. We argue that THOR’s ability is limited by its own prompt architecture, which is more concerned with aspect-level sentiment. This is likely to lead to significant errors. In contrast, 3Q does not initially specify entities. Instead, it encourages LLMs to infer the entity that best identifies sentence-level sentiment based on the domain. In the subsequent ‘why’ section, LLMs are prompted to understand and contextualize the reasons behind the most critical entity assertions. In addition, 3Q’s memory mechanism helps to model the relationship between the most crucial entity and other entities in the context, thereby improving the understanding of sentiment at the sentence level. Similar benefits are seen in both Zero-Shot CoT and Direct.
In addition, we note that as the model scale increases, not only does the gap between 3Q and THOR widen, but Direct has also surpassed THOR. These two phenomena are primarily due to the everwidening neutral F1 gap, which is related to the ‘over-interpretation of neutral emotions’ phenomenon that we discuss later.
The comparisons under fine-tuning are shown in Table 3. Overall, high-quality instructional fine-tuning
significantly improves performance. This is consistent with the conclusions of the Flan-T5 study [
Table 2 shows that the F1 score for neutral sample sentiment classification remains low regardless of the configuration. Performance improves slightly when the model parameters reach a large size, as seen in Llama 2-CHAT 70B and gpt-3.5-turbo-1106. This suggests that accurate identification of neutral sentiment continues to be a significant obstacle in ISA tasks.
It is worth noting that THOR maintains an average of 20% F1-neu in most cases. Such poor performance results from its second step, the induction of the implicit opinion. The original prompt is, ‘Based on common sense, what is the implicit opinion towards the mentioned aspect of t, and why?’. The phrase ‘implicit opinion’ causes LLMs to infer inappropriate implicit opinions from neutral sentences, which introduces great interference and error into the next step of judging sentiment polarity.
In contrast, 3Q outperforms THOR by nearly 50% in average neutral F1 and establishes a new SOTA score of 76.86% on the 70B Llama 2-CHAT model. It is also 9.88% higher than Direct under similar configurations. This is because in the ‘why’ section, 3Q does not introduce subjective interventions like ‘implicit opinion’. Instead, it only leads LLMs to infer the reasons for mentioning entities. In neutral sentences, the most critical entity tends to state a specific fact or serve a specific purpose. It is usually mentioned for objective reasons. Therefore, 3Q can efectively mitigate the problem of over-interpreting emotions. Figure 3 visualizes the entire analysis. While Direct also has similar benefits with a relatively high neutral F1 score, it is still 10% lower on average than 3Q. This suggests that the combination of both memory mechanisms and multi-step reasoning can provide a greater improvement in implicit sentiment analysis than neither memory mechanisms nor multi-step reasoning.
In Figure 4 we examine the influence of diferent LLM scales. As the model scale increases, the F1
for all three methods generally show a growth trend. This is consistent with the existing findings of
CoT prompting methods [
We examine the error rates and error cases of the 3Q model in both zero-shot and supervised fine-tuning settings on our test set. Errors are categorized into three types: Type A error represents over-analyzing neutral corpora, where the model mistakenly interprets neutral content as positive or negative sentiment; Type B error occurs when the model fails to capture sentiment clues in the corpora and misjudges negative and positive as neutral; Type C error means the model confusing two non-neutral emotions, labeling positive as negative or vice versa. The distribution of these three types of errors is illustrated in Figure 5.
From the experimental results, it is evident that in the zero-shot setting, the gpt-3.5-turbo-1106 is predominantly associated with Type A errors, i.e, among the total error rate of 21.21%, Type A errors accounts for 18.72%. Supervision Fine-tuning reduces the Type A error rate to a low level (3.64%), i.e., eliminating approximately 80% of Type A errors. In contrast, the proportions of Type B and Type C errors show no significant changes before and after supervised fine-tuning. Moreover, these two error types account for a smaller portion of the overall errors, possibly due to the inherent dificulty of the self-built corpora and the limitation of model capabilities.
In contrast, the Llama2-CHAT-70B model, unlike gpt-3.5-turbo-1106, tends to misclassify neutral sentences. In the zero-shot setting, the Llama 2-CHAT-70B model exhibits a Type B error rate of 7.09%, which surpasses the Type A error rate (5.79%). Even after supervised fine-tuning, the Type B error rate (3.09%) exceeds the zero-shot results of the gpt-3.5-turbo-1106 model (1.60%). This suggests that the Llama 2-CHAT-70B model is prone to misclassifying sentiment-unnoticed samples as neutral.
In this research, we propose a What, Why, and How three-question (3Q) framework that incorporates a memory mechanism for past cases. Specifically, 3Q performs a sequential three-step prompt by simulating the human memory and reasoning process. It first determines what the most critical entity in the sentence is, then infers why the speaker mentions it. Finally, 3Q uncovers how the hidden emotions are based on the past cases. During the reasoning process, historical queries and responses are stored in memory as past case pairs. These pairs enable the retrieval and creation of enhanced prompts for any new query, thereby enhancing the implicit sentiment analysis capabilities of LLMs. To verify the efectiveness of it, we construct a Chinese-English bilingual dataset based on several oficial datasets. 3Q achieves a new SoTA under both supervised fine-tuning and zero-shot settings. Experimental results show that it significantly outperforms THOR, Direct and Zero-Shot CoT in F1 scores, and efectively reduces the negative efects of over-interpretation of emotions. We release the dataset in an associated repository and hope that the accessibility of the dataset will encourage the community to evaluate novel methods for implicit sentiment analysis. In the future, we will extend this approach to the real-world document understanding. In addition, we plan to further explore the ISA task by simulating the human nonlinear reasoning process from the perspective of Tree of Thoughts (ToT) and Graph of Thoughts (GoT).