This paper presents a qualitative analysis of the emotional component in manipulative online content (fakes). We show that emotion triggering is a crucial persuasion technique widely employed by unscrupulous content generators. Based on a dataset of real-life fakes analyzed by fact-checking professionals, we identify the most common types of triggered emotions to be used as a taxonomy for further annotation.
The manipulative content, ranging from propaganda to hate campaigns, fake news, trolling and similar, is becoming more and more widespread, threatening our access to truthful and unbiased information and thus undermining our rights to make informed decisions as individuals and as members of the society. While there is a growing body of multidisciplinary research on identifying untruthful content, there is still very limited understanding of the manipulative techniques the unscrupulous content writers employ to convince the reader and ultimately change their point of view. We believe that this manipulation occurs through multiple channels: careful selection of fact-checkable and non-fact-checkable claims, biased yet seemingly solid argumentation/analytics, multimedia support and, most importantly, emotional component. Our current study focuses on emotion triggering – a technique widely used by content writers: when the reader is experiencing a strong feeling, they become less critical and thus easily overlook deficiencies in the argumentation and get more prone to manipulation.
Fig. 1 shows examples of manipulative textual content with strong emotional triggering. In (1a), the message makes a very strong appeal to fear, by mentioning HIV. Moreover, this triggering efect is intensified by mentioning "children". The fact-checking report 1 informs the reader that the COVID-19 vaccines do not contain any HIV material, but do contain other lipids to protect the mRNA. The distressed users, however, might not trust this information fully, due to such a strong emotion as a fear for their children’s health. Ex(a) from Facebook (b) from Twitter ample (1b) shows a typical manipulative message not addressed properly by the state of the art verificationoriented technology. The message combines a verifiable true claim ("Murkowski, Collins, and Romney voted for Ketanji Brown Jackson") with a statement that looks like a similarly factual claim ("Murkowski, Collins, and Romney are pro-pro-pedophile"), but in reality is an explanation/opinion ofered by the writer. This triggers a rather strong anger at the powers/authorities under the spotlight, their presumed hypocrisy and their presumed (lack of) values. Here again, the triggering is intensified by bringing up a topic related to children. The fact-checking report 2 debunks this claim stating that "Sens. Murkowski,
-taylor-greene/greene-twists-logic-and-facts-pedophilia-charge-a Collins and Romney have clear track records of acting our study, we focus on real-world data, analyzing fakes against child exploitation, whether online or in person" generated with a real purpose, albeit not always clear and, moreover, the implied related accusations of Judge (and not necessarily malicious).
Johnson are "misleading". However, a reader driven by Giachanou et al. [2] address the impact of emotional emotions, might still remain manipulated ("no smoke signals on the credibility for fake news. This study shows without fire"), even if only partially. that emotional signals are extremely important as the
These examples show that fake news are way more emotion-aware system outperforms their baseline by a complex than simply untrue messages. They might com- large margin. This work, however, focuses on already bine true facts with partially false or impossible to check existing generic resources for defining emotions: either statements, provide biased analytics on top and add very lexicons of terms expressing specific sentiments or a corstrong emotional messages to manipulate readers. We pus of triggered sentiments with labels corresponding believe that while the NLP community is making an im- to five diferent Facebook reactions (love, joy etc). We pressive progress on the fact verification task, our un- believe that findings of Giachanou et al. [ 2] are extremely derstanding of other phenomena related to manipulative important and show that emotions triggered by manipcontent are still rather limited. The goal of our study is ulative content should be studied in a more principled to get a deeper and more realistic insight on the emo- way. We hope that our study could help define a more tional component of fakes. As a first step, we provide a triggering-oriented approach to emotions. qualitative data-driven analysis of emotion triggering. Several recent papers analyze emotion triggering as a
The contributions of this study are as follows: (i) we part of propaganda persuasion techniques. For example, provide data-driven analysis, focusing on real data, com- Da San Martino et al. [3] develop a taxonomy of propabining original (source) fakes and high-quality reports ganda techniques, whereas Piskorski et al. [4] propose by professional fact-checkers thus improving our insight, a shared task build upon this taxonomy. These studies (2) we aim at a taxonomy of triggered emotions cover- do not, however, focus on emotions specifically. For exing a majority of real-life fakes, departing from more ample, Piskorski et al. [4] group most emotions under theory-oriented labels and (3) we analyze perceived (i.e., the "manipulative" category, while some others (e.g., "aptriggered) emotions, as opposed to the common focus on peal to patriotism/pride" also known as "flag-waving") expressed emotions, as we believe that induced sentiment are classified based on reasoning fallacies associated with plays a more important role in manipulation/persuasion. them. Moreover, these studies focus on unscrupulous persuasion techniques introduced in the theoretical studies, e.g., on (in)formal argumentation fallacies. We advocate 2. Related Work a more data-driven approach: the phenomenon of manipulative online content is rather new and evolving, thus, There is a rapidly growing body of studies on online mis- it is not clear how well more traditional labels describe information detection. These works, however, mainly it. We aim at decoupling emotions from (fallacious) argufocus on the verification part ( Is the information truth- mentation and improving our insight into the variety of ful – i.e., supported by the evidence?), and not on the sentiments the content writers appeal to. persuasion (How is the information presented to manip- Finally, some of the discussed triggers, especially "fear", ulate the reader?). Thus, most computational models have been a focus of multidisciplinary studies, ranging are built upon the FEVER corpus [1]: a large collection from psychology (see an overview in [5]) to ethics [6]. of true/false claims generated by human annotators, an- At the same time, there exist much less research on more notated as supported/refuted/unknown by the evidence. complex triggers.
FEVER claims are originally extracted from Wikipedia (true) and then mutated (false). An example FEVER claim is "Shakira is Canadian”. Note a strong diference be- 3. Data tween this example and (1a-b) above: the Shakira claim was generated with no manipulative purpose in mind and does not involve any specific persuasion/triggering techniques. The claims in (1), on the contrary, have a strong manipulative component and have been generated with a genuine unscrupulous intent. For example, (1a) cannot be fully accounted for by a simple mutation: the choice of "HIV" is crucial to induce fear and thus the same manipulative efect would not be achieved if "HIV lipids" were replaced with any other kind of lipids. In
of developing reliable annotation guidelines that provide good coverage for triggered sentiments. We have therefore opted for in-depth analysis of a small number of documents. Our analysis relies on both the documents themselves and their corresponding fact-checking reports by PolitiFact. This way, we make sure that we ourselves do not fall victim to the manipulation techniques and can assess them impartially.
We rely on PolitiFact reports from mid-March to midg/
May 2022. We filter out fakes that originate on TV, intervies and other sources ourside of social media. This leaves us with 160 "claims", each associated with their corresponding social media post and high-quality PolitiFact report, written by professional fact-checkers. We then annotate them with metadata, overall professional fact-checking judgement, atomic fact veracity, reasoning flaws (e.g., "simplification") and, most importantly, triggered emotions. The latter is done in data-driven bottom-up fashion, with the set of considered emotions under constant refinement.
Appeal to Fear is the most studied and widely used manipulative technique: by making the readers believe that they are in imminent personal danger, the author can influence their attitude toward the message, suppress critical thinking, instill doubt and ultimately manipulate their behavior. There are multiple studies showing the eficacy of this persuasion technique, see [ 5] for an overview. From the data-driven perspective, however, it is not always easy to define the boundaries of "personal danger".
Thus, our example (1a) shows a clear case of appeal to fear, since the governments’ policies strongly suggest all the population to be vaccinated. Consider our example in Figure 2. This post informs a rather limited group of people of the alleged imminent danger, thus inducing fear. However, when going viral, it might have a feartriggering efect on the whole population, stating that the authorities are able to and, in practice, do employ carcinogenic chemicals against humans. choices, because everybody is doing so. For example, bandwagon is commonly used in advertisement, where a In this section, we discuss the emotions triggered in ma- lot of products are marketed as a must since everybody nipulative online messages. We start with commonly buys them. Surprisingly, we haven’t found a single exacknowledged and studied triggers, such as "fear" and ex- ample of an appeal to common practice in manipulative pand the label set to accommodate data-driven categories online content in our data. However, we have observed not suficiently covered in the literature. the opposite appeal: the authors urges the reader not to follow the common practice, appealing to their uniqueness and superiority.
Figure 3 shows a very common example of appeal to uniqueness/superiority: the authors state that while most people are brainwashed by mainstream information channels and left to believe in some fake reality, the readers should – and is definitely capable of – avoid falling for the same trap. This boosts the readers’ ego, improves their trust in fake news while, at the same time, undermines mainstream media and paves the path for various conspiracy theories. We have observed this opinion framing strategy on a variety of polarized topics, ranging from vaccination to government spending or climate.
While this direct appeal to readers’ uniqueness/ego is very widespread and seemingly rather efective, we are not aware of any in-depth studies of this phenomenon, especially from the NLP perspective.
nique triggering strong antagonising feelings between Bandwagon and Anti-bandwagon. Another rela- "us" ("the good people") and "them" ("the corrupt powers: tively widely studied technique is an appeal to common government, rich, media etc"). Populism plays an ever practice/belief ("safe choice"), also known as "bandwagon rising role in the modern political discourse, afecting and fallacy". This technique urges the reader to adopt specific polarizing people’s views. While it is widely studied in political science, the related psychological mechanisms are still underresearched [7]. We have observed multiple cases of appeal to populism throughout the data.
Thus, in a Facebook post on Figure 4, the author makes it pretty clear that the rich are responsible for and beneifting from the sufering of "us" – in this specific case, the formula milk crisis. The same strategy is used throughout our data to implicate diferent kinds of powers: the administration, the rich or the media and sometimes a mixture or just a generic/underspecified "power". The appeal to populism is often combined with other emotions: for example, triggering the fear or unfairness/injustice for the outcome of "their" actions as well as uniqueness/ego for uncovering the plot.
(b) undermining trust ("everybody lies"), Facebook
Appealing to (Un-)Fairness is a very strong technique, often used in combination with appealing to populism (see an example on Figure 5). eficient triggers in manipulative content, an urgent at
In some cases, the authors trigger this sentiment in a tention from the research community, including NLP, positive way, inviting the reader to celebrate the victory might have a considerable impact and help fight online of fairness. misinformation.
In both cases, however, the content writers trigger a very strong and deep desire for (social) justice, that deflecting the readers’ attention from inconsistencies and Appeals to honesty are very popular in manipulative misrepresentation in the presented facts and arguments. content. This category includes allegations of hypocrisy,
To our knowledge, appealing to fairness is acknowl- inconsistency or accusations of lying, aimed at casting a edged as a powerful technique by a variety of practicing doubt on specific persons (Figure 6a). professionals, e.g., negotiators or copywriters. However, However, a far more widespread appeal to honesty there is still virtually no research on this specific emotion. is the technique where some information coming from We believe that since this is one of the most frequent and mainstream media or oficial sources is presented as a lie, with no clear and specific purpose (Figure 6b). This type of fakes promote the idea of everything being unreliable and slowly but steadily push the readers to become less critical of various conspiracy theories.
In this section, we discuss the distribution of triggered emotions in the manipulative content collected and analyzed by PolitiFact. Most importantly, we have observed that a vast majority of fakes trigger emotions: 128 documents (80%) in our collection unambiguously aim at afecting the readers’ emotional state. For comparison, only 88 documents (55%) contain clearly untrue atomic facts and 95 documents (59%) employ fallacious argumentation. We believe, once again, that these numbers suggest that the eficient approach to manipulative content analysis should expand from mere fact verification to modeling fallacious argumentation and emotion triggering.
Trigger populism fear (personal) fear (empathy) fairness honesty values uniqueness disaster other #documents 62 18 16 27 22 15 18 8 6 Table 1 Values. Certain online posts make appeal to values, Triggers in the PolitiFact data. promoting responsible choices or condemning someone else’s behavior as unethical. This type of triggering is Table 1 shows the document statistics for each of the often used in polarized contexts to attack the opposite triggers discussed in this section. The most common catside and thus misrepresent their position (Figure 7). egory is populism, which might be due to the political
Appeals to values are often used as a part of the re- orientation of our domain. Note that populism is also duction/simplification fallacy: the fact-checkable facts in relatively easy to identify: our preliminary experiments the message are true (e.g., the statement above is focused show very little disagreement on this label. Appeal to fear on "A National Terrorism Advisory System bulletin", ad- is the second most popular category: unscrupulous condressing the threats of online misinfromation), yet their tent writers are well aware of its eficiency. Annotating interpretation is fallacious and manipulative, introduc- it reliably, however, requires extra work on guidelines, ing loaded lexica ("attack", "criminalize") to misrepresent since the boundaries between personal fear and empathese facts, substituting objective reporting with moral- thy for others are very subjective. Depending on the istic judgement. This type of fakes are therefore particu- definition of fear, we observe 11-21% of such documents. larly problematic for state-of-the-art NLP models, based Fairness, honesty and values are also rather common. on fact verification. Finally, only 6 documents (4%) appeal to other emotions that are not covered by our taxonomy.
Disasters. We have observed a large number of fakes The same post can trigger multiple emotions. In parfocusing on natural and man-made disasters. Media cov- ticular, appeals to populism ("they are bad") are often erage of disasters has been shown to attract a large num- combined with any other trigger ("they are bad: they ber of readers/viewers, triggering a wide variety of inter- are threatening our existence, imposing unfair policies related negative emotions, in particular fear and anxiety and lying"). A rather common combination through[8]. Unscrupulous content generators abuse the users’ in- out all the fakes we have analyzed is "they (the meterest in catastrophic events for their own purposes (e.g. dia/administration) are lying, but you are smart and click-bait). We label this specific type of fear/anger as you don’t believe them, we will tell you the truth" (anti"disaster" for the lack of better term, since a more precise bandwagon + honesty + populism). Note that this trigger analysis is still an open research issue in psychology. makes it very dificult to respond to and counter the efect of manipulative content: if the readers are convinced that "they" are lying, they can simply discard a fact-checking report since they perceive fact-checkers as liars (paid by "them") or, at the very least, brainwashed (by "them").