This paper presents an in-depth exploratory quantitative study of the interaction between multimedia and textual components in online manipulative content. We discuss relations between content layers (such as proof or support) as well as unscrupulous techniques compromising visual content. The study is based on fakes reported and analyzed by PolitiFact and comprises documents from Facebook, Twitter and Instagram. We identify several pervasive phenomena currently, afecting the impact of manipulative content on the reader and the possible strategies for efective de-bunking actions, and discuss possible research directions.
understanding of the way the authors integrate multimedia into their content: most research so far has focused Manipulative online content (fake news, propaganda, on a specific component and not on their interplay. Our among others) is growing at an alarming rate, hinder- study aims at identifying the role of multimedia part of ing our access to truthful and unbiased information and manipulative messages. thus threatening principles of the democratic society. Figure 1 shows some examples from potential fakes The problem has been addressed by professional jour- analyzed by PolitiFact. We observe diferent relations nalists, who – with the help of crowd-workers – fight a between the text and the image. In particular, in (1a), never-ending battle to prevent information contamina- the video is supposed to prove the claim by providing tion. To enable a large-scale response to the misinforma- direct evidence, whereas in (1b), the image provides a tion threat, the AI community has invested a considerable support (appeal to authority). In (1c), the image is a viefort into building competitive models for identifying sual paraphrase of the claim, enhancing its appeal but non-transparent content, such as false claims or altered not providing extra proof, support or informational mavideos (deep fakes). However, we still lack a thorough terial. Finally, in (1d), the photo is an illustration that, understanding of the manipulative content and multi- while depicting the discussed person, does not aim at ple aspects afecting its perception and impact on the being relevant to the claim’s veracity or impact. While reader. This paper aims at an in-depth analysis of one of understanding the relation between the image and the such aspects, namely, the interaction between diferent text is interesting from the scientific perspective, it is (multimedia) layers of the manipulative message. More also a crucial prerequisite for eficient and meaningful specifically, we study the semantics underlying the re- fact-checking response. For example, if a supposed proof lation between multimedia and textual parts of the fake is a compromised photo, the response should highlight news. Our study is based on around 800 fakes from Jan- this fact (e.g., the video in (1a) has been cropped misuary till September 2022, as identified and analysed by representing the quote, which should be highlighted in PolitiFact.1 the fact-checking report). On the contrary, if a compro
Multimedia content, such as videos, reels, photos, mised photo is used as a mere illustration, the efective screenshots or images is becoming increasingly popu- fact-checking report should focus on the textual claim lar in social media: it is an appealing and powerful way per se. of expressing and/or enhancing one’s message. Never- Another important angle is the issue with the multitheless, as a scientific community, we still have little media part. In our example, the video in (1a) is cropped. On the contrary, (1b) represents an authentic screenshot, yet, it has been miscaptioned by the claim: an older content, irrelevant for the current events/topics, has been CLiC-it 2024: Tenth Italian Conference on Computational Linguistics, Dec 04 — 06, 2024, Pisa, Italy $ uryupina@gmail.com (O. Uryupina)
© 2024 Copyright for this paper by its authors. Use permitted under Creative Commons License repurposed. 1PolitiFactAt(trhibtuttipons:4/.0/Iwntewrnawtio.npaol(lCiCtiBfYa4c.0t)..com/) is an independent journal- The current paper focuses on these two aspects to anistic agency and one of the most experienced fact-checking orga- alyze empirically the interplay between multimedia and nizations, providing detailed analytics for non-transparent online textual components in fake news, as identified by Politicontent since 2007.
(a) Biden to teachers: “They’re not somebody else’s
children. They’re yours when you’re in the classroom."
(VIDEO)
(b) Now you know why there’s suddenly "a formula
shortage". The new age robber barrons have conveniently
invested in some unholy breast milk made from
human organs.
(c) In honor of #TaxDay, I remind you that Governor Evers
wanted to increase your taxes by $1 billion just for
heating your homes. Instead, Republicans cut your
taxes by more than $2 billion.
(d) Italian football agent Mino Raiola has died after
suffering from an illness. RIP
Fact. To this end, we reannotate the PolyFake dataset [
While fact checking has been receiving an increasing These studies have paved the way for evaluation
camamount of attention recently both from NLP and Vision paigns and benchmarking resources, for example, [
A breakthrough approach by Vempala and Preoţiuc- the annotators are not professional fact-checkers: while
Pietro [
In a recent survey, Mubashara et al. [
Our study aims at an in-depth exploratory analysis well. This way we make use of the wealth of analytics of the multimodal online content. To this end, we focus provided by experienced professional fact-checkers by on more specific labels to describe the relationship be- encoding it in more structured annotation labels. tween diferent layers/modalities. We extend the scope PolyFake covers fakes from diferent social media of our study to cover all the three major platforms (Face- (Twitter, Facebook, Instagram, TikTok, Threads and book, Instagram and Twitter). Moreover, our input is YouTube). Note that manipulative content often gets not only the claim per se, but the professionally created propagated across platforms through re-posts, sharing, fact-checking report from PolitiFact. In our experience, linking or just copying. For example, a large proporPolitiFact reports contain a wealth of information about tion of Facebook videos originates from TikTok (in this online manipulation: as opposed to 2-3 binary labels of case, PolitiFact typically analyzes the Facebook message, common NLP fact-checking benchmarks, PolitiFact char- hence a low number of TikTok entities in the table). In acterizes each claim with 1-3 pages of analytics. This the following study, we omit TikTok, YouTube and Teleanalytics, however, comes in a free textual form. While gram as largely underrepresented categories with rather it might be still impossible for the NLP community to en- straightforward patterns. code these reports for building high-quality fact-checking systems, we believe that we should at least learn from 3.2. Multimedia and Layered Content them to get better insights, stop trivializing the task and highlight understudied, yet impactful, subtasks.
3.1. PolyFake
Our study is based on the PolyFake dataset [
related to online manipulation: from fallacious/propaganda reasoning to emotive appeals, factual veracity etc. Current study aims at an in-depth analysis of a specific angle. The Appendix discusses the distribution of veracity labels across PolyFake documents.
Layer Types. Table 1 shows the distribution of diferent media types for each platform. We have identified several types of layered content: parts of the message rendered together with the initial post. The most common ones are videos (including reels), photos and screenshots (typically, complex visual objects combining textual content with photos/images and referring the reader to a diferent source). We have also observed images (infographics, maps or drawings), links (this content typically is rendered with a photo/stillshot, yet it explicitly points to a diferent online location, for example, promotion website) or threads (characteristic for Twitter, this type of layering helps to contextualize the message). On rare occasions, social media posts might contain more than role content anchor proof support paraphr. context illustr. action other total one extra layer (e.g., videos and photos). dia levels play in PolyFake documents. We distinguish
Most importanly, only 18% of PolyFake documents are between the following roles: content (the essential part of purely textual: adhering to the popular adage that a pic- the content is presented on the multimedia layer, whereas ture is worth a thousand words, manipulative content the textual layer just adds minor details or suggests opincreators use visuals for a variety of purposes, from in- ions), proof (the multimedia layer ofers a physical proof – creasing the outreach to improving the credibility. More- cf. Example (1a)), support (the multimedia layer provides over, the prevalence of multimedia content is way more some material to support the claim, from a reputable critical for Facebook and Instagram – the two platforms source – cf. Example (1b)), paraphrase (the multimedia not typically addressed by NLP practitioners. This alone layer paraphrases the claim without adding any extra suggests that we need to pay much more attention to joint angle – cf. Example (1c)), context (while the textual claim models and start with deeper understanding of relevant is generally self-contained, it cannot be interpreted withphenomena. out the context given by the multimedia part (e.g., the
A large percentage of documents are re-using or claim contains pronouns and the image presents their spreading already existing information. This is true for referents)), illustration (the multimedia layer shows some screenshots (21% in total) and links (5%), but also for objects/persons mentioned in the claim without any conmany videos – only very few videos represent original nection to its semantics – cf. Example (1d)) and action content. While there exist some studies on identifying (the multimedia layer suggests an appropriate reaction to previously fact-checked claims, they are restricted to the the claim, for example, a scam website). Finally, a rather textual content. We believe that a more complex multi- common role for videos and photos is anchor: in such modal approach would be beneficial here. cases, the textual claim is about the multimedia itself (for
For presentation issues, in what follows we merge our example, "the sharpest image of the sun ever recorded."; underrepresented categories link, image and thread with here, the multimedia is not compromised per se and the roughly functionally similar major categories screenshot, textual claim contains no falsehoods about the world, yet photo and screenshot respectively. the combination might be very misleading.
Layer Roles. Table 2 shows diferent roles multime- In more than half of the documents, multimodal layers provide essential content. This is true for all the media types (videos, photos and screenshots). We have observed several possible factors contributing to this efect: in general, social media users tend to repost existing "fancy" content and not create their own texts. Even in authentic self-created posts, the message is often put in a visual, whereas only some emotions are added in a text. We believe that there is a wide variety of potential reasons for this behaviour (e.g., videos and photos get more likes, whereas texts are mostly ignored by peers), requiring a more specialized study.
Almost one third of multimedia layers, especially videos, supposedly present proofs. Such compromised proofs are out of reach for the modern evidence-based automatic fact-checking: while a fact-checking model can provide extensive evidence to refute a claim, the user would still trust the video/photo and not the model. Human fact-checkers address such proofs from a diferent, more promising, perspective: they try to explicitly attack and debunk the proof. We believe that this is a very important and largely unaddressed research direction.
Issues with multimedia layers. Finally, we have identified the most common unscrupulous techniques relevant for multimedia layers. Those include: crop (essential part(s) of the original message are omitted to render it out of context – cf. Example (1a)); miscaption (while the image/video is authentic, the textual claim misleads w.r.t. some crucial details, e.g. events or timeline – cf.
Example (1b)); altered/fake (the image/video has been altered – beyond cropping – with the specialized software, including deep fakes); misperception (the image/video is – deliberately or not – deceiving because of its low quality, unclear angle, optical efects etc); noproof (the – typically long – video does not contain any components relevant for the claim); falsehood (the video/image is authentic, yet its content is untrue – i.e., the textual claim spreads the original fake generated by the video/image); and explain (the textual part explains – misleadingly – what we are supposed to see in the video, often of a rather low quality).
Table 3 summarizes the distribution of problematic issues across the three main multimedia types, showing several trends. First, video layers provide more possibilities for unscrupulous content generators: cropped, otherwise altered or low quality videos are pervasive in manipulative content. While most of the research focuses on images, they do not exhibit such a variety of manipulative strategies. Screenshots – authentic or fake – are largely used to disseminate falsehoods. At the same time, an increasing amount of authentic videos, mostly originating from TikTok, is created to spread falsehoods and promote "critical thinking" (i.e., conspiracy theories as opposed to rational argumentation). These remain largely understudied, despite their large impact on the audience. Another rather unstudied area are explanatory claims: authentic videos/photos accompanied by misleading explanations of what we see and what it means; in such cases, the factual component might be non-compromised, yet the biased explanation makes the whole message an impactful and hard to debunk propaganda tool. Finally, unlike videos and screenshots, most photos represent true authentic information – the textual claims either rely on them as illustrations or use them as building blocks to support fallacious argumentation.
We have presented an in-depth analysis of the interaction between textual and multimedia components of compromised social media documents. We have identified several high-impact issues, insuficiently studied by the community at the moment. These include the interaction between diferent modalities, the role of the multimedia part and its impact on selecting the successful factchecking strategy, the diference between platforms and media types (current NLP studies predominantly focus on Twitter and images) and the importance of a more principled approach to content re-use. We hope that this study, motivated by human fact-checking expertise, can sparkle a meaningful discussion and improve automatic modeling.
FC label pants-on-fire false mostly false half true mostly true true total
claims: even though PolitiFact occasionally fact-checks statements that turn out to be true, most of their materials are "false", "mostly false" or even "pants on fire". Moreover, even true claims often exhibit signs of user manipulation. In this appendix, we show statistics for fake vs. true content in PolitiFact reports (Table 4).