This study examines narrative construction patterns in disinformation and trustworthy news articles through analysis of English articles covering international events from 2015 to 2023. Using a methodological framework that combines large language models and knowledge graphs, di!erences between disinformation and trustworthy news articles were examined. The research revealed distinct patterns in narrative strategies: disinformation sources predominantly employed emotional language, moral absolutism, and cultural threat narratives, while trustworthy sources maintained measured tones, methodological transparency, and analysis with rich context. Knowledge graph analysis demonstrated that disinformation narratives exhibit fragmented structures that rely on hubs with emphasis on abstract connections, whereas trustworthy narratives show more balanced, interconnected networks with concrete relationship patterns.
The increase of disinformation in the media presents signi"cant challenges to public understanding
and democratic discourse as well [
In recent years we have witnessed a lot of coordinated disinformation campaigns, particularly
concerning major geopolitical events [
Also, Knowledge Graph-based approaches have emerged as powerful tools for enhancing semantic precision and causal relationships in narrative analysis, such as in [13]. Causal relationships are essential for understanding story development complex news narratives [14]. Besides, in the domain of bias detection, it was demonstrated that incorporating frame-based knowledge with text models signi"cantly improves the detection of bias and stance in news narratives, providing a more nuanced understanding of narrative framing [15]. The construction of domain-speci"c Knowledge Graphs for news has been another signi"cant development, such as in [16], where developed comprehensive ontology in news contexts enabled sophisticated applications in bias detection and narrative synthesis.
Furthermore, cross-cultural narrative analysis has revealed interesting patterns in how di!erent societies frame similar issues, such as climate change narratives across cultures [17], where LLMs were used in "nding distinct emphases in North American Chinese sources, highlighting the importance of cultural context in narrative construction and interpretation. In addition, it was demonstrated that Temporal Knowledge Graphs combined with unstructured data from news articles can improve temporal reasoning and event prediction [18], expanding applications in event-centric narrative reasoning, particularly valuable for tracking narrative evolution over time.
The impact of narratives on public sentiment has been quantitatively studied, e.g., in [19], who found signi"cant correlations between exposure to pessimistic news narratives and heightened negative public sentiment. Moreover, event-centric approaches to news analysis have been advanced in, e.g., [20], who developed methods for retrieving event-relevant news articles to support narrative continuation tasks, enhancing the construction of cohesive narratives from fragmented events. Finally, recent developments in processing long narratives have addressed previous limitations in text analysis, e.g., by introducing a novel architecture integrating dynamic Knowledge Graphs with LLMs [21], which helps to improve story comprehension capabilities.
These brie#y introduced advances in computational narrative analysis, among others, have signi"cantly enhanced our ability to understand and analyze news media content. This study aims to identify and analyze the distinctive patterns in narrative construction in disinformation and trustworthy news articles to examine how sociocultural elements are framed and deployed in each type of content in order to evaluate the implications of di!erent narrative approaches for public understanding and social cohesion.
For this study part of the dataset for multilingual detection of pro-Kremlin disinformation in news articles [22], containing data on disinformation and trustworthy news articles, was used. As the full text of the news articles was not publicly available, to reconstruct the dataset URL links of the articles were used with Di!Bot API1 (free for academic purposes) to acquire them. Only articles in English were selected. Some articles attempted to acquire were no longer available or have been modi"ed. So, although the full dataset has 18 249 articles in 42 languages that span over 8.5 years (since 2015) and cover 508 topics, after all the "ltering and cleaning the part used in this study was made of 308 disinformation news articles (most frequent sources: RT, TASS, Sputnik) and 302 – trustworthy news articles (most frequent sources: BBC, the Guardian, Polygraph.info, covering such topics as Western / NATO -Russian relations, Russia-Ukraine con#ict, allegations of chemical weapons, Syrian civil war, geopolitical tensions, disinformation/propaganda narratives, conspiracy theories). Labeling news articles as ‘disinformation’ and ‘trustworthy’ is based on the original dataset.
1. Step 1. Summarization: to simplify construction of knowledge graph to be used in later stages, the cleaned news articles were summarized with DistilBART-CNN-12-6 model2, which is a distilled version of the BART model, speci"cally optimized for text summarization tasks [23]. To learn language representations, BART and its derivatives combine a transformer-based design with a denoising autoencoder.
For generating summaries, all the articles in the dataset were truncated to a maximum input length of 1024 tokens. The generated summaries were between 50 and 128 tokens in length. To ensure quality, summaries were manually inspected and evaluated. These summaries were integrated into the dataset. 2. Step 2. Relational triple extraction: to extract triples, we applied REBEL (Relation Extraction By End-to-end Language generation) [24], which uses a sequence-to-sequence architecture based on BART for extraction of relational triples. It combines Named Entity Recognition (NER) and Relation Classi"cation (RC) into a single task and covers >200 relationship types.
The pre-trained version without "ne-tuning was used and it was applied to summaries of the articles from Step 1. The triples (Entity-Relation-Entity) were extracted for each sentence and then aggregated across all sentences in each summary, ensuring consistency and removing duplicates. The extracted triples were manually validated for correctness and integrated into the dataset to be used for constructing a knowledge graph. 3. Step 3. Augmentation with sociocultural data: ChatGPT-4o was used for augmenting data with cultural references, idioms and collocations, sentiments and emotions, sociocultural context and relationships between cultural references and sentiments-emotions. The following prompt was applied for this task:
Text: [Insert article text here] Instructions: From the above text, identify and list: - Cultural references - Idioms or expressions - Sentiments and emotions expressed - Sociocultural context or factors influencing the narrative - Relationships among these elements Example Output: - Cultural References: - "Global Climate Coference" - Reference to international efforts on climate change - Idioms/Expressions:
- "Leading by example" - Sentiments/Emotions: - Optimism - Concern - Sociocultural Context: - Emphasis on environmental responsibility in Country A - Economic concerns in industrial sectors - Relationships: - The president’s pledge reflects Country A’s cultural emphasis on sustainability. - Industrial lobbyists’ criticism highlights societal tension between economic growth and environmental conservation.
To validate and/or enrich the results, Wikidata3 was used for cross-referencing ChatGPT-4o outputs using SPARQL queries. A manual review of the extracted sociocultural data was performed as well to con"rm accuracy and relevance. This data was integrated into the dataset and was used in the knowledge graph as well. 4. Step 4. Knowledge graph construction: For constructing the knowledge graph, we used extracted relational triples. In this graph structure entities (nodes) of the triples were linked by their relations (edges). Metadata and extracted sociocultural data were included in the graph as well. Python libraries NetworkX4 and rd#ib5 were used for this task.
The schema of the knowledge graph focus on key entities – people, organizations, locations, events, articles. These entities are interconnected through the relationships implemented in REBEL model, such as locatedIn, leaderOf, memberOf, relatedTo, participatedIn, etc. Furthermore, each entity have associated metadata, e.g., source article, the article’s publisher, publication date, class (’disinformation’ or ’trustworthy’), sentiment. Additional properties include sociocultural context, such as cultural references, idioms, and topics that are associated with the entities. Events are also described by a point in time. This schema was validated via heuristic rules. 3Accessible at https://www.wikidata.org/ 4Accessible at https://github.com/networkx/networkx 5Accessible at https://github.com/RDFLib/rd#ib 5. Step 5. Comparative analysis: the analysis of narrative construction and sociocultural framing in disinformation and trustworthy news articles was performed by querying over knowledge graph in natural language with an LLM [25, 26, 27], for which Claude-3.5-Sonnet-200k was chosen due to its long context window. Thus prompting was applied to extract key patterns with supporting examples. The prompts consisted of a series of analytical questions, which were applied for querying over graphs constructed from disinformation and trustworthy news articles separately (Table 1). The extracted patterns were inspected manually and a qualitative analysis was performed for comparison.
Stage Prompts 1. Initial Ex- "I want to perform an analysis of how sociocultural factors influence narrative conploration and struction in news articles. The knowledge graph I want to use is attached. What Pattern Iden- are the key patterns? Can you provide an example for each of the key patterns and tification present the results visually, e.g., mermaid diagram?" 2. Pattern "Could you elaborate on these patterns and provide more specific examples from the Elaboration knowledge graph?" 3. Cultural "Analyze the cultural context patterns in more detail from the knowledge graph, Framework breaking it down into key cultural frameworks and their influence on the narrative Analysis construction." "Could you elaborate on the cultural framework concept as represented in the knowledge graph?" 4. Specific Fo- "Please examine in more detail cultural di!erences in the narratives." cus Areas "Please analyze how cultural di!erences are systematically used to frame political conflicts in the knowledge graph." "How do historical narratives support current political positions according to the data?" "Elaborate on how cultural frameworks shape narrative construction based on the knowledge graph data." "What are the key findings from this analysis? Please provide specific examples from the knowledge graph." "Could you provide more context from the knowledge graph to support these findings?" To extract key patterns of narrative construction and sociocultural framing and to compare their di!erences with respect to disinformation and trustworthy news articles, 2 knowledge graphs following the steps in Section 3 were constructed. The summary of both knowledge graphs is presented in Table 2.
Both graphs share similar entity categories, however, the disinformation graph emphasizes abstract concepts and a broad “other” category, while the trustworthy graph highlights speci"c events and veri"able concepts, which re#ect di!erent focuses. Regarding relationships, in the disinformation knowledge graph among the most common ones are diplomatic_relation and location which often blur speci"city by including entities in broad, abstract contexts. The trustworthy graph focuses on more concrete links like shares_border_with and point_in_time. While both graphs feature political and diplomatic relationships, the disinformation graph exhibits a broader variety of abstract connections, whereas the trustworthy graph demonstrates a more structured and evidence-based relationship network.
Both graphs have a core-periphery structure [28] with Russia and Ukraine as central hubs. The disinformation graph clustered around military and political themes and appears fragmented and hub-reliant. The trustworthy graph, on the other hand, is more balanced and interconnected with denser subclusters (diplomatic, military, health), which shows clearer thematic distinctions.
This section presents summarized patterns of narrative construction in disinformation and trustworthy news articles. The patterns were extracted by querying over knowledge graphs in natural language with an LLM (Claude-3.5-Sonnet-200k). A summary of the "ndings is presented in Table 3. Disinformation Narrative Patterns. A set of interconnected narrative strategies was found prevalent in disinformation news, revealing e!orts to manipulate emotions, construct rigid worldviews, and dismiss alternative perspectives, which were often used to enhance polarization and reinforce speci"c agendas [29]. One dominant characteristic of disinformation narratives is the use of the binary framework, which involved a reliance on absolutist claims and stark oppositions, often framed as "us versus them" – "us" (e.g., Russia, China, Belarus) as allies with positive framing, while "Them" (e.g., NATO, EU, US) as adversaries with negative framing. An example of this pattern would be portrayal of Russia as the betrayed party and the West as untrustworthy. These narratives present issues as moral dichotomies and use emotional triggers such as moral outrage or cultural anxiety to amplify the persuasive impact as outlined in [30].
Another recurring pattern is cultural identity framing, where cultural di!erences are presented as "xed and inherently oppositional (e.g., Russian culture was depicted as rooted in tradition, unity, and moral values, while Western culture is described as declining). Narratives also highlighted civilizational clashes and the notion that traditional values are under threat (e.g., Russian identity is framed as under siege by Western liberalism). Also, selective historical interpretations were employed to support these claims, reinforcing political agendas. Furthermore, another frequent strategy involved authority claims, where disinformation sources asserted unquali"ed authority over the truth while discrediting opposing viewpoints. This approach often incorporated conspiracy theories and selective citations that align with the desired narrative, thus aiming for distrust in mainstream or evidence-based sources [31].
Finally, disinformation narratives engaged in crisis portrayal, framing issues as existential threats to cultural identity or civilization. Instead of presenting crises as practical challenges that required a solution, they were depicted as moral or cultural emergencies via emotionally charged language, moral absolutism, and selective evidence to heighten fear, urgency, and defensiveness. More extensive qualitative studies, such as [32], support these results.
Trustworthy News Patterns. In contrast, trustworthy news sources employed narrative patterns that aligned with journalistic standards, i.e., fostering critical thinking and nuanced understanding via factual accuracy, emotional balance, and the acknowledgment of complexity as it has been outlined in
Trustworthy News Patterns 1. Measured emotional framing • Balanced emotional content • Factual documentation • Analytical context • Journalistic objectivity 2. Cultural complexity recognition • Evolving cultural identities • Shared values acknowledgment • Multiple cultural perspectives • Contextualized interpretation 3. Multi-stakeholder perspective • Multiple viewpoints • Diverse stakeholder voices • Complexity recognition • Balanced representation 4. Evidence-based documentation • Multiple authoritative sources • Independent verification • Documentary evidence • Cross-referenced information 5. Solution-focused approach • Practical policy proposals • Conflict resolution mechanisms • International cooperation • Constructive developments e.g., [33]. A key feature was a measured emotional framing, where emotional impact was acknowledged and used to highlight human impact, but balanced with factual documentation and analytical context (e.g., coverage of Ukraine in relation to Russo-Ukrainian war emphasizes "resilience" (emotional impact) while documenting military and geopolitical developments (factual context)).
Trustworthy sources also recognized cultural complexity via highlighting the changing and sophisticated nature of cultural identities. Rather than portraying cultural di!erences as "xed or oppositional, they focused on shared values, presented multiple perspectives and provided a context for understanding cultural dynamics (e.g., reports on Crimea include narratives about cultural preservation for Crimean Tatars, Russian heritage, and Ukrainian sovereignty, illustrating the region’s complex and interconnected identities). Also, trustworthy sources included multi-stakeholder perspectives, presenting diverse viewpoints, re#ecting the complexity of issues and o!ering practical frameworks for solutions, thus promoting inclusivity and encouraging audiences to engage with di!erent perspectives as it has been described in, e.g., [34].
Evidence-based reporting was another key pattern in trustworthy news which included reliance on multiple authoritative sources, independent veri"cation, and cross-referencing. Such practices increase credibility and ensure that narratives are grounded in factual accuracy [35]. Finally, trustworthy reporting applied a solution-focused approach, emphasizing practical policy proposals, con#ict resolution mechanisms, and frameworks for international cooperation, which aimed to inform and empower audiences with a focus on constructive developments and actions.
This work-in-progress highlighted how narrative patterns impact the presentation of information in disinformation and trustworthy news articles. Disinformation narratives tended to employ cognitive biases via oversimpli"cation (e.g., using absolutist claims), emotional manipulation (e.g., using highintensity emotional language), and cultural isolation (e.g., employing selective historical interpretations). This way disinformation aims to foster social polarization and discourage rational discussions as introduced in [30]. In contrast, trustworthy news narratives promoted informed decision-making (e.g., via evidence-based documentation), cross-cultural dialogue (e.g., via acknowledging shared values), and democratic discourse (e.g., by providing multi-stakeholder perspective) in following journalistic standards and o!ering nuanced, evidence-based perspectives.
These "ndings highlight the need for counter-disinformation strategies that go beyond fact-checking to address disinformation narrative structures that make it compelling to the public. For example, broader strategies could include media literacy [29, 36], strategic communication [37, 38, 39], and other approaches. Such interventions should reduce emotional manipulation, as reported in [40], also challenge simplistic dichotomies as, e.g., described in [29], and promote narratives that encourage critical thinking, as outlined in [41]. Also, this research contributes to the identi"cation and study of disinformation by providing a structural framework for analyzing narrative construction via the use of knowledge graphs. This approach demonstrated a potential to identify disinformation patterns and distinguish reliable journalism from misleading content. The graph-based methodology also allowed examining how sociocultural framing, emotional triggers, and cultural references are strategically used to in#uence public perception.
However, some limitations of this study need to be taken into consideration. The study’s focus on English content restricts its applicability to other linguistic and cultural contexts. Also, while manual inspection of the results was performed at each stage of the study, a more thorough evaluation combining automating and expert-based methods would be bene"cial. Temporal shifts in narrative patterns also require further exploration to understand the evolution of disinformation strategies. Additionally, the binary classi"cation of sources as either disinformation or trustworthy oversimpli"es the complexities of contemporary media. Therefore, automated analysis may overlook some elements, especially subtle ones such as irony, cultural nuances, or implicit biases. It is planned to address these issues in the future to develop a more comprehensive understanding of narrative dynamics in media.
This research was funded by the Research Council of Lithuania (LMTLT), grant agreement No. S-PD24-88. [5] R. Bernard, G. Bowsher, R. Sullivan, F. Gibson-Fall, Disinformation and epidemics: Anticipating the next phase of biowarfare, Health Security 19 (2020) 3 – 12. doi:10.1089/hs.2020.0038. [6] J. Jaiswal, C. LoSchiavo, D. C. Perlman, Disinformation, misinformation and inequality-driven mistrust in the time of covid-19: Lessons unlearned from aids denialism, AIDS and Behavior (2020). doi:10.1007/s10461-020-02925-y. [7] D. McRae, M. del Mar Quiroga, D. Russo-Batterham, K. Doyle, A pro-government disinformation campaign on indonesian papua, Harvard Kennedy School Misinformation Review (2022). doi:10. 37016/mr-2020-108. [8] C. Maathuis, C. D. Ridder, S. Stuurman, Analyzing the role of ukrainian and russian diaspora in disinformation campaigns, European Conference on Social Media (2023). doi:10.34190/ecsm. 10.1.1118. [9] F. Pierri, C. Piccardi, S. Ceri, A multi-layer approach to disinformation detection in us and italian news spreading on twitter, EPJ Data Science 9 (2020). doi:10.1140/epjds/ s13688-020-00253-8. [10] J. Jing, Z. Zhang, K.-K. R. Choo, K. Fan, B. Song, L. Zhang, Inference of user desires to spread disinformation based on social situation analytics and group e!ect, IEEE Transactions on Dependable and Secure Computing 20 (2023) 1833–1848. doi:10.1109/TDSC.2022.3165324. [11] X. Tian, Q. Mai, S. An, Senpr disinformation spreading model to discuss the control of the spread of disinformation in emergencies, International Journal of Modern Physics B (2023). doi:10.1142/s0217979224502837. [12] M. Ernst, Identifying textual disinformation using large language models, Proceedings of the 2024 Conference on Human Information Interaction and Retrieval (2024). doi:10.1145/3627508. 3638315. [13] M. de Kok, Y. Rebboud, P. Lisena, R. Troncy, I. Tiddi, From nodes to narratives: A knowledge graphbased storytelling approach, in: TEXT2STORY 2024, 7th International Workshop on Narrative Extraction from Texts (Text2Story), colocated with ECIR 2024, 2024. [14] A. Romanou, S. Montariol, D. Paul, L. Laugier, K. Aberer, A. Bosselut, Crab: Assessing the strength of causal relationships between real-world events, in: Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 2023, pp. 15198–15216. [15] C. Li, Improving Stance and Bias Detection in Text by Modeling Social Context, Ph.D. thesis,
Purdue University, 2021. [16] P. Yang, B. Song, Z. Zhang, Research on knowledge graph construction methods for news domain,
Academic Journal of Science and Technology 11 (2024) 58–64. [17] H. Zhou, D. Hobson, D. Ruths, A. Piper, Large scale narrative messaging around climate change: A cross-cultural comparison, in: Proceedings of the 1st Workshop on Natural Language Processing Meets Climate Change (ClimateNLP 2024), 2024, pp. 143–155. [18] Z. Han, R. Liao, B. Liu, Y. Zhang, Z. Ding, H. Köppl, H. Schütze, V. Tresp, Enhanced temporal knowledge embeddings with contextualized language representations, ArXiv abs/2203.09590 (2022). [19] A. Macaulay, W. Song, Narrative-driven #uctuations in sentiment: Evidence linking traditional and social media, SSRN Electronic Journal (2022). [20] N. Voskarides, E. Meij, S. Sauer, M. de Rijke, News article retrieval in context for event-centric narrative creation, in: Proceedings of the 2021 ACM SIGIR International Conference on Theory of Information Retrieval, 2021, pp. 103–112. [21] B. R. Andrus, Y. Nasiri, S. Cui, B. Cullen, N. Fulda, Enhanced story comprehension for large language models through dynamic document-based knowledge graphs, in: Proceedings of the AAAI Conference on Arti"cial Intelligence, volume 36, 2022, pp. 10436–10444. [22] J. A. Leite, O. Razuvayevskaya, K. Bontcheva, C. Scarton, Euvsdisinfo: A dataset for multilingual detection of pro-kremlin disinformation in news articles, in: Proceedings of the 33rd ACM International Conference on Information and Knowledge Management, 2024, pp. 5380–5384. [23] A. Sakhapara, D. Pawade, B. Dodhia, J. Jain, O. Bhosale, O. Chakrawar, Summarization of tweets related to disaster, in: Proceedings of International Conference on Recent Trends in Computing: ICRTC 2021, Springer, 2022, pp. 651–665. [24] P.-L. H. Cabot, R. Navigli, Rebel: Relation extraction by end-to-end language generation, in:
Findings of the Association for Computational Linguistics: EMNLP 2021, 2021, pp. 2370–2381. [25] C. V. S. Avila, M. A. Casanova, V. M. Vidal, A framework for question answering on knowledge graphs using large language models, in: ESWC, 2024. [26] A. Chakraborty, Multi-hop question answering over knowledge graphs using large language models, arXiv preprint arXiv:2404.19234 (2024). [27] R. Jia, B. Zhang, S. J. R. Méndez, P. G. Omran, Leveraging large language models for semantic query processing in a scholarly knowledge graph, arXiv preprint arXiv:2405.15374 (2024). [28] S. P. Borgatti, M. G. Everett, Models of core/periphery structures, Social networks 21 (2000) 375–395. [29] R. Arcos, M. Gertrudix, C. Arribas, M. Cardarilli, Responses to digital disinformation as part of hybrid threats: a systematic review on the e!ects of disinformation and the e!ectiveness of fact-checking/debunking, Open Research Europe 2 (2022) 8. [30] S. McKay, C. Tenove, Disinformation as a threat to deliberative democracy, Political Research
Quarterly 74 (2020) 703 – 717. doi:10.1177/1065912920938143. [31] W. Bennett, S. Livingston, L. Bennett, L. Carroll, The disinformation order: Disruptive communication and the decline of democratic institutions, European Journal of Communication 33 (2018) 122 – 139. doi:10.1177/0267323118760317. [32] S. Amanatullah, S. Balani, A. Fraioli, S. M. McVicker, M. Gordon, Tell Us How You Really Feel: Analyzing Pro-Kremlin Propaganda Devices & Narratives to Identify Sentiment Implications, Technical Report, The Institute for European, Russian, and Eurasian Studies, The George Washington University, 2023. [33] D. Garusi, B. M. Leonhardt, Towards a processual understanding of trust: A study of trust-building strategies of constructive journalists, Journalism (2024). doi:10.1177/14648849241287746. [34] S. A. Kamboh, M. Ittefaq, A. Rehmat, A coalition for science journalism as a multi-stakeholder community of practice: A proposal from south asia, Journalism Studies 25 (2023) 566 – 574. doi:10.1080/1461670X.2023.2293823. [35] Z. Reich, A. Barnoy, Justifying the news: The role of evidence in daily reporting, Journalism 24 (2021) 22 – 40. doi:10.1177/14648849211036835. [36] J. Abuín-Penas, F. Fernández-Medina, J.-M. Corbacho-Valencia, Media literacy against disinformation: the vision of south american factcheckers., Observatorio (OBS*) 18 (2024). [37] J. Kirchner, C. Reuter, Countering fake news: A comparison of possible solutions regarding user acceptance and e!ectiveness, Proceedings of the ACM on Human-computer Interaction 4 (2020) 1–27. [38] M. Innes, D. Dobreva, H. Innes, Disinformation and digital in#uencing after terrorism: Spoo"ng, truthing and social proo"ng, Contemporary Social Science 16 (2021) 241–255. [39] M. Kyriakidou, S. Cushion, C. Hughes, M. Morani, Questioning fact-checking in the "ght against disinformation: An audience perspective, Journalism Practice 17 (2023) 2123–2139. [40] J. Roozenbeek, E. Culloty, J. Suiter, Countering misinformation, European Psychologist (2023). [41] C.-T. Ziemer, T. Rothmund, Psychological underpinnings of misinformation countermeasures: A systematic scoping review., Journal of Media Psychology: Theories, Methods, and Applications (2024).