One of the most serious challenges facing humanity in the digital age is propaganda [1]. The main purpose of propaganda is the manipulation of objects in order to achieve certain political, social, economic or cultural goals [2,3].
Objects of propaganda are understood as individuals, groups, organizations, social strata, as well as phenomena or institutions to which propaganda efforts are directed in order to influence their consciousness, emotions, behavior and public opinion [4].
The aim of the research is to develop a method for detecting propaganda objects using deep learning neural network models, which allows detecting specific objects targeted by specific propaganda techniques. The main contributions of the paper can be summarized as follows:
• Method for detecting propaganda objects is developed, which enables the identification of the specific individuals or entities targeted by propaganda techniques within the analyzed texts.
Information Technology and Implementation (IT&I-2024), November 20-21, 2024, Kyiv, Ukraine Corresponding author. yuri.krak@gmail.com (I. Krak); zalutsk.olha@gmail.com (O. Zalutska); m.o.molchanova@gmail.com (M. Molchanova); exe.chong@gmail.com (O. Mazurets); alexander.barmak@gmail.com (O. Barmak) 000-0002-8043-0785 (I. Krak); 0000-0003-1242-3548 (O. Zalutska); 0000-0001-9810-936X (M. Molchanova); 0000-0002-8900-0650 (O. Mazurets); 0000-0003-0739-9678 (O. Barmak) • The effectiveness of developed method application has been experimentally proven, which allows, in contrast to existing analogues, to expand the list of available propaganda objects at the expense of words representations objects, in addition to searching for NER. • Visual interpretation of the identified propaganda targets was conducted, enabling clear observation of the subjects influenced within the framework of applied propaganda techniques.
The next section presents an overview of related works in the field of detecting propaganda and its objects. Section 4 offers an overview of the experiment to investigate the effectiveness of the proposed approach. Section 5 offers an overview of the obtained results with comparisons and discussions.
The emergence of diverse media channels in the information age has marked a new chapter in communication and information accessibility, simultaneously amplifying opportunities for propaganda and the manipulation of public opinion [5]. Identifying propagandistic content requires not only recognizing the content but also determining its intended targets or objectives. Identification of objects of propaganda is critically important for understanding the mechanisms of manipulation, therefore it attracts the special attention of scientists.
Existing researches highlight the problem relevance of automated detection of propaganda in text web content. A significant number of studies are currently focused on the intellectualization of propaganda detection processes to address various technological challenges in monitoring media sources [6] and distinguishing propaganda techniques from other forms of manipulative influence [7,8]. Propaganda encompasses subjects, content, forms, methods [9], and channels of information dissemination [10]. Its primary target is social groups or audiences subjected to influence.
Detecting propaganda in text using natural language processing (NLP) presents considerable challenges due to its reliance on subtle manipulation techniques and contextual dependencies. Researchers in [11] explored the effectiveness of modern large language models for identifying propaganda, conducting experiments on datasets containing news articles. Their findings indicate that modern large language models achieve performance levels comparable to the current state-ofthe-art methods [12].
Effective approaches for semantic text content analysis, such as multimodal visual-text object graph attention networks [13] and statistical test analysis [14], have been noted as valuable tools for detecting propaganda [15]. Additionally, the use of transformer-based neural networks [16,17] and complex neural architectures, including RoBERTa [18], GPT [19], and RNN [20], has emerged as a prominent direction in propaganda detection research.
Certain studies focus on specific components of propaganda, such as racial propaganda [21] or fake news [22]. The authors of [23] emphasize that most existing methods for detecting propaganda primarily analyze linguistic features of its content but often overlook the contextual information from the external media environment where propaganda originates and spreads.
Another problem with propaganda detection is the lack of reliably labeled data sources. To help the scientific community identify propaganda in text news, research [24] proposed a library of propaganda texts (ProText). Verity labels are assigned to ProText repositories after manual and automated verification using fact-checking techniques. The authors used a natural language processing approach to create a system that uses deep learning to automatically identify propaganda in the news. A fine-tuned robustly optimized BERT (RoBERTa) pre-training approach and word embedding using multi-label, multi-class text classification is proposed. Performance estimation accuracies of 90%, 75%, 68%, and 65% were achieved on ProText, PTC, TSHP-17, and Qprop, respectively. The big data method, especially with deep learning models, can help to fill the unsatisfactory big data in the new text classification strategy.
There are two main approaches to detecting propaganda: a named object detection approach and a text classification approach.
When detecting propaganda as a named entity recognition (NER) task, the problem arises that fragments of text containing propaganda are longer than NER (for example, names of persons or places) and can include tens of words. In [25], the value of the range length that affects the recognition of propaganda is investigated, showing that the difficulty of the task does increase with the increase of the range length. Various commonly used approaches to the task are systematically assessed by evaluating their ability to accurately replicate the length distribution of actual intervals. Additionally, a novel solution is introduced, incorporating an adaptive convolution layer designed to enable information exchange between distant words. This approach enhances length preservation while maintaining overall performance.
Document-level approach to propaganda detection is presented in [26], which explores the potential of using large language models (LLMs), such as those underlying ChatGPT, to identify propaganda features in news articles. The methods discussed build upon the work of Martino et al., who defined a set of 18 distinct propaganda techniques. Using these techniques, an enhanced framework is developed that integrates news articles from Russia Today (RT), a well-known statecontrolled media outlet, with the SemEval-2020 Task 11 dataset. The tip and article content are then fed to the gpt-3.5-turbo model OpenAI to determine what propaganda techniques are present and make a final decision on whether an article is propaganda or not. A subset of the results obtained is then analyzed to determine whether the LLM can be effectively used in this way. The authors report an accuracy of 25.12% using the SemEval-2022 dataset.
Thus, based on the results of the analysis of related works in the field of identifying propaganda techniques and propaganda objects, two problems were revealed: the lack of a comprehensive analysis of the relationships between propaganda techniques and objects in the texts; lack of synthesized insights regarding propaganda targets and their alternative references in textual materials. Propaganda detected by searching for named entities only has no visibility into the direction of propaganda through the use of techniques; however, the propaganda techniques detected at document level do not reflect the objects of propaganda direction [27]. At the same time, when identifying propaganda as named entity recognition (NER) task, the problem arises that propaganda objects are not presented by proper names or even synonyms, and their semantic proximity is not reflected in an obvious way. The paper proposes solution to these problems.
It is believed that the propaganda techniques used in the text can be detected by one of the existing neural network approaches [11], while the secondary result will be appropriate neural network models trained to detect individual propaganda techniques. Thus, the proposed approach for identifying propaganda objects and comparing them with the applied techniques is illustrated in Figure 1. This approach provides a comprehensive analysis of the relationships between propaganda techniques and their objects within texts, as well as generalizations for the objects of propaganda and their alternative mentions in the texts. Its distinction from existing methods lies in combining two complementary tasks: the identification of propaganda objects and correlating the identified objects with numerical values assigned to the detected techniques.
Within the framework of the proposed approach, 17 separate pre-trained neural network models of the transformer architecture will be used, which allows determining 17 main propaganda techniques, such as: «Appeal to fear-prejudice», «Causal Oversimplification», «Doubt», «Exaggeration», «Flag-Waving», «Labeling», «Loaded Language», «Minimisation», «Name Calling», «Repetition», «Appeal to Authority», «Black and White Fallacy», «Reductio ad hitlerum», «Red Herring», «Slogans», «Thought terminating Cliches», «Whataboutism» [28].
The proposed neural networks are trained on labeled data collected by the Analysis Project team (https://propaganda.qcri.org/index.html), which analyzed the texts, detecting all segments that incorporate propaganda techniques along with identifying their specific types. Specifically, they developed a dataset of news articles that were manually annotated at the segment level based on eighteen distinct propaganda techniques. Dataset includes 788 articles.
The method of detecting propaganda objects created in accordance with the proposed approach uses neural network tools, and allows detecting the belonging of objects to the used propaganda techniques, and uses the following presentation of propaganda semantic model SMP in the test text for test text T (Figure 2). Method for detecting propaganda objects as input data has test text for detect propaganda objects T, set of used techniques in the test text and set of neural network models , trained to analyze every propaganda techniques.
The first stage is named entity recognition (NER) using by «STANZA» neural network library. Since named entities can contain repetitions, all repetitions are removed at the level of lems at this stage as well. The initial data of first stage is the set of propaganda objects PO by NER without repetitions.
The second stage will be to search for words-objects that are close in meaning to each named entity. This need arises because objects of propaganda are a slightly broader concept than NER contained in the set of PO. They also include aspects of culture, groups of objects united by certain characteristics, etc. A pre-trained «FastText» model developed by Facebook AI Research will be used to find NER-like objects. «FastText» supports the «CBOW» and «Skip-gram» models, which makes it possible to effectively analyze the context of words and identify semantic relationships between them [29]. The use of «FastText» in this context is appropriate, as the model allows the detection of similar words and objects based on contextual vectors, which is useful for expanding the range of detected propaganda objects beyond named entities. The «FastText» model is retrained before being used on propaganda texts. The initial data of the stage is expansion of PO at the expense of words representations objects PO'. The minimum semantic proximity is set depending on the task, determined empirically. No threshold was applied in this study.
The next step is the step of constructing CW context windows for each propaganda object with . In the framework of the work, the context window is understood as the sentence where the specified object of propaganda is found [30]. If one context window contains several objects of propaganda, the windows are not duplicated (for one propaganda object).
According to CW', detecting level of propaganda techniques use in CW' by neural network models NM' is carried out. The assessment of the context windows' belonging to the used techniques is carried out by vectorizing the context windows CW' with corresponding vectors of traffic jams NM' and analyzing its belonging to each of techniques presented in text T.
The last stage is the building set of important relations between techniques and objects for text T. If the strength of the manifestation of the technique is below a certain threshold value, within the framework of the evaluation of the element of the set , the technique is not considered applied to the group of objects.
The initial data of the proposed approach are visual representation of semantic model of propaganda SMP for text T which include: set of propaganda objects; set of words representations objects with their semantic evaluation; set of important relations between techniques TT' and objects PO' in text T with semantic importance evaluation.
Therefore, an approach has been developed that allows detection of objects of propaganda that go beyond the detection of NER, and also allows to assess the affiliation of detected objects of propaganda to the propaganda techniques used in the text.
Accordingly, developed method for detecting propaganda objects allows to convert input data in form test text for detect propaganda objects (T), set of used techniques in the test text ( ) and set of neural network models ( ) trained to analyze every propaganda techniques into output data in form semantic model of propaganda SMP for test text T. Semantic model of propaganda SMP include set of propaganda objects PO, set of words representations objects with their semantic evaluation and set of important relations between techniques and objects in test text T with semantic importance evaluation .
To evaluate the effectiveness of the developed method for detecting propaganda objects, specialized software was created. This software facilitated the identification of propaganda objects and their comparison with the applied propaganda techniques. The obtained results in the form of visual analytics were compared with the results of the analysis of these sources by authoritative resources and experts in detecting and countering propaganda.
To conduct the experiment, software was created in the form of a web application in the Python programming language. Interface of the experimental software is shown in Figure 3. For the analysis, posts from social media were selected, processed by the «Strategic Communications Center» (https://spravdi.gov.ua/) and the «Disinformation Countering Center» (https://cpd.gov.ua/). These posts have been previously labeled and include expert conclusions, which can be compared with the original data produced by the proposed approach. Specialized software was developed for this analysis, incorporating a method for detecting propaganda objects.
The web application utilizes 17 pre-trained neural network transformation models (derived from previous research), the «Stanza» neural network library for named entity recognition (NER), the «Flask» framework, and the «FastText» pre-trained model, which has been retrained on propaganda texts. The software provides functionality for detecting the techniques used in text, identifying propaganda objects, and analyzing the association between detected objects and the applied techniques. Figure 4 shows the step-by-step illustration of proposed approach. The next stage is the construction of context windows for the found objects. Window 1: {Greece (NER): «chaos» (0.32) , «mistakes» (0.26), «collapse» (0.21)}: «If you do not support our plan, you will face an economic collapse, as happened in Greece during the debt crisis. Our opponents only promise change, but their actions will lead to complete chaos. Do you want to repeat their mistakes, risking the future of your children?» (in original Ukrainian: { (NER) }: « »). Window 2: {China (NER): «plan» (0.51), «future» (0.27)}: «China deserves better. If you do not support our plan, you will face an economic collapse, as happened in Greece during the debt crisis» (in original Ukrainian: { (NER) }: « »). The last step is to search for techniques in context windows. The assessment of the presence of the technique is carried out by evaluating the context window with a neural network trained to identify the techniques expressed in the text. For this example: {Greece: «chaos» (0.32), «mistakes» (0.26), «collapse» (0.21)} (in original Ukrainian: { (0.21)}) «Appeal to fear-prejudice» technique was rated at 0.78, « Causal Oversimplification» technique was rated at 0.65. At the same time, for {China: «plan» (0.51), «future» (0.27)} (in original Ukrainian: { (0.27)), «Appeal to fear-prejudice» was rated at 0.51, «Causal Oversimplification» was rated at 0.68.
Considering the content of the context windows, Window 1 has more pronounced manifestations of both techniques (especially «Appeal to fear-prejudice»). Window 2 has a less pronounced «Appeal to fear-prejudice» technique and a more pronounced «Causal Oversimplification» technique due to the simplified interpretation of causes and effects.
When training neural network models of the transformer architecture based on pre-trained BERT models, an accuracy of more than 80% was achieved for detecting propaganda techniques [18], the accuracy results of the BERT models are shown in Table 1.
Regarding the study of the effectiveness of the proposed method for identifying propaganda techniques and the objects they are aimed at, here the results correlate with the conclusions of researchers from the «Strategic Communications Center» (https://spravdi.gov.ua/) and «Disinformation Countering Center» (https://cpd.gov.ua/). For example, a post from the propaganda channel and the expert's conclusion are shown on Figure 5.
As can be seen from Figure 5, the post is aimed at:
• justifying the use of nuclear weapons against the United States;
• further Western intervention threatens to escalate the war;
• portraying the West as a weak opponent (discrediting).
As part of the analysis obtained by using the developed software installation, the following data were obtained:
• used propaganda techniques with manifestation powers: «Appeal to fear-prejudice» with rating of 0.598, «Loaded Language» with rating of 0.46 and «Exaggeration» with rating of 0.356; The use of the «Appeal to Fear-Prejudice» technique in the text is followed by the intention to create fear about the potential use of nuclear weapons, intimidation with massive strikes and the description of the USA as a vulnerable target creates a feeling of anxiety and uncertainty in the reader.
The use of the «Loaded Language» technique is followed by expressions such as and deliberately selected to evoke strong emotions such as anger or fear.
The «Exaggeration» technique is expressed by exaggerating the weakness of the US defense systems and the ability to deliver an irresistible blow, An example of the analysis performed by the program is shown in Figure 6. As shown in Figure 6, the analysis not only identified propaganda objects and the techniques used but also determined how the identified objects were linked to the techniques applied.
Thus, the study evaluating the effectiveness of the developed method revealed that the technique for detecting propaganda objects produces results that align perfectly with the expert findings. Additionally, by treating propaganda as a unified model and employing visual analytics for the results, it was possible to conduct a thorough analysis of the relationships between propaganda techniques and objects, as well as to generalize the objects of propaganda and their alternative references in the texts.
As part of the main goal, approach to solving the problem of identifying propaganda objects was developed, which allows to find in propaganda texts who and what propaganda techniques are aimed at. Significant problems of propaganda detection were solved, namely: absence of a thorough examination of the connections between propaganda techniques and their targets within texts; lack of synthesized insights regarding propaganda targets and their alternative references in textual materials. The effectiveness of the proposed approach has been experimentally demonstrated, offering a significant advantage over existing methods. In addition to performing named entity recognition (NER) with the «STANZA» neural network library, it expands the list of identified propaganda objects in texts by utilizing the «FastText» machine learning library. Furthermore, the approach provides an assessment that correlates the detected objects with the applied techniques. For user convenience, a visual representation of the identified propaganda objects is also provided, enabling a clear visualization of the influence targets within the framework of the used propaganda techniques.
Developed method for detecting propaganda objects use deep learning neural network models and includes stages: building set of propaganda objects by named entity recognition; text preprocessing and expansion of propaganda objects at expense of words representations objects; building context windows and they association by every words representations objects as context multiwindows; detecting level of propaganda techniques use in context multiwindows by neural network models; building set of important relations between propaganda techniques and propaganda objects for test text. Accordingly, developed method for detecting propaganda objects allows to convert input data in form test text for detect propaganda objects, set of used techniques in the test text and set of neural network models trained to analyze every propaganda techniques, into output data in form semantic model of propaganda for test text. Semantic model of propaganda include set of propaganda objects, set of words representations objects with their semantic evaluation and set of important relations between propaganda techniques and propaganda objects in test text with semantic importance evaluation.
To evaluate the effectiveness of the developed method for detecting propaganda objects, specialized software was created. This software facilitated the identification of propaganda objects and their comparison with the techniques employed in the content. The obtained results in the form of visual analytics were compared with the results of the analysis of these sources by authoritative resources and experts in detecting and countering propaganda. As a result, proposed method for detecting propaganda objects has demonstrated full alignment with the results obtained by expert evaluations. Further research is planned to be directed to the automation of the process of comparing the obtained results with the results obtained by experts. Also, further research will be aimed at developing a method for automated dynamic determination of minimum threshold values for detecting propaganda techniques occurrences, which will allow identifying the key features of combination of objects and propaganda techniques used in test text, while not overloading the model by noise in the form of insignificant semantic manifestations of sentiment.
Authors have not employed any Generative AI tools.