Propaganda is an integral component of information manipulation and includes various forms, methods and means of influencing people in order to change their psychological attitudes in the desired direction, so its timely detection is an urgent task of information technologies. Such manipulations are often used to change the psychological climate in society, mobilize support or discredit opponents [1].
14th International Scientific and Practical Conference from Programming UkrPROG'2024, May 14-15, 2024, Kyiv, Ukraine In connection with the growth of the consumption of textual Internet content, the threat of the propagandistic destructive manipulative influence of textual political media is growing. Propaganda, which is distributed on the Internet, represents a large-scale threat to the national security of the country [2], the untimely solution of which can lead to devastating consequences [3]. Therefore, the automation of the processes of detecting propaganda in textual Internet content by means of natural language processing is extremely relevant in modern conditions [4], and is capable of providing quick and timely targeted detection of hostile manipulative influence in large-scale volumes of Internet content.
Modern scientific publications highlight the relevance of the problem of automated detection of propaganda in textual Internet content. Research areas dedicated to the intellectualization of propaganda detection processes, which allows avoiding a number of technological problems in monitoring media sources [5], and the problem of separating manifestations of propaganda techniques from other manipulative influences [6,7] are especially relevant at the moment. It is noted that the elements of the propaganda model include the subject, content, forms and methods [8], as well as means or channels of information transmission [9].
The subject of propaganda is a social group that seeks to influence the audience. The content of propaganda is determined by the subject's social interests and their relation to the interests of society in general. Forms and methods of propaganda are chosen depending on the goals and the audience to be influenced. Media include print media, radio, television, etc. The object of propaganda is the audience or social groups that are the target of influence. Social interests of the subject of propaganda influence its content and choice of forms, methods and means of information transmission [10].
Detecting propaganda using NLP in text is a challenging task due to propaganda's use of subtle manipulation techniques and context dependencies. To solve this problem, the authors of [11] investigated the effectiveness of modern large language models, such as GPT-3 and GPT-4, for detecting propaganda. Experiments were performed using the SemEval-2020 task 11 dataset, which contains news articles tagged with 14 propaganda techniques. The performance of the models was determined by evaluating metrics such as F1 score, precision, and recall, comparing the results to the current state-of-the-art approach using RoBERTa. The obtained results show that GPT-4 achieves results comparable to the current state-of-the-art technology [12].
Statistical analysis of tests [14], multimodal visual-textual object graph attention network [15] are noted as sufficiently promising and effective means of semantic analysis of textual content [13] that can be used to detect propaganda. Also, at the present stage, the use of transformer-based neural network [16,17], neural network models of complex architecture, such as RoBERTa [18], GPT [19,20] and recurrent neural network [21], is a relevant direction for automated detection of propaganda. in turn focusing on the detection of our components or propaganda techniques, such as racial propaganda [22] and fake news [23].
At the same time, the authors [24] note that the existing methods of identifying propaganda are primarily focused on identifying the linguistic features of its content. However, these methods usually miss the information presented in the external news environment from which the propaganda news originated and spread. It is noted that methods for detecting propaganda in different languages may differ, depending on the type of language inflection [25]. The authors of [26] analyze how mass media influenced and reflected public opinion during the first month of the Russian invasion using articles and news channels in Telegram in Ukrainian, Russian, Romanian, French, and English. Two methods of multilingual automated identification of pro-Kremlin propaganda based on transformers (BERT) and linguistic features (SVM) were proposed and compared.
The purpose of the article is to create a method for political propaganda detection in internet content using recurrent neural network models ensemble, which will work with the Ukrainian language, as well as its approval.
As part of the research, the task was also completed: preparation of an educational Ukrainianlanguage data set; development of software that implements the created method; training of an ensemble of neural network classifiers; conducting a study of the effectiveness of the method using the developed software.
The main contribution of the article is the development of a workable method of automated detection of political propaganda in Ukrainian-language texts.
Considering the insufficient amount of Ukrainian-language data, there is a need to create an own labeled data set that will be used for training neural networks.
For training models of recurrent neural networks, a data set with the volume of more than 25,000 posts was formed, which were marked according to belonging to the categories "Propaganda" and "Non-propaganda". The lists of propaganda and verified sources were formed according to the official channels of the President and Verkhovna Rada of Ukraine, as well as according to the data of other analytical international authoritative studies and analytical summaries.
To normalize the input data, records with a length of less than 200 and more than 6300 characters were discarded. As a result of data filtering, a set consisting of 21,222 items was obtained, where 10,737 records belong to the "propaganda post" class and 10,485 records to the "nonpropaganda post" class.
To normalize the input data, entries shorter than 200 characters and longer than 6300 characters were discarded. As a result of data filtering, a set consisting of 21,222 items was obtained, where 10,737 records belong to the "propaganda post" class and 10,485 records belong to the "nonpropaganda post" class. The graph of data distribution by length in characters is shown in Figure 1. As can be seen from Figure 1, the number of records that do not contain propaganda and are in the length range of 200..800 characters makes up more than half of the set, and this may negatively affect the quality of classification in the future. At the same time, the set of propaganda texts is more evenly distributed. All multilingual fragments were automatically translated into Ukrainian.
The described data set will be used to train neural network models within the framework of the developed method of detecting political propaganda.
The scheme of the method of detecting political propaganda is shown in Figure 2. The input data of the method is an ensemble of trained models of recurrent neural networks with tokenizers, and a text post for analysis. In step 1, the ensemble of RNN models and their tokenizers are selected and loaded.
Next, step 2 pre-processes the user post for analysis, which includes converting the text to lowercase, removing stop chants and punctuation, etc. In step 3, the pre-processed text is converted into numerical sequences that will be fed to neural networks for further binary classification. Step 4 is the analysis of the post for the presence of propaganda, which includes obtaining the percentage indicators of the presence of propaganda in the post as analyzed by each RNN model.
At step 5, a conclusion is formed regarding the presence of propaganda. It is proposed to use two ensemble approaches -binary (stacking) and discrete (bagging). For the binary approach to determining the level of propaganda for ensemble neural networks, binary scores are obtained, where the score 0 is no propaganda, 1 is propaganda. In the discrete approach, the evaluation of neural networks is taken as a discrete value from 0 to 1, where 1 is the maximum manifestation of propaganda, and 0 is its absence.
In the case of stacking, a binary score is obtained, and a conclusion about the class of the post is formed according to the rules: "propaganda post", if more than 50% of the models received a binary score of 1; "post without propaganda" if more than 50% of the models received a binary score of 0; "suspicious post" if neural network models have parity voting results (about half with scores of 0 and half with scores of 1).
To determine the level of propaganda in the case of a discrete evaluation, the limits of three classes are determined by experts: the upper limit of the "post without propaganda" class and the lower limit of the "propaganda post" class.
After that, total discrete assessment of the post's belonging to specified classes is calculated (1):
where are chosen empirically, depending on the focus of the process on detecting the propaganda of the relevant species.
According to the above material, the result of the proposed method is the level and percentage estimate of the presence of propaganda as per each RNN-model of the ensemble, as well as the generalized level and percentage estimate of the presence of propaganda in the researched post.
An ensemble of two neural network models was formed to conduct an experiment on the effectiveness of the developed method of detecting propaganda in Internet content. In particular, recurrent neural networks of BiLSTM and GRU architectures were used [27]. The selection of different neural network models is due to their specific capabilities for analyzing text sequences [28].
The architectures of BiLSTM and GRU neural networks for detecting propaganda in Internet content are shown in Figure 3. BiLSTM by using the hidden state allows the analysis of text sequences in forward and reverse directions, which helps to eliminate the barriers of traditional RNNs in detecting propaganda [29]. GRU has gate mechanisms that allow for more efficient management of gradients in time, which makes it more resistant to the problem of vanishing gradients compared to classical RNNs, which are also able to effectively detect propagation events [30].
In the case of using the BiLSTM and GRU architectures to conduct the experiment, formula (1) will take the form:
where 1 k -influence coefficient of the discrete estimate obtained by the neural network BiLSTM, 2 k -influence coefficient of the discrete estimate obtained by the neural network GRU, r BiLSTM As can be seen from Table 1, GRU has higher accuracy than BiLSTM under the same parameters. Figure 4 shows the distribution of correctly classified texts by the GRU neural network (a) and the distribution of incorrectly classified texts (b).
3573 records were used as validation data, of which 1951 belonged to the "propaganda post" class, and 1622 belonged to the "post without propaganda" class. Of them, 1912 texts of the "propaganda post" class and 1565 texts of the "post without propaganda" class were correctly classified ". 57 texts of the "post without propaganda" class were falsely classified as propaganda by the neural network, and 39 texts of the "propaganda post" class were falsely classified as nonpropaganda. The overall accuracy on the validation data is 0.97. As can be seen from the numerical data, the class "post without propaganda" is classified somewhat worse than the class "propaganda post". As can be seen from Figure 4a and Figure 5a, the texts have a sufficiently high level of interclass resolution, while as can be seen in Figure 4b and Figure 5b, the incorrectly classified data are concentrated closer to the central part of the graphs, which indicates the expediency of the partitioning approach into 3 classes: "propaganda post", "post without propaganda", "suspicious post".
The software implementation of method of detecting political propaganda shows in Figure 6. To study the effectiveness of the developed method of detecting political propaganda in Internet content, which includes the ensemble use of RNN models of the BiLSTM and GRU architectures, a software implementation of the method was created using the Python language. The interface of the software component responsible for the learning module of neural network models is shown in Figure 6a. The interface of the software part responsible for the process of detecting propaganda by the developed method is shown in Figure 6b.
With the introduction of the "suspicious post" category, the percentage of errors of the first and second kind decreased. When using the binary approach, 178 samples out of 3573 turned out to be incorrectly classified. However, out of 178 samples, only 71 samples are false, the remaining 107 were classified as "suspicious post". Out of 71 false samples, only 26 texts containing signs of political propaganda were falsely assigned to the "non-propaganda post" class, and 45 texts were falsely assigned to the "propaganda post" class. As for the results of the discrete approach, 130 samples turned out to be incorrectly classified, which in general did not worsen the statistics of the GRU neural network indicators, but out of 130 samples, 37 texts containing signs of political propaganda were wrongly assigned to the "post without propaganda" class, and 52 texts were wrongly assigned to the "propaganda post".
Accuracy, Precision, Recall and F1 metrics were used to study the effectiveness of detecting political propaganda in textual Internet content using the developed method [11]. The values of the metrics for the discrete and binary variations of the method are shown in Table 2. Although the binary approach gave worse results for the Accuracy metric, it gave better results for the Precision, Recall and F1 metrics, while the discrete Accuracy approach practically did not deteriorate, but at the same time the metrics Precision, Recall and F1 are somewhat inferior to it. For the experiment, the parameters of discrete approach were as follows: k 1 =0.5, k 2 =0.5, l 2 = 0.45, l 4 = 0.55. Although binary approach gave worse results on Accuracy metric, it gave better Precision, Recall, and F1 metrics, while the discrete Accuracy approach did not deteriorate, but the Precision, Recall, and F1 metrics were somewhat inferior to it. The chart of metric values is shown in Figure 7. However, the advantages of the discrete method are its flexibility and the ability to be customized depending on the task. Experiments in this direction in the future are promising.
As for the texts that were identified as suspicious, specific signs were found in them. For example: "let the l-t community calm down. They seem to be the same people (including nationalists), which means they can be joked about like everyone else. 2) Managers are satire, therefore fire. 3) Fortunately, Ukraine is not the USA, so you can joke about real graduates of the Ternopil Medka. P.S. The best joke where the musorina stops Best, takes a bribe. And then he comes back and takes it off in the heat of the moment. P.S.S. Where is the review of the sketch with Khmelnitsky and the Moscow ambassador? There's only one American subject worth anything!" (in original Ukrainian: «хай л-т спільнота успокояться . Вони ж начеб то такі самі люди (туди ж і націоналістів ) , а значить над ними можна жартувати як і над усіма іншими ..2) Менеджери то сатира тому агонь .3) Україн на щастя не США , тому можна жартувати і над реальними випускниками тернопольської медки.П. С. Найкращий жарт де мусоріна зупиняє Беста , бере хабар . А той потім повертається і знімає його на гарячому.П. С. С. Де огляд скетчу з Хмельницьким та послом московським ? Там один фак американський чтого тільки вартий !»). The text contains a number of trigger words, such as "nationalists" (in original Ukrainian: «націоналістів»), "Moscow" (in original Ukrainian: «московським»), and the context is similar to propaganda.
There were also erroneously assigned texts in the data set. For example, the following text in the dataset was marked as a "post without propaganda", but its content: "Russia does not claim the territory of Ukraine, if Ukraine did not attack Russia, there would be no military action. Rather, Russia was attacked by NATO countries on the territory of Ukraine, because the authorities of Ukraine sold the country and betrayed their people and agreed to fight for the interests of Biden, Macron, Sunak and Scholtz until the last living Ukrainian." (in original Ukrainian: «Росія не претендує на територію України, якби Україна не напала на Росію, жодних військових дій не було б. Вірніше на Росію напали країни НАТО на території України, тому що влада України продала країну і зрадила свій народ і погодилася воювати за інтереси Байдена, Макрона, Сунака та Шольца до останнього живого українця.») is outright propaganda, and both neural network models rated it highly for political propaganda, giving scores of 0.92 (GRU) and 0.97 (BiLSTM), for a total score of 0.944.
A method for political propaganda detection in internet content using recurrent neural network models ensemble, which works with the Ukrainian language, has been proposed, and its approbation has been carried out. The method for detecting political propaganda in Internet content using ensemble of recurrent neural network models is intended to identify and analyze potentially propagandistic or manipulative content spread on the Internet. The input data of the method is an ensemble of trained models of recurrent neural networks with tokenizers and a text message for analysis. The output data are the level and percentage of propaganda presence for each neural network model of the ensemble and in general. As part of the research: preparation of an educational Ukrainian-language data set was completed; test training of an ensemble of classifiers based on BiLSTM and GRU neural network architectures was performed; software was developed that implements the created method for political propaganda detection in internet content using recurrent neural network models ensemble, and a study of its effectiveness was conducted.
The applied efficiency research of propaganda detection by an ensemble of classifiers based on the BiLSTM and GRU recurrent neural network architectures was conducted. The proposed approach is capable of detecting political propaganda by an ensemble of RNN models with Accuracy 0.97, Precision 0.973, Recall 0.981, and F1 0.976 in the bagging mode, and Accuracy 0.95, Precision 0.977, Recall 0.987, and F1 0.981 in the stacking mode . The developed method has a limitation: it works with text posts from 200 to 6300 symbols long. For shorter and longer texts, performance degradation is observed. Further research will be aimed at analyzing the dependence of the considered performance indicators of the proposed method on the features and parameters of the analyzed post, such as size, genre, and subject matter. A promising direction for further research is also an increase in the number of RNN models in the ensemble to improve performance indicators, and the specialization of models for certain types of propaganda.