In the paper, a proposed approach for improving efficiency of web propaganda patterns detection by transformer neural networks is presented. Approach consists of sequential use of three developed methods: method for dataset balancing, method for fine-tuning individual binary neural network models and method for detecting web propaganda patterns. Compared to existing analogues, the use of proposed approach allowed achieving an efficiency increase of 0.1 by F1 metric when detecting propaganda patterns in web texts using transformer neural networks due to dataset balancing optimization. Analyzing the impact of parameter that determines proportion of texts without web propaganda patterns allows assessing how the models ability to distinguish propaganda patterns from neutral texts and texts with other propaganda patterns. This allows finding the optimal ratio of dataset classes to increase the overall effectiveness for detecting web propaganda patterns. Conducted research has established that the highest results were achieved when forming the training dataset with a percentage of texts without patterns of 30% using the RoBERTa neural network, and was achieved 0.725 by F1 metric. Proposed approach ensures the determination of the optimal ratio between text sets with propaganda patterns and neutral text set, which improving the generalization ability of models and reduce their bias.
In the modern information environment, propaganda content plays a significant role in shaping
public opinion, political views, and social behavior [
The research is closely related to the UN Sustainable Development Goals, as it contributes to the
formation of quality education (SDG No. 4) through the development of media literacy and critical
thinking [
The aim of paper is to improve the efficiency of detecting propaganda patterns in web texts using transformative neural networks by optimizing the dataset balancing. Research is aimed at reducing the impact of class imbalance, increasing the accuracy of classification and improving the generalization ability of model.
The main paper contribution is created methodology that includes method for fine-tuning individual binary neural network models to detect propaganda patterns, method for balancing the dataset, and method for detecting web propaganda patterns. The paper also provides an analysis of the impact of the balance of the training sample on the effectiveness of models for detecting manipulative patterns in social media. An experimental study of the performance of the BERT and RoBERTa transformative language models depending on the distribution of training examples between classes was conducted. The results obtained contribute to a deeper understanding of the role of the training sample in improving algorithms for detecting manipulative texts and can be used to increase the reliability of automated systems for analyzing the information space.
The issue of automated detection of web propaganda patterns in social media has widely attracted the attention of researchers.
The research [
In [
The authors of [
In [
The study [
In [
The authors of [
The application of machine learning models to identify manipulation patterns in text content is
considered in the study [
The study [
Paper [21] proposes an ensemble model for identifying manipulation patterns in texts obtained from memes. The authors consider the use of modern pre-trained language models, as well as optimization methods, in particular data augmentation and combining multiple models. The model evaluation was carried out on the SemEval-2021 Task 6 dataset, and the results showed that proposed approach allows achieving an F1-micro measure of 60.4% on the test set.
Authors of [22] used a two-stage process to determine the optimal threshold for classifying manipulation patterns to assess the effectiveness of the model. First, experiments were conducted with macrothresholds in the range from 0.1 to 0.9, the threshold with the highest F1 score was selected, after which microthresholds were added for further optimization. The XLM-RoBERTa models were trained using the Adam optimizer, and early termination was used to prevent overtraining. The Accuracy, Precision, Recall, and F1-measure metrics were used to assess performance at each stage.
From above reviews of scientific publications, it is clear that the issue of balancing datasets in existing methodologies was considered only from the perspective of creating synthetic samples, and the issue of the influence of the number of texts without manifestations of propaganda patterns was not considered at all. Therefore, our study is relevant and aims to eliminate this drawback by analyzing the influence of the number of texts without propaganda patterns on the effectiveness of transformer models.
The paper aims to determine the optimal ratio between texts with propaganda patterns and neutral texts, which will improve the generalizability of the models and reduce their bias.
To solve the problem of detecting web propaganda patterns, it is first necessary to fine-tune the neural networks to detect each of the web propaganda patterns. Accordingly, this can be formalized as the problem of training a set of individual binary neural network models NN, where each model nni corresponds to a certain propaganda pattern pi from the set of propaganda patterns P:
P={ p1 , p2 , … , pk }, (1) where pi – i-th propaganda pattern, k – number of unique propaganda patterns, i=1..k. Within the scope of the study, k=10, and the set P acquires the following elements: p1=”Loaded Language”; p2=”Glittering Generalities”; p3 =”Euphoria”; p4=”Appeal to Fear”; p5=”FUD”; p6=”Bandwagon”; p7=”Thought-Terminating Cliche”; p8=”Whataboutism”; p9=”Cherry Picking”; p10=”Straw Man”.
This set of propaganda patterns is linked to the existing data source presented within the framework of UNLP 2025 [23], dedicated to the competition for detecting manipulative propaganda patterns in the Ukrainian-language media space [24].
Accordingly, {NN} will take the form:
NN ={nn1 , nn2 , … , nnk }, (2) where nni – i-th neural network for i-th propaganda pattern.
Approach for detection of web propaganda patterns by transformer neural networks consists of sequential use of three developed methods: method for dataset balancing, method for fine-tuning individual binary neural network models and method for detecting web propaganda patterns (Figure 1).
Proposed approach ensures the determination of the optimal ratio between text sets with propaganda patterns and neutral text set, which improving the generalization ability of models and reduce their bias. This improves the efficiency of detecting propaganda patterns in web texts using transformer neural networks through optimizing the dataset balancing.
Method for dataset balancing is designed to transform the general set of data in the input dataset into 2 datasets (training dataset and validation dataset), which will allow to increase the accuracy of detecting propaganda patterns in web texts. Scheme of training dataset prepare is shown in Figure 2.
Percent of texts without manipulation patterns m – the studied parameter for analyzing the influence of the balance of the training sample on the effectiveness of models for detecting manipulative propaganda patterns in social media. This parameter has an impact on the formation of the training dataset.
In addition to the training dataset, a validation dataset is constructed, which consists equally of all types of web propaganda patterns and texts without propaganda. This allows determining whether the model does not confuse patterns with each other and whether it is able to detect them independently of each other, which is critically important for the multi-label classification problem. Accordingly, the result of the method of dataset balancing will be 2 datasets: training dataset and validation dataset. Schematically, their composition is shown in Figure 3.
Document
count
It is worth noting that the base dataset is annotated at the fragment level, and the training dataset and validation dataset contain not the full text, but fragments (sentences that are marked as propaganda patterns).
The dataset contains annotated data at the fragment level that determine the presence of manipulative influence patterns from the set P. A typical text of the dataset from the category "propaganda patterns" can have either one or several labels. A typical text of the dataset from the category "without propaganda patterns" does not contain any web propaganda patterns from the set P. According to the marked data, the number of documents corresponding to the patterns p1 – p10 has the distribution shown in Table 1.
This approach to dataset generation allows us to assess the impact of sample balancing on the quality of propaganda pattern detection, as well as to avoid the dominance of the most common classes in the training set [25]. Using separate binary models for each pattern allows us to model them independently, which is important in problems with class intersection, when one text may contain several types of manipulation. This allows us to investigate how each pattern is separated within the data corpus and how it is affected by the imbalance of the training sample.
As can be seen from Table 1, the data have an uneven distribution, so using a single multi-class neural network model will not allow to obtain high results. A multi-class model tends to dominate widely represented classes, which leads to a decrease in accuracy for poorly represented classes. As a result, the model may simply ignore small categories, which will lead to a significant imbalance in predictions. In addition, multi-class classification assumes that the text belongs to only one class [26], which contradicts the nature of the task, where 1 text can have several labels corresponding to certain web propaganda patterns. Accordingly, using separate binary models for each pi pattern allows to train each model separately without the influence of the imbalance of other classes to take into account texts with several patterns, since each model from NN set works independently and does not limit the choice to only one class.
To investigate the impact of the balance of the training sample on the detection of web propaganda patterns using a set of individual binary neural network models NN, it is necessary to first present a method for obtaining a typical individual binary neural network model nni for detecting propaganda pattern pi, the scheme of which is shown in Figure 4.
The input data of the method are prepared datasets for training and validation and pre-trained model nn. On Step 1, Fine-Tuning of typical nni on training DataSet, formed by method of datasets balancing, is performed. Fine-Tuning within the framework of the study will be carried out for individual binary neural network model BERT [27] and RoBERTa [28] with «HuggingFace» library [29].
Accordingly, on Step 2, evaluation of individual binary neural network model nni is performed, for evaluations both training dataset and validation dataset, which were formed by method of datasets balancing, will be used. Evaluation of models will be carried out by metrics Accuracy, Precision, Recall and F1. On Step 3, save of validated nni is performed. Accordingly, output data is fine-tuned model nni with metrics.
As pre-trained model nn, the use of BERT-like architectures is proposed, since these models can be applied to the analysis of Ukrainian texts even in the absence of large volumes of marked-up data [30, 31]. This feature is associated with pre-training on large text corpora, which allows these models to form universal language representations that can be refined on specific datasets to detect propaganda patterns. Fine-tuning allows you to adapt the model to the specifics of manipulative discourse, in particular in the Ukrainian language environment, which contains both unique stylistic and syntactic features.
After forming datasets and training a set of individual binary neural network models NN, detection of web propaganda patterns occurs. Scheme of method of web propaganda patterns detection by transformer neural networks is shown in Figure 5.
Input data of the method detection of web propaganda patterns by transformer neural networks are fine-tuned models NN, web content for analysis and threshold t.
On Step 1, preprocessing of web content for analysis occurs, which includes of splitting into sentences, after which tokenization is performed [32, 33]. The result of web content splitting for analysis will be the representation (3):
S={s1 , s2 , … , sn }, (3) where sj – j-th sentence in web content for analysis, n – count of sentence.
Step 2 performs web content labeling by each of nni . Each sentence sj is evaluated separately by each of nni, and if the output value of the neural network model nni for sentence j exceeds the given threshold t –propaganda pattern pi is considered to be manifested in sentence j. Accordingly, each sentence will be given a subset PPj of the elements of the set P:
S score=
PP j⊆ P , PP j={ pi|scorei , j>t }, (4) where scorei,j – the output value nni of the model for j-th sentence in {S}.
At Step 3, the formation of output view takes place, which is performed according to rules: if there are already manifestations of other propaganda patterns for sentence sj, then such propaganda patterns are considered manifested in the text, however, the maximum value max_scorej will be displayed with highlighting: max¿= max scorei , j , (5)
pi⊆ PPj if there are multiple sentences with the pi propaganda pattern, the overall score of the manifestation in web content for analysis is calculated as the arithmetic mean:
1 (6) ¿ SSi∨¿ ∑ scorei , j , SSi={s j∨ pi∈ PP j }¿
sj⊆ SSi where SSi, – a set of sentences in which is found.
Output data of the proposed method are probabilities of each of propaganda patterns in web content and highlighted sentences in which identified patterns are most evident [34].
The proposed in sections 3.1 – 3.2 methods are investigated experimentally in section 4.
In accordance with purpose of research, problem of improving efficiency via dataset balancing arises, which can be mathematically represented as a problem of maximizing the F1 metric: m¿=arg max f ( m ) , (7)
m where f(m) – the value of the F1 metric of the nni model obtained after fine-tuning on the dataset with the selected percentage value m.
The solution of the optimization problem will be carried out experimentally, changing the % of non-propaganda texts in the Training DataSet from 10% to 70% in steps of 20%.
For the experimental part, specialized software was created, consisting of 2 modules: a training module (without a graphical user interface) and a neural network validation module (the application is shown in Figure 6). The Python language, PyTorch libraries [35], transformers [36], datasets [37] were used to develop the training module. The PySide6 libraries [38], transformers, PyTorch were used to develop the validation module.
Propaganda patterns m % p1 p2 p3 p4 p5 p6 p7 p8 p9 p10
Target
Accordingly, the created test software obtained the results shown in Section 5.
After filling the Training DataSet using the method described in section 3.1, data sets were obtained, the quantitative distributions of which are given in Table 2.
The Precision (P), Recall (R), F1 metrics for fine-tuned individual binary neural network models at different percentage values of the parameter m on the test sample (20% of the Training DataSet, which did not participate in training) are given in Table 3. The Precision (P), Recall (R), F1 metrics for fine-tuned individual binary neural network models at different percentage values of the parameter m on the training sample (80% of the Training DataSet, which participated in training) are given in Table 4.
Target B E R T
Propagand a patterns p1 p2 p3 p4 p5 p6 p7 p8 p9 p10 p1 p2 p3 p4 p5 p6 p7 p8 p9 p10
The Precision (P), Recall (R), and F1 metrics for fine-tuned individual binary neural network models at different percentage values of parameter m on validation dataset are given in Table 5.
Comparisons by the Accuracy metric (average value) for fine-tuned individual binary neural network models of the BERT and RoBERTa architectures at different percentage values of the parameter m on the Validation DataSet are shown in Figure 7.
Comparisons by F1 metric (average value) for fine-tuned individual binary neural network models of the BERT and RoBERTa architectures at different percentage values of the parameter m on the Validation DataSet are shown in Figure 8.
It is also worth providing a table comparing the obtained results with the data of existing studies (Table 6).
Therefore, the problem of improving efficiency via dataset balancing, given in the form (7), has a solution m* = 30. An analysis of the obtained results is given in Section 6.
In the presented results of testing models (Table 3) for detecting manipulative propaganda patterns on the test sample (20% of the training sample) with different percentages of texts without manipulations m (10%, 30%, 50%, 70%), one can observe a clear trend towards improving the performance of models with an increase in the value of the parameter m, i.e. with an increase in the percentage of texts without propaganda patterns in the training sample. For fine-tuned models based on BERT, it is seen that Precision, Recall and F1-measure for each category of propaganda patterns gradually increase from m=10% to m=70%. For example, for the “Loaded Language” category, the F1-measure increases from 0.498 at m = 10% to about 0.798 at m = 70%, which indicates a significant improvement in the model’s ability to distinguish target and non-target examples with an increase in the proportion of text samples without manipulations.
Comparing the performance of models for different propaganda patterns shows that some categories, such as “Glittering Generalities” and “Bandwagon”, “Cherry Picking”, have consistently high F1-measures as m increases, indicating that the characteristic features of these patterns are easier to separate with balanced training. In contrast, other categories, such as “Straw Man” and “Thought-Terminating Cliche”, show relatively lower performance, which may be due to greater variability or subtlety of the linguistic features characterizing these patterns.
Similar analysis for RoBERTa-based models shows similar trends, with the overall performance being slightly higher compared to BERT models. This is explained by the more robust pre-training and optimized architecture of RoBERTa, which allows the model to generalize information better. The improvement in the evaluation indicators with an increase in the proportion of unmanipulated text samples highlights the importance of balancing the dataset to overcome the problem of class imbalance, which, in turn, contributes to more reliable and stable detection of propaganda patterns by individual binary neural networks. For the BERT neural network, on average, for the F1 metric, the delta between m = 10% and m = 30% is +0.048, between m = 30% and m = 50%, the delta is 0.063, and between m = 50% and m = 70%, the delta is 0.091. At the same time, for the RoBERTa neural network, delta of +0.0632 is observed between m = 10% and m = 30%, a delta of 0.056 is observed between m = 30% and m = 50%, and delta of 0.082 is observed between m = 50% and m = 70%.
The analysis of the data from Table 4 indicates the ability of neural network models to remember, and here, naturally, as in Table 3, there is a tendency for metrics to increase with increasing parameter m. For the BERT neural network, on average, for the F1 metric, there is a delta between m = 10% and m = 30% of +0.049, between m = 30% and m = 50%, there is delta of 0.02, and between m = 50% and m = 70%, there is delta of 0.031. At the same time, for the RoBERTa neural network, a delta of +0.028 is observed between m = 10% and m = 30%, delta of 0.022 is observed between m = 30% and m = 50%, and delta of 0.024 is observed between m = 50% and m = 70%. Accordingly, RoBERTa demonstrates a gradual increase in metrics, which indicates stable generalization due to the optimized architecture. BERT demonstrates somewhat jumpy increases, which may be due to the lower flexibility of its architecture in adapting to changes in the proportion of text samples without propaganda patterns.
For the RoBERTa neural network, when detecting manipulation patterns “Glittering Generalities”, “Appeal to Fear”, “FUD”, “Bandwagon”, “Whataboutism”, an F1 value of more than 0.95 is observed. For the BERT neural network, a value above 0.95 is observed only for “FUD”. In general, the use of different values of the parameter m affects the ability of neural networks to remember the features of the training set. However, the metrics calculated on the training data allow us to assess how well the model remembered this data, but do not give a complete picture of its ability to generalize new information.
The most relevant estimates of the experiment are given in Table 5, since here the model was validated on data that did not participate in training, and which contain equally represented propaganda patterns and texts without such patterns.
According to Table 5 and Figures 7 and 8, at the parameter m=30% the metrics demonstrate the highest result, where the average value of the Accuracy metric is 0.733 for the RoBERTa neural network, and 0.704 for the BERT architecture. The F1 metric for RoBERTa is 0.725, and for the BERT architecture – 0.693. This suggests that the initial addition of data allows to increase the metrics, but then the effect is smoothed out or even worsened due to overloading with less useful information, such as texts without propaganda patterns. Accordingly, while neural networks show a tendency to better distinguish propaganda patterns at higher values of m during training, testing on a balanced validation set refutes the hypothesis that the higher the resolution of the training data, the better the generalization ability of the neural network model. It is possible that as the proportion of m increases, the models are overtrained due to the lack of unique values inherent in each of the web propaganda patterns. The conclusion that the m*=30 found is also confirmed by the minimum mean deviation between the test data for m=30 (Table 3 and Table 5) for both the BERT architecture neural network (0.05) and RoBERTa (0.06).
The comparison with analogues is carried out in Table 6, and for the purity of the comparison of
the developed approach and existing analogues, the F1 value was taken specifically on the validation
data. Accordingly, the highest F1 indicator for the RoBERTa architecture at m=30% is 0.725, which is
0.1 higher than the analogue described in [
However, the proposed approach has limitations. In this study, an approach at the sentence level was used. This may have an impact on the quality of detecting propaganda patterns, which may work at the level of paragraphs or even entire texts, rather than individual sentences. Also, a single sentence may be neutral in itself, but in the context of propaganda text its meaning changes. These issues will be addressed in further research. There are also limitations at the level of the data source. The manual labeling used in the dataset may contain subjective judgments, which affects the training of the model.
In the paper, a proposed approach for improving efficiency of web propaganda patterns detection by transformer neural networks is presented. Approach consists of sequential use of three developed methods: method for dataset balancing, method for fine-tuning individual binary neural network models and method for detecting web propaganda patterns. Compared to existing analogues, the use of proposed approach allowed achieving an efficiency increase of 0.1 by F 1 metric when detecting propaganda patterns in web texts using transformer neural networks due to dataset balancing optimization.
In addition to the training dataset, consisting of texts with target propaganda pattern in the target category, as well as texts without any propaganda patterns and texts with other propaganda patterns, without target, a validation dataset was built, which consists equally of all types of web propaganda patterns and texts without propaganda. This allows us to determine whether the model does not confuse patterns with each other and is able to detect them independently of each other, which is critically important for the patterns detection task.
An analysis of the impact of the balance of the training sample on the effectiveness of propaganda pattern detection models in social media was performed, which showed that of the considered options for forming the training dataset with different percentages of texts without manipulations (10%, 30%, 50% and 70%), the highest results were achieved at 30% using the RoBERTa neural network, and are 0.725 according to the F1 metric. The results obtained contribute to a deeper understanding of the role of training sample balancing in improving propaganda pattern detection algorithms and can be used to increase the reliability of automated information space analysis systems.
Building a validation dataset that contains an equal number of texts with all types of propaganda patterns, as well as neutral texts, provides a fair assessment of the performance of the models. This prevents bias towards the most represented classes and allows for more accurate performance metrics for each individual pattern. In addition, this approach allows for the identification of potential relationships between different types of manipulation, since texts can contain multiple patterns at the same time.
Analyzing the impact of parameter that determines proportion of texts without web propaganda patterns allows assessing how the models ability to distinguish propaganda patterns from neutral texts and texts with other propaganda patterns. This allows finding the optimal ratio of dataset classes to increase the overall effectiveness for detecting web propaganda patterns.
The proposed approach has the limitation of analyzing at the sentence level, which may not take into account the broader context of propaganda patterns at the paragraph or whole text level. In addition, the use of manual data labeling may contain subjective judgments, which affects the training of the model.
This study was conducted using dataset made available by UNLP 2025 Shared Task initiative (GitHub repository) [24]. Authors are grateful to organizers and contributors for compiling and sharing this resource, which supports ongoing research in propaganda detection techniques.
During the preparation of this work, the authors used Grammarly in order to: Grammar and spelling check. After using this tool, the authors reviewed and edited the content as needed and take full responsibility for the publication’s content.