In this paper, we describe our participation in the Automatic Detection and Characterization of Propaganda Techniques and Narratives from Diplomats of Major Powers shared task (DIPROMATS). For this edition, we experimented with data augmentation, leveraging both English and Spanish training sets in a crosslingual setting. As in the previous edition, the use of contextual features of the posts was also considered to improve their interpretation and subsequent classification. Our results show a slight increase in classification performance by incorporating more training instances. In particular, our strategy of crosslingual data augmentation obtained competitive scores in the binary propaganda identification task : the eighth position for English out of 26 runs, and the eighth position for Spanish out of 30 runs.
Propaganda can be defined as “an evolving set of techniques and mechanisms which facilitate
the propagation of ideas and actions” [
Our strategy this year is focused on binary propaganda identification , which involves deciding
whether a tweet contains propagandist content. In addition to resuming aspects of our
participation in the 2023 edition (i.e., exploiting contextual information [
The rest of this paper is structured as follows. Section 2 describes the proposed approach. Section 3 covers our experimental settings. Section 4 discusses the obtained results. Finally, Section 5 draws conclusions about our participation in the shared task.
When designing our propaganda detection models, it was noted that a potential factor hindering
the classifiers’ performance could be data scarcity. In order to investigate this, we conducted an
initial experiment using the data from DIPROMATS 2024 to assess how the classification
performance was afected when the proportions of training data were altered. For this experiment,
7,146 tweets were considered for the English training set and 5,202 tweets for Spanish. These
volumes represented training with "100%" data. Then, using stratified random sampling, smaller
volumes of data were created: 25%, 50% and 75%. With the four aforementioned data volumes,
BERT-based classifiers [
Our approach to Subtask 1a consists of three main modules: data preparation, features computation, filtering process, and BERT-based classification.
Data preparation. We extracted the country of each tweet (which corresponds to that of
the user who posted it) as well as the kind of post (how it was disseminated). The text of each
tweet, from now on referred to as , was passed through a tokenization procedure handled
by diferent pre-processing functions depending on their corresponding language 1. We also
included an additional feature regarding emotional information, in particular, we considered the
categorical model of emotions [
Used Features. After the pre-processing step, we conducted our experiments using the following features: • Text of the tweet (): raw contents of the tweet. • Country: The source country of the person who posted the tweet. • Type: The way how the tweet was disseminated: tweet, retweet, reply, or quote. • Emotion: Emotional category assigned according to the corresponding pre-trained language model, as described above.
Filtering process. The methodology employed for augmenting data to the English train set
is as follows. Initially, a series of five BERTweet models are trained using all available tweets
in English. Subsequently, these models are used to generate predictions over the train set in
Spanish (previously translated to English). If a minimum of three out of the five predictions
align with the true label of the tweet (originally in Spanish), it is selected for inclusion in the
1We used BERTweet’s tokenizer module [
BERT-based classifiers. Our approach relies on Bidirectional Encoder Representations from
Transformers (BERT) models, which allowed us to create pre-trained language representations
combining left and right contexts (thus generating a deep bidirectional Transformer) [
3.1. Data The datasets are a collection of tweets published by authorities from China, Russia, the European Union, and the United States between January 1st, 2020, and March 11th, 2021 [20]. Table 1 shows the distributions of the train and test sets for both English and Spanish (refer to [21] for further information). The last section of the table reports the emotion distribution, inferred as described in Section 2.1.
We used a BERTweet based classifier for English and a RoBERTuito for Spanish. These models were implemented using Python 3.7 [22], and the HuggingFace library [23]. The hyperparameters used for both classifiers were a batch size of 32, learning rate of 2e-5, 3 epochs, max sequence length of 250, and Adam optimizer.
As mentioned before, in our approach, we took advantage of both translated propaganda instances and the auxiliary input of the transformer models. For our participation in the shared task, we submitted the following five diferent configurations: • Run 1 - Vanilla. No data augmentation, no contextual attributes. • Run 2 - With contextual attributes. Country + type + emotion. • Run 3 - Data augmentation. Adding all translated propaganda instances from the other language.
During the development phase, our approach involved splitting the training sets (for both English and Spanish) into two fixed partitions: 85% allocated to a training partition (which we will subsequently refer to as "custom train set"), and the remaining 15% designated for testing (hereafter referred to as "custom test set"). Table 2 shows the results for Task 1, which correspond to the average of 5 executions of BERTweet/RoBERTuito along with their respective standard deviations.
The boxplots in Figure 3 show the distribution of the runs, in terms of F1-score over the propaganda class, submitted by all the participant teams at the shared task. Our cross-lingual augmented runs are positioned either slightly above or close to the upper quartile, while our vanilla runs are between the median and upper quartile.
Attempting to explain the rationale behind the modest performance diferences observed in our models when using or not data augmentation, we conduct supplementary analyses. The ifrst one is based on the speculation that perhaps both English and Spanish train sets shared Twitter accounts, and subsequently some tweets used to augment the training sets are likely to both come from these common accounts and contain similar propaganda. However, our examination refuted this first hypothesis at least for the data augmentation in Spanish, showing that there is no strong overlap of accounts in both languages. In summary we observed the following: • It turned out that the number of unique Twitter accounts in the English Train set was 491, while the number in the Spanish Train set was 128. • The total number of accounts in common (in other words, the intersection of these sets) is 39. • Of the 498 Spanish tweets that were added to the English Train set, the number of tweets belonging to the common accounts was 284 (more than half). • Of the 1030 English tweets that were added to the Train set in Spanish, the number of tweets that belong to the common accounts was 82.
Additionally, we calculated Kullback–Leibler divergences [25, 26] between term frequencies of original and augmented sets for each language. For English, the divergence between the custom train set and the augmented train set was 0.0599, while for Spanish the divergence was 0.1636. The divergence in Spanish is greater than the divergence in English, possibly because twice as much data was augmented in Spanish as in English; however both values are rather small. Both results, which are very close to 0, indicate that the vocabulary frequency distributions between the sets had very few diferences and, therefore, that the collections in both languages have similar tweets, possibly corresponding to common propaganda campaigns.
This paper describes our participation in the 2024 DIPROMATS shared task. The proposed approach showed good performance in both English (8ℎ position) and Spanish (8ℎ position). In particular, we focused on cross-lingual data augmentation, while also resuming part of our previous participation by combining text messages with contextual information. Through our participation, it has been confirmed that improvements in the classification outcomes can be achieved for this subtask, by incorporating additional training resources, particularly sourced from the same domain (diplomatic entities) and time period. As future work, we would like to explore the idea of continuing to incorporate diferent data collections and determine if other types of propaganda are "compatible" with those in the DIPROMATS corpus. language model for social media text in Spanish, in: Proceedings of the Language Resources and Evaluation Conference, European Language Resources Association, Marseille, France, 2022, pp. 7235–7243. URL: https://aclanthology.org/2022.lrec-1.785. [18] J. A. Fuentes-Carbajal, M. Montes-y Gómez, L. Villaseñor-Pineda, Does This Tweet Report an Adverse Drug Reaction? An Enhanced BERT-Based Method to Identify Drugs Side Efects in Twitter, in: Pattern Recognition: 14th Mexican Conference, MCPR 2022, Ciudad Juárez, Mexico, June 22–25, 2022, Proceedings, Springer, 2022, pp. 235–244. [19] F. Sánchez-Vega, A. P. López-Monroy, BERT’s Auxiliary Sentence focused on Word’s
Information for Ofensiveness Detection., 2021. [20] DIPROMATS 2024, Task 1 Data page, https://sites.google.com/view/dipromats2024/task-1/ data, 2024. [Online; accessed 06-June-2023]. [21] P. Moral, J. Fraile, G. Marco, A. Peñas, J. Gonzalo, Overview of DIPROMATS 2024: Detection, Characterization and Tracking of Propaganda in Messages from Diplomats and Authorities of World Powers, Procesamiento del Lenguaje Natural 73 (2024). [22] G. Van Rossum, F. L. Drake, Python 3 Reference Manual, CreateSpace, Scotts Valley, CA, 2009. [23] T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz, J. Davison, S. Shleifer, P. von Platen, C. Ma, Y. Jernite, J. Plu, C. Xu, T. Le Scao, S. Gugger, M. Drame, Q. Lhoest, A. Rush, Transformers: State-of-the-Art Natural Language Processing, in: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, Association for Computational Linguistics, Online, 2020, pp. 38–45. URL: https://aclanthology.org/2020.emnlp-demos.6. doi:10.18653/v1/2020.emnlp-demos.6. [24] E. Amigo, A. Delgado, Evaluating Extreme Hierarchical Multi-label Classification, in: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Association for Computational Linguistics, Dublin, Ireland, 2022, pp. 5809–5819. URL: https://aclanthology.org/2022.acl-long.399. doi:10.18653/v1/2022. acl-long.399. [25] S. Kullback, R. A. Leibler, On Information and Suficiency, The Annals of Mathematical Statistics 22 (1951) 79 – 86. URL: https://doi.org/10.1214/aoms/1177729694. doi:10.1214/ aoms/1177729694. [26] C. E. Shannon, A mathematical theory of communication, The Bell System Technical Journal 27 (1948) 379–423. doi:10.1002/j.1538-7305.1948.tb01338.x.