Our study presents an approach to a shared task of propaganda identification and characterization at the DIPROMATS 2024 hosted by the Iberian Languages Evaluation Forum. As the DSHacker team, we participated in the propaganda detection task, which comprised three subtasks, each with varying levels of detail in identifying propaganda types. The first subtask required binary identification of propaganda in tweets authored in either English or Spanish by diplomats and authorities from major powers. The second subtask focused on a coarsegrained classification of propaganda, while the third subtask demanded a fine-grained approach to identifying specific propaganda techniques. To tackle these challenges, we fine-tuned diferent BERT-based pre-trained models, including the XLM-RoBERTa model, and achieved remarkable success. Our system secured first place across all language categories, including monolingual and bilingual approaches, for the second and third subtask. Moreover, we attained high rankings in the binary propaganda classification. Our research also delves into the potential of detecting propaganda using Large Language Models with a few-shot prompting approach. We conducted experiments with two GPT models, including the recently released GPT-4o by OpenAI. Furthermore, we investigated the efectiveness of linguistic features and traditional machine learning models in propaganda detection. Overall, our study highlights our system's exceptional performance and provides valuable insights into the capabilities of modern language models and machine learning techniques in identifying propaganda.
In the digital age, online news often uses diferent propaganda techniques. Propaganda, as defined
by Sparkes-Vian [
Nowadays, information spreads from many online sources. Platforms like X (formerly known as Twitter) have become vital places for sharing news and opinions. However, they have also become channels for spreading propaganda, which can influence people’s thoughts and actions. As a result, it is crucial to detect propaganda, as it afects public discourse and people’s decisions.
DIPROMATS 2024 organized as a part of the Iberian Languages Evaluation Forum 2024 (IberLEF)
[
DIPROMATS 2024 introduces the shared task focused on the automatic detection and characterization
of propaganda techniques and narratives used by diplomats from major powers. In our experiments,
we decided to focus on propaganda detection tasks. This task includes three diferent subtasks listed
below [
Propaganda characterization, coarse-grained Systems must determine which of the four
categories each tweet belongs to: Not propagandistic, Appeal to commonality, Discrediting the opponent,
and Loaded language. Each tweet can be assigned to one or more categories, making it a multiclass,
multilabel classification task [
Propaganda characterization, fine-grained Systems must classify each tweet according to the
specific propaganda techniques it contains. There is one negative class and seven positive classes: Flag
Waving, Ad Populum/Ad Antiquitatem, Name Calling/Labeling, Undiplomatic Assertiveness/Whataboutism,
Appeal to Fear, Doubt, and Loaded Language [
The identification of propaganda in social media and web articles has gained significant attention in
recent years due to the increasing influence of online information on public opinion and political
discourse [
Research on propaganda detection significantly intersects with persuasion detection due to the
numerous shared characteristics and techniques [
The detection of propaganda has also been a research focus in the most recent shared task at DIPROMATS 2023 [18]. Casavantes et al. [19] utilized BERTweet [20] and RoBERTuito [21] and aimed to improve the performance of the detection of propagandistic tweets by combining the text of tweets with contextual attributes such as their geographical origin, type of message, and emotions. UniLeonUniBO Team utilized transfer learning between diferent tasks of propaganda detection [ 22]. Another two systems focused on employing data augmentation to improve performance in propaganda detection [23, 24]. Moreover, the best-performing system in binary propaganda classification in English was based on cascades of language models, adopting GPT-J as the backbone model [25].
The dataset includes tweets in both Spanish and English authored by diplomats representing China,
Russia, the United States, and the European Union. These tweets come from oficial government
accounts, embassies, ambassadors, consuls, and other diplomatic profiles. The tweets were collected
using the Twitter API for Academic Research and were posted between January 1, 2020, and March 11,
2021. The data contains features such as tweet ID, text, country, annotated labels, and a creation time
stamp. Table 1 summarizes the presence of diplomatic authorities in the dataset [
The task authors split the original data into training and test sets based on time. They chose a date
for each dataset that divides positive tweets into a 70/30 proportion. The 70% subset, consisting of the
oldest tweets, became the training set, while the 30% subset, containing the newest tweets, became the
unseen test set utilized for final systems’ scores [
Our experiments focused solely on using the tweet text and gold labels, disregarding any other columns in the dataset. In our model-building process, we included a phase for optimizing hyperparameters. To facilitate this, we divided the training data further into a new training subset and a validation subset with a ratio of 85/15. Table 2 shows the characteristics of the datasets.
Our approach relied on fine-tuning pretrained BERT-based models using the labeled dataset. Fine-tuning allows the model to learn the nuances and patterns relevant to our task while retaining the general language understanding from its initial training. We employed both monolingual and multilingual pretrained models loaded from the HuggingFace repository: 1. ENGLISH (ROB-EN) - FacebookAI/roberta-large - the language model (355M parameters) trained on English data in a self-supervised fashion [26].
We began with hyperparameter optimization for all the pretrained models we tested. This involved iftting models on the training subset with gold labels and using the remaining labeled data for validation. refers here to the number of diferent combinations of hyperparameter values. We selected the best model based on the F1 score from the validation data, and this model was used for the final submission. Monolingual models were fine-tuned exclusively on tweets in a single language, while multilingual models were fine-tuned on a combination of English and Spanish tweets. Please refer to our Appendix A, which presents optimal hyperparameters of each fine-tuned model.
In DIPROMATS 2024 Task 1, the Information Contrast Model (ICM) score determines the best propaganda
categorization model, addressing the classes’ hierarchical nature [
In subtask 1a, our best model for the English language, ROB-EN fine-tuned on English tweets, secured
obtained 4th place on the multilingual leaderboard. In subtask 1b, the XLM-BI model granted us 1st position in all language categories. We also achieved first place on all language leaderboards in subtask 1c. For English, the top results were achieved by the ROB-EN model, while for the other leaderboards, the XLM-BI model prevailed. Tables 3, 4, and 5 show our final results on all subtasks.
In this workshop, our DSHacker team won in all categories for multiclass multilabel classification. Additionally, we secured a strong position in the binary classification subtask. Our XLM-BI model consistently emerged as the top solution among final submissions. However, in certain situations, monolingual models like ROB-EN can perform better than multilingual approaches or yield comparable results. 1 4 1 1
Another technique we explored is few-shot prompting using GPT models. In few-shot prompting, the prompt includes a brief description of the task followed by a few input-output pairs demonstrating the desired behavior. This technique allows the model to infer the patterns and rules of the task from the limited examples and generate appropriate outputs for new inputs.
We applied this approach only in the subtask 1a binary classification setting. Our experiments included OpenAI’s gpt-4o (GPT-4o) and gpt-3.5-turbo-1106 (GPT-3.5) generative models. We implemented the few-shot prompting technique using the OpenAI Chat Completions API. Each prediction request sent to the GPT model consisted of a list of messages presented to the model. Each message contains the role and content attribute. There are three roles available: 1. system message helps set the behavior of the model (assistant) by providing it context and guidelines. 2. user messages can provide exemplary requests for the assistant. In our case - examplary requests for a provided text’s check-worthiness evaluation. 3. assistant messages indicate the expected output of the assistant.
Due to time constraints, we could not submit results produced by GPT models. However, we conducted post-deadline experiments and evaluated these models on the validation part of the data for binary classification from subtask 1a. In our experiments, the prompt is formatted starting with a system message that clarifies the task (See Listing 1). This is followed by alternating pairs of user and assistant messages. One pair for each few-shot example, where a user message asks whether the example’s content contains propaganda, and the corresponding assistant message provides the gold label for the example, either ’Yes’ or ’No’ (See Listing 2). The final message following the pairs is one user message with the actual text to be classified by the model (See Listing 3). For each instance to be classified, we included four examples of few-shot prompting from the training dataset, two containing propaganda. The chosen few-shot examples were consistent in a given language. The prompt templates remained consistent for both the GPT-4o and GPT-3.5 experiments.
The table shows the subtask 1a F1 scores of our models on English and Spanish validation datasets. GPT-3.5, using few-shot prompting, had moderate scores of 0.5193 for English and 0.5354 for Spanish. GPT-4o improved on these, with scores of 0.5665 for English and 0.6622 for Spanish. The multilingual model, XLM-RoBERTa-large (XLM-BI), performed better, scoring 0.7440 for English and a top score of 0.7907 for Spanish. The monolingual RoBERTa-large model for English (ROB-EN) achieved the highest score of 0.7692, while its Spanish counterpart (ROB-ES) scored 0.7684. Overall, fine-tuned BERT-based models outperformed GPT-based models, with multilingual and monolingual models showing similar performance. 7. Linguistic Features for Propaganda Detection
StyloMetrix was sucessfully utilized in persuasion detection in Polish [17]. The study of persuasion
detection significantly intersects with the study of propaganda detection due to the numerous similarities
they share [
With StyloMetrix, we can create text representations that are interpretable, normalized, and
reproducible [
In our experiments, we employ StyloMetrix vectors to predict propaganda in English tweets. The
StyloMetrix vectors for English encompass a comprehensive set of 196 metrics, categorized into several
groups: Detailed grammatical forms, General grammar forms, Detailed lexical forms, Additional lexical
items, Parts of speech, Social media, Syntactic forms, General text statistics [
1https://github.com/ZILiAT-NASK/StyloMetrix/tree/main 2https://github.com/valeriobasile/hurtlex
As the DSHacker team, we explored various techniques for propaganda detection across multiple languages and subtasks, employing both state-of-the-art pretrained BERT-based models, few-shot prompting with GPT models, and classical machine learning algorithms utilizing StyloMetrix linguistic features. Our fine-tuned BERT-based models demonstrated strong performance in the DIPROMATS 2024
for multiclass multilabel classification and in the subtask 1a binary classification of Spanish tweets. The multilingual XLM-BI model consistently delivered top results, especially in multilingual and Spanish tasks. Monolingual models like ROB-EN also showed competitive performance, particularly for English tasks, indicating that language-specific models can sometimes outperform multilingual counterparts. Few-shot prompting with GPT models yielded moderate performance on binary propaganda classification. While GPT-4o beat GPT-3.5, both were still outperformed by fine-tuned BERT-based models. Classical machine learning models like LightGBM and XGBoost, combined with well-engineered linguistic features from StyloMetrix, performed well on the binary task of propaganda detection. LightGBM, in particular, achieved a F1 score close to that of the best BERT-based model on the English validation dataset, highlighting the potential of classical models with rich feature sets on this specific task of propaganda detection. [10] P. Vijayaraghavan, S. Vosoughi, TWEETSPIN: Fine-grained propaganda detection in social media using multi-view representations, in: M. Carpuat, M.-C. de Marnefe, I. V. Meza Ruiz (Eds.), Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Association for Computational Linguistics, Seattle, United States, 2022, pp. 3433–3448. URL: https://aclanthology.org/2022.naacl-main.251. doi:10.18653/v1/2022.naacl-main.251. [11] G. Martino, A. Barrón-Cedeno, H. Wachsmuth, R. Petrov, P. Nakov, Semeval-2020 task 11: Detection of propaganda techniques in news articles, arXiv preprint arXiv:2009.02696 (2020). [12] D. Dimitrov, B. B. Ali, S. Shaar, F. Alam, F. Silvestri, H. Firooz, P. Nakov, G. D. S. Martino, Semeval-2021 task 6: Detection of persuasion techniques in texts and images, arXiv preprint arXiv:2105.09284 (2021). [13] M. Hasanain, F. Alam, H. Mubarak, S. Abdaljalil, W. Zaghouani, P. Nakov, G. Da San Martino, A. Freihat, ArAIEval shared task: Persuasion techniques and disinformation detection in Arabic text, in: H. Sawaf, S. El-Beltagy, W. Zaghouani, W. Magdy, A. Abdelali, N. Tomeh, I. Abu Farha, N. Habash, S. Khalifa, A. Keleg, H. Haddad, I. Zitouni, K. Mrini, R. Almatham (Eds.), Proceedings of ArabicNLP 2023, Association for Computational Linguistics, Singapore (Hybrid), 2023, pp. 483–493.
URL: https://aclanthology.org/2023.arabicnlp-1.44. doi:10.18653/v1/2023.arabicnlp-1.44. [14] J. Piskorski, N. Stefanovitch, G. Da San Martino, P. Nakov, Semeval-2023 task 3: Detecting the category, the framing, and the persuasion techniques in online news in a multi-lingual setup, in: Proceedings of the 17th International Workshop on Semantic Evaluation (SemEval-2023), 2023, pp. 2343–2361. [15] A. Pauli, R. Sarabia, L. Derczynski, I. Assent, Teamampa at semeval-2023 task 3: Exploring multilabel and multilingual roberta models for persuasion and framing detection, in: Proceedings of the 17th International Workshop on Semantic Evaluation (SemEval-2023), 2023, pp. 847–855. [16] T. Hromadka, T. Smolen, T. Remis, B. Pecher, I. Srba, Kinitveraai at semeval-2023 task 3: Simple yet powerful multilingual fine-tuning for persuasion techniques detection, arXiv preprint arXiv:2304.11924 (2023). [17] A. Modzelewski, W. Sosnowski, M. Wilczynska, A. Wierzbicki, Dshacker at semeval-2023 task 3: Genres and persuasion techniques detection with multilingual data augmentation through machine translation and text generation, in: Proceedings of the 17th International Workshop on Semantic Evaluation (SemEval-2023), 2023, pp. 1582–1591. [18] P. Moral, G. Marco, J. Gonzalo, J. Carrillo-de Albornoz, I. Gonzalo-Verdugo, Overview of dipromats 2023: automatic detection and characterization of propaganda techniques in messages from diplomats and authorities of world powers, Procesamiento del lenguaje natural 71 (2023) 397–407. [19] M. Casavantes, M. Montes-y Gómez, D. I. Hernández-Farías, L. C. González, A. Barrón-Cedeño, Propaltl at dipromats: Incorporating contextual features with bert’s auxiliary input for propaganda detection on tweets (2023). [20] D. Q. Nguyen, T. Vu, A. T. Nguyen, Bertweet: A pre-trained language model for english tweets, arXiv preprint arXiv:2005.10200 (2020). [21] J. M. Pérez, D. A. Furman, L. A. Alemany, F. Luque, Robertuito: a pre-trained language model for social media text in spanish, arXiv preprint arXiv:2111.09453 (2021). [22] F. Jáñez-Martino, A. Barrón-Cedeño, Unileon-unibo at iberlef 2023 task dipromats: Roberta-based models to climb up the propaganda tree in english and spanish (2023). [23] V. Ahuir, L. F. Hurtado, F. García-Granada, E. Sanchis, Elirf-vrain at dipromats 2023: Cross-lingual data augmentation for propaganda detection (2023). [24] F.-J. Rodrigo-Ginés, J. Carrillo-de Albornoz, L. Plaza, Hierarchical modeling for propaganda detection: Leveraging media bias and propaganda detection datasets (2023). [25] L. Tian, X. Zhang, M. M.-H. Kim, J. Biggs, Eficient text-based propaganda detection via language model cascades (2023). [26] Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, V. Stoyanov, Roberta: A robustly optimized BERT pretraining approach, CoRR abs/1907.11692 (2019). URL: http://arxiv.org/abs/1907.11692. arXiv:1907.11692.
This appendix includes the optimal hyperparameter values for our best models.
In this appendix, we present the prompt messages included with each text classification request. For brevity, the prompts are provided only in English.
"role": "system", "content": """You are an assistant who detects propaganda, manipulation ˓→ and persuasion techniques.
You know the definition of propaganda very well: Propaganda is the ˓→ deliberate systematic attempt to shape perceptions and manipulate ˓→ cognitions and direct behavior to achieve a response to further the ˓→ desired intent of the propagandist. """ ˓→ ˓→ ˓→ ˓→ """ ˓→ ˓→ ˓→ """ ˓→ ˓→ ˓→ """ }, { }, { "role": "assistant", "content": "Yes" }, # EXAMPLE 2 { "role": "assistant", "content": "No" }, # EXAMPLE 3 { }, { }, # EXAMPLE 4 { "role": "assistant", "content": "Yes" #Example 1
# EXAMPLE 1
Listing 1: Used initial system prompt. "role": "user", "content": f"""Answer the question whether or not the text contains ˓→ propaganda. Answer using only a single word Yes or No.
TEXT: Today, I reflect on the great honor of serving the American people this past year, and look forward to continuing to advance a diplomacy true to our core values and emboldened by U.S. leadership that may turn our greatest challenges into our greatest triumphs. Happy New Year! "role": "user", "content": f"""Answer the question whether or not the text contains ˓→ propaganda. Answer using only a single word Yes or No.
TEXT: The Islamic Republic of #Iran has fundamentally failed the Iranian people, and I am convinced that the Iranian people know that. And you’ve seen President @realDonaldTrump make very clear we will continue to support the Iranian people. "role": "user", "content": f"""Answer the question whether or not the text contains ˓→ propaganda. Answer using only a single word Yes or No.
TEXT: The Chinese government’s decision to explore its own virtual currency is already monumental, and if it ultimately moves forward it will be a global game changer. The future of global currencies may very well rest firmly in China’s hands. }, { } ˓→ ˓→ ˓→ """ "role": "user", "content": f"""Answer the question whether or not the text contains ˓→ propaganda. Answer using only a single word Yes or No.
TEXT: Over the past few years, the #US has repeatedly blocked @UN Security Council’s statements condemning attacks on other countries’ embassies. The US missile strike in Baghdad will only result in escalating tensions in the region - #Zakharova
Listing 2: Used pairs of user and assistance prompts.
Listing 3: Used final user prompt.