This paper presents our system's participation in CheckThat! Lab Task 3, focuses on identifying persuasion techniques in Arabic text. We solely focused on Arabic, a low-resource language for this task. The task required identifying any persuasion technique applied to individual tokens within the text. Only the test set was provided for Arabic for this task, without any corresponding development or training sets. Our research aimed to investigate how a resource-rich language like English could benefit the low-resource Arabic language in the context of persuasion detection. To that end, we utilized a multilingual BERT which incorporated English and Arabic knowledge during its pre-training stage. Our system achieved first place on the Arabic leaderboard in the shared task. The result, achieved without training on Arabic data, highlights the efectiveness of multilingual BERT models. This also demonstrates the potential of using resource-rich languages like English to enhance performance in low-resource languages such as Arabic for persuasion detection tasks.
Throughout history, propaganda has played a significant role in shaping public opinion. Propaganda uses various persuasive techniques to influence the way people think and act. With the advent of the digital age, the impact of propaganda has grown even stronger. Nowadays, persuasive techniques are widely used as tools for spreading propaganda through digital platforms. The increasing use of these persuasion techniques highlights the need for advanced methods to identify and critically evaluate them. This need has become urgent as the volume of digital content continues to rise, making it easier for propaganda to spread rapidly.
This paper describes our approach to the CheckThat! task 3 [
The paper is structured as follows: Section 2 reviews previous works in this area. Section 3 outlines our proposed system in detail. Section 4 presents the results. Section 4 discusses our findings and their implications. Finally, Section 5 concludes the paper and suggests directions for future research.
Persuasion detection has traditionally focused on analyzing entire documents or paragraphs. However,
a fairly recent study introduced the task of identifying persuasion techniques at the token level [
Several recent studies have explored persuasion detection via shared tasks like SemEval and ArAIEval
[
Pre-trained multilingual models are integral to the NLP tasks for low-resource languages. BERT
[
The growing popularity of cross-lingual transfer learning ofers a promising approach to improve performance on Arabic NLP tasks. This is demonstrated by employing task-specific fine-tuning on English and French data to improve Arabic NLU performance [15]. Similarly, for abstractive summarization of Arabic text, fine-tuning multilingual models (mBERT and mBART) on Hungarian or English before fine-tuning again on Arabic data demonstrated performance gains [ 16]. These findings highlight the efectiveness of cross-lingual transfer learning in improving the performance of Arabic language processing tasks.
In this section, we describe the methodology employed for detecting persuasion techniques in Arabic articles using a multilingual BERT model fine-tuned on English data.
The data for this task was provided in the form of article files, with the corresponding labels given in a separate file. The label file contained information about the persuasion techniques used and the ofsets indicating the span of text within the articles where these techniques were applied. There are 23 labels representing diferent persuasion techniques. These techniques are identified within the text at the token level, allowing for multi-label classification where each token can be associated with one or more techniques. This detailed annotation allows the model to recognize and classify multiple techniques within a single span of text.
For preprocessing, we first split the articles into paragraphs. This was done based on empty lines, efectively treating each paragraph as a separate instance. Once the articles were divided into paragraphs, we calculated the ofsets for the persuasive spans within each paragraph. This allowed us to align the provided labels with the appropriate paragraphs.
We formulated the task as a multi-class, multi-label token classification problem. This means that each token (or word) in the input text could be classified into one or more persuasion technique categories. This approach enables the model to recognize multiple techniques that may be present in a single span of text. After predicting labels for each of the tokens, consecutive tokens with the same labels define a span. Table 1 demonstrates an example.
We employed a multilingual BERT model for this task. Multilingual BERT (mBERT ) is pre-trained on multiple languages, including Arabic and English, making it suitable for cross-lingual transfer learning. For the loss calculation, we used binary cross-entropy, which is well-suited for multi-label classification tasks.
Given the lack of Arabic training data and the zero-shot nature of the task for Arabic, we used the provided English training data to fine-tune the mBERT model. Since there was no Arabic data provided for validation, we utilized the Arabic validation dataset from the ArAIEval shared task on propaganda detection 2024.1 This validation dataset consists of 921 documents, with an average of 30.25 tokens per document, and follows the same labelling and annotation guidelines. The following hyperparameters were used during training: • Learning Rate: 5e-5 • Number of Epochs: 75 • Maximum Input Length: 256 tokens
Additionally, we utilized pos_weights to adjust the loss calculation. This helps in handling the class imbalance, ensuring that the model does not become biased towards the more frequent classes.
For evaluation, we used the modified F1-micro score, which accounts for partial matching of the spans. All the scores reported in this paper use that modified F1. Our model, fine-tuned using the English training data and validated on the Arabic dev dataset, achieved an F1-micro score of 0.0998 on the dev set. When evaluated on the test set, the model’s performance improved significantly, achieving an F1-micro score of 0.3009. The diference in performance between the dev and test sets could be attributed to the domain-specific nuances and potential distributional diferences in the test set.
Below is a detailed breakdown of the F1-micro scores per technique on the validation set, as shown in Table 2.
The results reveal a significant variation in the model’s performance across diferent persuasion techniques. Techniques such as Appeal to Time, Consequential Oversimplification, and Appeal to Values were detected more reliably, indicating that the model can efectively identify these patterns.
In contrast, techniques like Loaded Language, Straw Man, and Whataboutism showed moderate performance. Techniques like Questioning the Reputation, Repetition, False Dilemma-No Choice, and Appeal to Hypocrisy posed significant dificulties for the model. These techniques may be underrepresented in the training data, further complicating their detection.
The variation in performance can also be attributed to the nature and categorization of the techniques. Techniques that belong to the same category, such as diferent types of logical fallacies or emotional appeals, may share linguistic features that the model struggles to distinguish. For example, both Straw Man and Whataboutism involve misrepresentation or diversion tactics, which could confuse the model. On the other hand, techniques like Appeal to Values and Appeal to Popularity, which are more explicit and direct, tend to be easier for the model to identify.
It’s important to note that no Arabic data was available for training. We relied on the English training data to fine-tune the multilingual BERT model. This cross-lingual transfer learning approach introduces additional challenges due to diferences in linguistic structures and contextual usage between English and Arabic.
With the increasing sophistication of persuasion techniques, particularly in Arabic-language content, it is crucial to focus research eforts on this area. This study investigated the efectiveness of a multilingual BERT model fine-tuned on English data for the task of Arabic persuasion detection. English was selected as the training language due to its extensive resources in Natural Language Processing (NLP) tasks, including propaganda detection. Our aim was to evaluate how these abundant resources could be leveraged to benefit languages with fewer resources, such as Arabic. This work achieved first place for Arabic on the leaderboard for the test set, demonstrating the potential of cross-lingual transfer learning [17]. However, there is still room for improvement.
Future work can explore how other high-resource languages impact the performance on Arabic. There might be various strategies to enhance the model’s performance. Increasing the diversity and quantity of training data, particularly for techniques where performance was low, through data augmentation or the collection of additional labelled data, can help balance the dataset. Advanced fine-tuning techniques like focal loss can adjust the loss function to focus more on hard-to-classify examples, while dynamic sampling strategies can address class imbalance.
Additionally, incorporating more sophisticated features such as syntactic and semantic information, part-of-speech tags, or dependency parsing can provide the model with greater context and improve classification accuracy. Exploring alternative hidden layer representations within BERT may also yield better classification performance. By addressing these areas, future research can further improve the accuracy and robustness of models in detecting a wide range of persuasion techniques, ultimately enhancing their utility in real-world applications.
We acknowledge the assistance of the LT-Bridge Project (GA 952194) and DFKI for the use of their Virtual Laboratory. Also, authors have been supported financially by the EMLCT 2 programme during this entire work. [10] W. Antoun, F. Baly, H. Hajj, AraBERT: Transformer-based model for Arabic language understanding, in: H. Al-Khalifa, W. Magdy, K. Darwish, T. Elsayed, H. Mubarak (Eds.), Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Ofensive Language Detection, European Language Resource Association, Marseille, France, 2020, pp. 9–15.
URL: https://aclanthology.org/2020.osact-1.2. [11] B. Tuck, F. Qachfar, D. Boumber, R. Verma, Detectiveredasers at araieval shared task: Leveraging transformer ensembles for arabic deception detection, in: Proceedings of ArabicNLP 2023, Association for Computational Linguistics, Singapore (Hybrid), 2023, p. 494–501. URL: https://aclanthology.org/2023.arabicnlp-1.45. doi:10.18653/v1/2023.arabicnlp-1.45. [12] G. Lample, A. Conneau, Cross-lingual language model pretraining, arXiv preprint arXiv:1901.07291 (2019). [13] Y. Liu, J. Gu, N. Goyal, X. Li, S. Edunov, M. Ghazvininejad, M. Lewis, L. Zettlemoyer, Multilingual denoising pre-training for neural machine translation, Transactions of the Association for Computational Linguistics 8 (2020) 726–742. [14] M. Lewis, Y. Liu, N. Goyal, M. Ghazvininejad, A. Mohamed, O. Levy, V. Stoyanov, L. Zettlemoyer, Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension, arXiv preprint arXiv:1910.13461 (2019). [15] K. Abboud, O. Golovneva, C. DiPersio, Cross-lingual transfer for low-resource arabic language understanding, in: H. Bouamor, H. Al-Khalifa, K. Darwish, O. Rambow, F. Bougares, A. Abdelali, N. Tomeh, S. Khalifa, W. Zaghouani (Eds.), Proceedings of the Seventh Arabic Natural Language Processing Workshop (WANLP), Association for Computational Linguistics, Abu Dhabi, United Arab Emirates (Hybrid), 2022, p. 225–237. URL: https://aclanthology.org/2022.wanlp-1.21. doi:10. 18653/v1/2022.wanlp-1.21. [16] M. Kahla, Z. G. Yang, A. Novák, Cross-lingual fine-tuning for abstractive arabic text summarization, in: Proceedings of the international conference on recent advances in natural language processing (ranlp 2021), 2021, pp. 655–663. [17] A. Barrón-Cedeño, F. Alam, T. Chakraborty, T. Elsayed, P. Nakov, P. Przybyła, J. M. Struß, F. Haouari, M. Hasanain, F. Ruggeri, X. Song, R. Suwaileh, The CLEF-2024 CheckThat! Lab: Check-worthiness, subjectivity, persuasion, roles, authorities, and adversarial robustness, in: N. Goharian, N. Tonellotto, Y. He, A. Lipani, G. McDonald, C. Macdonald, I. Ounis (Eds.), Advances in Information Retrieval, Springer Nature Switzerland, Cham, 2024, pp. 449–458.