Introduction

Throughout history, propaganda has played a significant role in shaping public opinion. Propaganda uses various persuasive techniques to influence the way people think and act. With the advent of the digital age, the impact of propaganda has grown even stronger. Nowadays, persuasive techniques are widely used as tools for spreading propaganda through digital platforms. The increasing use of these persuasion techniques highlights the need for advanced methods to identify and critically evaluate them. This need has become urgent as the volume of digital content continues to rise, making it easier for propaganda to spread rapidly.

This paper describes our approach to the CheckThat! task 3 [1,2], focuses on the identification of persuasion techniques within the textual spans of Arabic articles. The goal of this task is to detect various techniques used to persuade readers within Arabic texts. However, a significant challenge we faced was the lack of training data for Arabic. While the task provided training data for several languages, including English, French, Italian, German, Russian, and Polish, there was no training set available for Arabic. This lack of training data for Arabic made it difficult to develop a model specifically trained on Arabic texts. To overcome this challenge, we used the training data from the English set to fine-tune a multilingual BERT model [3] and then evaluated it on the Arabic test set. Thus, our study investigates the effectiveness of using a high-resource language, such as English, to enhance the performance of a model for a low-resource language like Arabic. In the context of the persuasion technique identification task, we aimed to demonstrate that a model trained on English data could still perform effectively when applied to Arabic texts. This approach is based on the idea of cross-lingual transfer learning, where knowledge gained from one language can be transferred to another language.

The paper is structured as follows: Section 2 reviews previous works in this area. Section 3 outlines our proposed system in detail. Section 4 presents the results. Section 4 discusses our findings and their implications. Finally, Section 5 concludes the paper and suggests directions for future research.

Related Work

Persuasion detection has traditionally focused on analyzing entire documents or paragraphs. However, a fairly recent study introduced the task of identifying persuasion techniques at the token level [4]. Their work is significant because it provides one of the earlier datasets annotated with propaganda techniques at the character level. This allows researchers to employ multi-label, multi-class classification techniques for persuasion detection with finer granularity [4]. The author utilized BERT [3] for this downstream task and evaluated using a modified F1 score to consider partial matching.

Several recent studies have explored persuasion detection via shared tasks like SemEval and ArAIEval [5,6,7]. BERT-based classifiers are a popular choice for these tasks due to their effectiveness in text classification [5]. There is a challenge with label distribution, as some persuasion techniques appear much less frequently than others. Moreover, most tokens within the data lack any persuasion labels. This is addressed by employing techniques like class weighting during loss calculation [8]. Additionally, multi-task architectures utilizing shared representations from pre-trained models like BERT have shown good results [9]. For persuasion detection in Arabic, previous works are commonly based on AraBERT [10] [7]. Propaganda detection in Arabic also benefits from preprocessing steps such as reversing code-switching and emoji conversion [11].

Pre-trained multilingual models are integral to the NLP tasks for low-resource languages. BERT [3] itself offers two multilingual versions: cased and uncased. These models are impressive in their scope, being trained on over 100 languages. The training process leverages masked language modelling and next token prediction objectives, allowing the model to learn generalizable representations across languages. XLM [12] is another set of multilingual models that uses translation objective alongside causal and masked language modeling for pre-training. Similarly, mBART [13] builds upon the BART model [14] by using a multilingual pre-training objective. The objective is reconstructing the original text from a corrupted version in multiple languages, allowing mBART to develop robust denoising capabilities.

The growing popularity of cross-lingual transfer learning offers a promising approach to improve performance on Arabic NLP tasks. This is demonstrated by employing task-specific fine-tuning on English and French data to improve Arabic NLU performance [15]. Similarly, for abstractive summarization of Arabic text, fine-tuning multilingual models (mBERT and mBART) on Hungarian or English before fine-tuning again on Arabic data demonstrated performance gains [16]. These findings highlight the effectiveness of cross-lingual transfer learning in improving the performance of Arabic language processing tasks.

Methodology

In this section, we describe the methodology employed for detecting persuasion techniques in Arabic articles using a multilingual BERT model fine-tuned on English data.

Data Preparation

The data for this task was provided in the form of article files, with the corresponding labels given in a separate file. The label file contained information about the persuasion techniques used and the offsets indicating the span of text within the articles where these techniques were applied. There are 23 labels representing different persuasion techniques. These techniques are identified within the text at the token level, allowing for multi-label classification where each token can be associated with one or more techniques. This detailed annotation allows the model to recognize and classify multiple techniques within a single span of text.

For preprocessing, we first split the articles into paragraphs. This was done based on empty lines, effectively treating each paragraph as a separate instance. Once the articles were divided into paragraphs, we calculated the offsets for the persuasive spans within each paragraph. This allowed us to align the provided labels with the appropriate paragraphs.

Task Formulation

We formulated the task as a multi-class, multi-label token classification problem. This means that each token (or word) in the input text could be classified into one or more persuasion technique categories. This approach enables the model to recognize multiple techniques that may be present in a single span of text. After predicting labels for each of the tokens, consecutive tokens with the same labels define a span. Table 1 demonstrates an example.

Table 1

An input sequence of length 256, where each token can be assigned one or more persuasion techniques. Consecutive tokens with the same labels form a span. In this example, {𝑥 3 , 𝑥 4 , 𝑥 5 } form a span for 𝑡 1 ; {𝑥 1 , 𝑥 2 } form a span for 𝑡 2 ; {𝑥 5 , 𝑥 6 } form a span for 𝑡 2 ; {𝑥 4 } is a span for 𝑡 3 ; and {𝑥 6 } is a span for 𝑡 3 .

𝑥 1 𝑥 2 𝑥 3 𝑥 4 𝑥 5 𝑥 6 ... 𝑥 256 𝑡 1 0 0 1 1 1 0 0 0 𝑡 2 1 1 0 0 1 1 0 0 𝑡 3 0 0 0 1 0 10

Model and Training

We employed a multilingual BERT model for this task. Multilingual BERT (mBERT ) is pre-trained on multiple languages, including Arabic and English, making it suitable for cross-lingual transfer learning.

For the loss calculation, we used binary cross-entropy, which is well-suited for multi-label classification tasks.

Given the lack of Arabic training data and the zero-shot nature of the task for Arabic, we used the provided English training data to fine-tune the mBERT model. Since there was no Arabic data provided for validation, we utilized the Arabic validation dataset from the ArAIEval shared task on propaganda detection 2024. 1 This validation dataset consists of 921 documents, with an average of 30.25 tokens per document, and follows the same labelling and annotation guidelines. The following hyperparameters were used during training:

• Learning Rate: 5e-5 • Number of Epochs: 75 • Maximum Input Length: 256 tokens Additionally, we utilized pos_weights to adjust the loss calculation. This helps in handling the class imbalance, ensuring that the model does not become biased towards the more frequent classes.

Results and Discussion

For evaluation, we used the modified F1-micro score, which accounts for partial matching of the spans. All the scores reported in this paper use that modified F1. Our model, fine-tuned using the English training data and validated on the Arabic dev dataset, achieved an F1-micro score of 0.0998 on the dev set. When evaluated on the test set, the model's performance improved significantly, achieving an F1-micro score of 0.3009. The difference in performance between the dev and test sets could be attributed to the domain-specific nuances and potential distributional differences in the test set.

Below is a detailed breakdown of the F1-micro scores per technique on the validation set, as shown in Table 2. The results reveal a significant variation in the model's performance across different persuasion techniques. Techniques such as Appeal to Time, Consequential Oversimplification, and Appeal to Values were detected more reliably, indicating that the model can effectively identify these patterns.

In contrast, techniques like Loaded Language, Straw Man, and Whataboutism showed moderate performance. Techniques like Questioning the Reputation, Repetition, False Dilemma-No Choice, and Appeal to Hypocrisy posed significant difficulties for the model. These techniques may be underrepresented in the training data, further complicating their detection.

The variation in performance can also be attributed to the nature and categorization of the techniques. Techniques that belong to the same category, such as different types of logical fallacies or emotional appeals, may share linguistic features that the model struggles to distinguish. For example, both Straw Man and Whataboutism involve misrepresentation or diversion tactics, which could confuse the model. On the other hand, techniques like Appeal to Values and Appeal to Popularity, which are more explicit and direct, tend to be easier for the model to identify.

It's important to note that no Arabic data was available for training. We relied on the English training data to fine-tune the multilingual BERT model. This cross-lingual transfer learning approach introduces additional challenges due to differences in linguistic structures and contextual usage between English and Arabic.

Conclusion and Future Work

With the increasing sophistication of persuasion techniques, particularly in Arabic-language content, it is crucial to focus research efforts on this area. This study investigated the effectiveness of a multilingual BERT model fine-tuned on English data for the task of Arabic persuasion detection. English was selected as the training language due to its extensive resources in Natural Language Processing (NLP) tasks, including propaganda detection. Our aim was to evaluate how these abundant resources could be leveraged to benefit languages with fewer resources, such as Arabic. This work achieved first place for Arabic on the leaderboard for the test set, demonstrating the potential of cross-lingual transfer learning [17]. However, there is still room for improvement.

Future work can explore how other high-resource languages impact the performance on Arabic. There might be various strategies to enhance the model's performance. Increasing the diversity and quantity of training data, particularly for techniques where performance was low, through data augmentation or the collection of additional labelled data, can help balance the dataset. Advanced fine-tuning techniques like focal loss can adjust the loss function to focus more on hard-to-classify examples, while dynamic sampling strategies can address class imbalance.

Additionally, incorporating more sophisticated features such as syntactic and semantic information, part-of-speech tags, or dependency parsing can provide the model with greater context and improve classification accuracy. Exploring alternative hidden layer representations within BERT may also yield better classification performance. By addressing these areas, future research can further improve the accuracy and robustness of models in detecting a wide range of persuasion techniques, ultimately enhancing their utility in real-world applications.