The Oppositional thinking analysis: Conspiracy theories vs critical thinking narratives task of PAN at CLEF 2024 involves two challenges: first, distinguishing between conspiracy and critical narratives as Subtask 1, and second, identifying key elements of oppositional narratives as Subtask 2. We consider these two challenges as binary classification and sequence labeling problems, respectively. We will perform both tasks in English and Spanish. In this paper, we introduce our method to address these challenges by fine-tuning a BERT-based model with an added BiGRU layer for Subtask 1 and employing a multi-task learning method for Subtask 2. Finally, our model for English achieves MCC scores of 0.821 in Subtask 1 and Span-F1 scores of 0.569 in Subtask 2 on the official test set.
As it is acknowledged that conspiracy theories pose significant harm to society and are challenging to identify [1], the difficulty lies in distinguishing them from critical thinking narratives, as both share similarities in oppositional thinking. However, it is crucial to differentiate between them, as failure to do so could push people toward conspiracy communities, as shown in [2]. The PAN at CLEF 2024 task [3] on oppositional thinking analysis [4] aims to address this problem. It includes two subtasks framed as a binary classification task and a token-level classification task, respectively.
The automatic detection of conspiracy theories in text using pre-trained language models has proven effective [5] in recent years. Combining the transformer-based model with downstream neural networks has achieved state-of-the-art performance in similar tasks [6]. Inspired by related works, we employ CT-BERT [7] and BiGRU (Bidirectional Gated Recurrent Units) [8] to address this task. By integrating the BERT-based layer with the BiGRU layer, we leverage the benefits of deep contextual embeddings and sequence-sensitive features.
At PAN 2024 there are two subtasks proposed for oppositional thinking analysis:
• Subtask 1: Distinguishing between critical and conspiracy texts. It is a binary classification task that aims to distinguish between two types of messages: the first contains critical messages that scrutinize significant decisions within the public health sector without endorsing a conspiratorial mindset; the second includes messages that interpret the pandemic or public health decisions as the result of a malignant conspiracy orchestrated by secretive, powerful entities. Our task is to categorize these texts into distinct categories: CONSPIRACY or CRITICAL.
• Subtask 2: Detecting elements of the oppositional narratives. It is a token-level classification task aimed at recognizing text spans corresponding to the key elements of oppositional narratives. A span-level annotation scheme that identifies the Agents (A), Facilitators (F), Campaigners (C), Victims (V), Effects (E), Objectives (O) in the oppositional narratives was developed. Our task is to identify specific spans in texts that should be annotated with the corresponding labels.
Generally speaking, our method consists of two main parts: the BERT-based encoder and the BiGRU downstream neural network layer for both Subtask 1 and Subtask 2. Our method involves three primary steps: 1) fine-tune the pre-trained BERT-based model with the given training dataset, 2) feed the sequence of embeddings from the BERT-based model into a BiGRU layer and 3) Use the outputs from the BiGRU layer, typically the final hidden states that encapsulate the information from the entire sequence, to classify the text into categories (e.g., critical or conspiracy) in Subtask 1 or to combine with different task heads for span annotation in Subtask 2.
In this section, we introduce the architecture for Subtask 1. Figure 1 shows the whole architecture.
Input Data ⋯ ⋯ The CT-BERT model is selected as our encoder, which was trained on a large dataset of COVID-19 Twitter messages. The corpus for this PAN 2024 task consists of COVID-19 Telegram texts, making our model particularly well-suited due to its training on similar content. Consequently, this model is expected to outperform other BERT-based models due to its superior understanding of this specific domain. Additionally, we have chosen RoBERTa [9] as a contrasting model to verify whether these expectations hold.
The BERT-based model provides rich contextual embeddings by considering the left and right contexts within the transformer architecture. The addition of a BiGRU layer introduces an extra level of sequential processing. It processes information in both forward and backward directions across the text, offering a comprehensive view of the temporal dependencies. Once the BERT-based layer has generated the sequence outputs, they are fed into the BiGRU layer. The BiGRU layer synthesizes the information captured by the BERT layer, adding a layer of understanding. This enhancement aids in detecting subtle cues and patterns that differentiate various narrative types.
The BiGRU outputs are then passed through additional dropout layers for regularization, followed by a linear classification layer that maps the BiGRU outputs to the target category.
The core architecture for Subtask 2 remains the same, however, we employ a multi-task learning method to more effectively address the specific challenges posed by Subtask 2, as shown in Figure 2. Given that the key elements to be identified in a text fall under one of six categories-Agents (A), Facilitators (F), Campaigners (C), Victims (V), Effects (E), and Objectives (O)-each can be considered a separate token classification task. All these tasks share the same need for embeddings. Therefore, we utilize a BERT-based encoder (primarily CT-BERT) as the backbone of our architecture, with token classification layers serving as task-specific heads. This forms our multi-task classifier architecture. Additionally, the token classification layer is integrated with a BiGRU layer, and through BIO tagging, we achieve the span output for each category.
Recent research [10] has proven the effectiveness of a multi-task classifier based on the domain-specific CT-BERT model. Utilizing a shared encoder, our model efficiently learns universal representations beneficial across all tasks, while the dedicated task modules concentrate on task-specific features.
For evaluation, we used the official metrics provided to evaluate Subtask 1: Matthews Correlation Coefficient (MCC) [12], per-class F1 scores: F1-Consp and F1-Crit and macro-averaged F1.
And we used the following metrics in Subtask 2: span-F1 [13], span-recall, span-precision and micro-span-F1.
The organisers of each subtask provided baselines in both languages for each subtask. BERT classifier is used for Subtask 1, and BERT-based multi-task token classifier is used for Subtask 2.
While training, we preprocessed the training set and divided it using stratified 3-fold cross-validation.
Our model is trained using a cross-entropy loss function and utilizes the AdamW optimizer with a learning rate of 2e-5, incorporating a scheduler for learning rate adjustments. Other hyperparameters include a batch size of 16 and a training duration of three epochs.
In Subtask 1, we selected CT-BERT and RoBERTa for experiments on the English corpus, and bertspanish [14] for the Spanish corpus. Each model was tested both with and without an added BiGRU layer. In Subtask 2, we selected CT-BERT as backbone on the English corpus, and bert-spanish for the Spanish corpus. Each model was tested both with an added BiGRU layer.
During the training process for Subtask 1, we evaluated our models and compared them with the official baselines. We anticipate that the CT-BERT + BiGRU model will outperform other models on the English corpus. For the Spanish corpus, due to the limited availability of multilingual models for experimentation, we used BERT-Spanish with a BiGRU layer.
As shown in Table 1, our model performed better than both the baseline and RoBERTa + BiGRU, demonstrating the effectiveness of the CT-BERT + BiGRU model in this binary classification task. When compared with CT-BERT without the BiGRU, the version with BiGRU showed slight improvement. However, the BERT-Spanish + BiGRU model slightly fell short of the Spanish baseline.
The Table 2 shows that our model still holds up, indicating that our model is robust and neither overfits nor underfits the training set. However, the BERT-Spanish + BiGRU model performed worse than the baseline.
This paper mainly introduces our work on oppositional thinking analysis at PAN 2024. Our work utilizes a BERT-based model with a BiGRU layer to enhance performance in both binary classification and sequence labeling tasks within this domain. The results from the official testing datasets indicate that our method achieved an improvement of approximately 0.04 MCC scores in Subtask 1 and reached 4th place in the Official Ranking for the English corpus. While the English model demonstrated strong performance, the Spanish model was less successful, with only marginal improvements attributed to the BiGRU layer. Therefore, future work should focus on investigating how this method impacts multilingual tasks.
This work is supported by the Social Science Foundation of Guangdong Province, China (No.GD24CZY02)