In the realm of Natural Language Processing, the computational detection and analysis of conspiracy theories (CTs) within textual data has gained significant momentum [4]. CTs are elaborate narratives attributing significant events to covert actions by powerful clandestine groups, contrasting with critical thinking, which challenges mainstream beliefs without endorsing conspiracies. Differentiating between these is crucial, as mislabeling opposing views as conspiratorial may sway individuals towards extreme viewpoints. Current research predominantly focuses on binary classification tasks aimed at accurately distinguishing between conspiratorial and critical texts. Existing methodologies for distinguishing between conspiratorial and critical texts typically involve leveraging advanced natural language processing techniques and machine learning models. Some common approaches include:
• Feature-based Classification: Using algorithms like SVMs (Support Vector Machines) [5] or logistic regression, which analyze word frequencies, n-grams, and syntax to classify texts. • Graph-based Methods: Representing texts as graphs, where nodes represent entities (e.g., words or phrases) and edges represent relationships (e.g., co-occurrence). Graph-based methods can capture structural patterns and semantic relationships indicative of conspiratorial or critical narratives.
• Sentiment Analysis: Analyzing the sentiment expressed in texts can provide insights into whether the text is promoting conspiratorial beliefs (e.g., distrust, fear) or engaging in critical discourse (e.g., skepticism, questioning).
Subtask 2 focuses on token-level classification within oppositional narratives, distinguishing between conspiracy theories and critical thinking. It aims to identify specific text segments-goals, effects, agents, facilitators, objectives, and negative effects-using advanced NLP techniques. This approach enhances understanding of nuanced narrative elements for effective content moderation and societal discourse analysis. The approach includes:
• Topic Modeling: Techniques like Latent Dirichlet Allocation (LDA) [6] or Non-Negative Matrix Factorization (NMF) [7] The two tasks discussed in this paper and their successful implementation collectively advance the field by enabling automated detection and analysis of conspiratorial narratives, facilitating nuanced understanding and effective management of such discourse in various domains, and thereby create stable and peaceful platforms for discussion on public health issues.
There are two tasks that have been worked upon, the first involves distinguishing between critical and conspiracy texts, while the second focuses on detecting elements within oppositional narratives. The dataset contains Telegram messages in English and Spanish. Both the tasks are performed exclusively in English. Subtask 1 involves classifying texts into two categories: (1) messages that critically question public health decisions without promoting conspiracy theories, and (2) messages that attribute pandemic or health decisions to secret, influential conspiracies. Each text in the dataset is labeled as either CONSPIRACY or CRITICAL. Evaluation of model performance is done based on Matthews Correlation Coefficient (MCC) [8], with a baseline established by a BERT classifier [9].
Subtask 2 involves a token-level classification challenge where the goal is to identify specific text segments that represent essential elements in oppositional narratives. Each text data in the input dataset contains span texts along with their starting and ending positions and the type of oppositional narrative the span text belongs to out of: AGENT, FACILITATOR, VICTIM, CAMPAIGNER, OBJECTIVE, and NEGATIVE EFFECT. The performance of models is evaluated using the macro-averaged span-F1 score, which assesses overall accuracy across all span categories.
This section outlines the process of preparing data for the two tasks.
In the data pre-processing stage for subtask 1, the dataset is initially split into two subsets: one for critical messages and another for conspiracy messages . Each subset is filtered based on the "category" column values. Exploratory Data Analysis (EDA) [10] begins with a count plot to visualize the distribution of categories ("CRITICAL" and "CONSPIRACY") [Fig. 1]. This provides an initial understanding of the dataset's class distribution. Following EDA, data cleansing involves checking for missing values. Addressing any missing data ensures the dataset is ready for subsequent steps such as tokenization, feature extraction, and model training for binary classification.
Subtask 2 pre-processing starts with extracting annotations from each JSON (JavaScript Object Notation) entry, gathering crucial details about the relevant text spans and their corresponding categories. This step prepared the dataset for subsequent pre-processing, ensuring its alignment with the machine learning pipeline. Post annotation extraction, the Hugging Face AutoTokenizer [11] tailored for BERT models was employed to tokenize the dataset. Tokenization converted raw text sequences into numerical token IDs suitable for ingestion by the BERT-based model. To meet BERT's input specifications, a truncation strategy was applied to handle sequences exceeding the model's maximum input length. This approach maintained consistency in sequence lengths across the dataset, optimizing computational efficiency during training and evaluation phases.
The Tiny BERT Text Classifier model [12] is a variant of BERT optimized for English text classification tasks, specifically focusing on the SST (Stanford Sentiment Treebank)-2 dataset [13] for sentiment analysis. Built on transformer architecture, this model enables bidirectional understanding of language nuances, enhancing accuracy in classifying sentences as either critical or conspiracy in nature. This capability is crucial for distinguishing between texts that question public health decisions (critical) and those that attribute them to malevolent conspiracies (conspiracy). By leveraging bidirectional context, these models can capture subtle linguistic cues that differentiate between these two types of narratives effectively.
Methodologies of subtask 2 typically involve initial dataset preparation by sourcing annotated text spans and categorizing them for training, validation, and test sets to ensure unbiased model evaluation. Utilizing tools like AutoTokenizer from Hugging Face's Transformers library [11], raw text sequences are tokenized into numerical token IDs, with strategies like truncation and padding managing sequence lengths. Model selection focuses on transformer-based architectures pretrained on extensive text corpora, fine-tuned for span-level classification using transfer learning techniques. Training optimizes model parameters with Adam optimizer and Binary Cross-Entropy loss [14], while evaluation metrics such as span-level F1-score, precision, recall, and micro-averaged F1-score assess model performance.
To implement subtask 1, the dataset is structured into a format where each text sample is categorized either as "CRITICAL" or "CONSPIRACY". The BertClassifier model from keras-nlp.models is then employed with specific configurations for binary classification. Pre-trained weights are loaded, and a sigmoid activation function is utilized to facilitate binary output. The model is trained on the training data to distinguish between critical viewpoints and conspiracy theories regarding public health decisions. Evaluation is performed on the test set to assess the model's capability in accurately classifying these texts. This approach leverages the capabilities of BERT for semantic understanding, thereby supporting the task's objective of discerning between critical analyses and conspiratorial narratives in the domain of public health.
The BERT-Based Multi-Label Text Classifier was implemented in Python using the bert-base-uncased model architecture from Hugging Face's Transformers library. The dataset, sourced from JSON files, contained annotated text spans (span text) categorized into specific classes (category). After partitioning the dataset into training (70%), validation (10%), and test (20%) sets, annotations were extracted to prepare the data for tokenization. The Hugging Face AutoTokenizer [11] was employed to tokenize the text sequences into numerical token IDs, with a truncation strategy applied to handle sequences longer than BERT's maximum input length. The model was fine-tuned for multi-label classification, optimizing with the Adam optimizer and Binary Cross-Entropy loss function [14] over multiple epochs. Evaluation on the validation set involved monitoring metrics such as accuracy, precision, recall, and F1-score to ensure model performance. Finally, the trained model and tokenizer were saved for deployment, emphasizing reproducibility and scalability in future applications. The BERT-Based Multi-Label Text Classifier was implemented in Python using the bert-base-uncased model architecture from Hugging Face's Transformers library. The dataset, sourced from JSON files, contained annotated text spans (span text) categorized into specific classes (category). After partitioning the dataset into training (70%), validation (10%), and test (20%) sets, annotations were extracted to prepare the data for tokenization. The Hugging Face AutoTokenizer was employed to tokenize the text sequences into numerical token IDs, with a truncation strategy applied to handle sequences longer than BERT's maximum input length. The model was fine-tuned for multi-label classification, optimizing with the Adam optimizer and Binary Cross-Entropy loss function over multiple epochs. Evaluation on the validation set involved monitoring metrics such as accuracy, precision, recall, and F1-score to ensure model performance. Finally, the trained model and tokenizer were saved for deployment, emphasizing reproducibility and scalability in future applications.
Based on the provided results for subtask 1 and subask 2 in English, the performance of the BERT-Based Multi-Label Text Classifier was evaluated. For subtask 1 [Table 1], focusing on conspiracy and critical categorization, the model achieved an F1-macro score of 0.3700 and 0.8255, respectively, indicating moderate performance in identifying critical texts compared to conspiracy-related ones . In subtask 2 [Table 2], which evaluated span-level F1-score and micro-averaged F1, the model attained scores of 0.0150 and 0.0600, respectively, suggesting challenges in precise span-level predictions. The implementation utilized Python with the bert-base-uncased model from Hugging Face's Transformers library, leveraging AutoTokenizer for tokenization and fine-tuning with Adam optimizer and Binary Cross-Entropy loss. The results underscore the model's effectiveness in critical text classification but highlight areas for improvement in span-level prediction accuracy.
The task Oppositional Thinking Analysis: Conspiracy vs Critical tackles the challenge of distinguishing conspiratorial from critical narratives in oppositional texts, especially regarding COVID-19. Conspiracy theories, often depicting events as manipulated by secretive, powerful groups, are complex and hard to separate from genuine critical thinking. The competition aims to enhance understanding and automatic detection of these narratives, crucial for content moderation on social media. Differentiating conspiratorial messages from critical ones is essential, as mislabeling can push individuals toward conspiracy communities. This task involved developing sophisticated NLP models to discern these nuances for accurate classification and better content management. The approach included preprocessing steps like text cleaning and feature extraction using TF-IDF (Term Frequency-Inverse Document Frequency) [15] and word embeddings. Both traditional machine learning algorithms, such as logistic regression and support vector machines, and advanced deep learning models, like LSTM (Long Short-Term Memory) and BERT, were used. Evaluations with metrics such as accuracy, precision, recall, and F1-score showed deep learning models, especially BERT, outperformed traditional ones. Cross-validation ensured robustness and mitigated overfitting. The methodologies from this competition promise to improve automatic detection of conspiratorial versus critical narratives, aiding effective content moderation on digital platforms.