This paper provides an overview of our contributions to the PAN at CLEF2024 Oppositional thinking analysis shared task, which focuses on distinguishing between conspiratorial and critical thinking narratives. The competition featured two main tasks. The first task is a binary classification task that aims at determining whether a text is conspiratorial or critical. The second task is a span-level detection task, in which the goal is to detect elements of oppositional narratives in the texts. Two annotated datasets, one in English and one in Spanish, were provided, each of 5K telegram comments. Our best-performing approaches combined custom fine-tuned Transformer models with data augmentation techniques. We achieved an F1-Score of 0.8917 for English and of 0.8293 for Spanish for task 1, and a span-F1 score of 0.6279 for English and 0.6129 for Spanish for task 2. Our task 2 approach achieved the best results in the shared task for both English and Spanish.
Conspiracy theories ofer elaborate explanations for significant events, attributing them to hidden
schemes by secretive and powerful groups. Recently, there has been increasing interest in automatically
detecting these theories in text, often framed as a binary classification problem, with some approaches
exploring multi-label or multi-class classification. However, key issue with existing methods is their
dificulty in distinguishing between critical thinking and conspiracy theories [
The shared task “Oppositional thinking analysis: Conspiracy theories vs critical thinking narratives”1
[
Our approach to these tasks involved the use of custom fine-tuned Transformer models, which were enhanced through data augmentation techniques. For the binary classification task, we employed a Soft Voting Ensembling method combining multiple Transformer models to improve robustness and accuracy.
For the span-level detection task, we treated the problem as a token classification task, segmenting text into sentences to mitigate issues related to text length limitations in Transformer models.
In this section, we describe our submitted systems. Our approaches for distinguishing between critical and conspiracy texts (Task 1) are described in Section 2.1, and our approaches for detecting elements of oppositional narratives (Task 2) in Section 2.2.
For Task 1, the general approach involved fine-tuning Transformer-based models and applying data augmentation techniques. Both English and Spanish datasets were processed similarly. The main diference between Run 1 and Run 2 lies in the method used to make predictions. In Run 1, only the best model checkpoint was used to make predictions. In Run 2, an ensembling method was employed, which combined the predictions from multiple models.
The approach for both languages involved several key steps. We experimented with various
Transformer-based models for fine-tuning and diferent hyperparameters. In addition to model selection
and training, data augmentation played an important role in our approach. To increase the diversity and
quantity of training data, we applied translation-based augmentation. For this, for English, the Spanish
dataset was translated into English using the Helsinki-NLP/opus-mt-es-en2 model [
In Run 1, predictions were made using the best model checkpoints identified during the training
phase. However, in Run 2, we enhanced the prediction process by employing an ensembling method.
This is described in more detail in the following two subsections.
2.1.1. Run 1 task 1
2.1.2. Run 2 task 1
In Run 1, we focused on using the best model checkpoint to make predictions for both English and
Spanish datasets. For English, the facebook/roberta-base4 [
In Run 2, an ensembling approach was used. The general approach involved using a Soft Voting Ensembling method composed of three custom fine-tuned Transformer-based models and data augmentation. We used the following three models for English:
For Spanish, we used the following three models: 2https://huggingface.co/Helsinki-NLP/opus-mt-es-en 3https://huggingface.co/Helsinki-NLP/opus-mt-en-es 4https://huggingface.co/FacebookAI/roberta-base 5https://huggingface.co/dccuchile/bert-base-spanish-wwm-uncased 6https://huggingface.co/FacebookAI/roberta-base 7https://huggingface.co/google-bert/bert-base-uncased 8https://huggingface.co/allenai/scibert_scivocab_uncased
For ensembling, the best checkpoint from each fine-tuned Spanish model was used in a Soft Voting
ensemble for predictions. This approach involved combining predictions from multiple models using a
Soft Voting ensemble [
We approached Task 2 by fine-tuning a transformer model with a token classification head, therefore
treating it as a token classification problem. Having only one head (instead of a classification head per
label, as is implemented in the provided baseline [
We also performed data augmentation. We could not easily augment the data through translation,
because working at the span-level means that the annotated spans are provided in terms of indices
that match the original text. Therefore, we used an alternative data augmentation technique which
consisted in replacing words in the texts by synonyms or semantically-related words, using static
word embeddings (word2vec) [
In Run 2, the best model checkpoint from Run 1 was trained for one more epoch using the augmented dataset. This final model checkpoint was then used to detect the elements of oppositional narratives in the test dataset. 9https://huggingface.co/dccuchile/bert-base-spanish-wwm-uncased 10https://huggingface.co/PlanTL-GOB-ES/bsc-bio-ehr-es-pharmaconer 11https://huggingface.co/bertin-project/bertin-roberta-base-spanish 12https://github.com/mmihaltz/word2vec-GoogleNews-vectors 13https://github.com/dccuchile/spanish-word-embeddings 14https://huggingface.co/FacebookAI/roberta-base 15https://huggingface.co/PlanTL-GOB-ES/roberta-base-bne
Experiments were conducted on an NVIDIA GeForce GTX 1080 (8 GB memory). To ensure uniformity and comparability of results, the same experimental setting was consistently applied across all tasks, runs, and languages under study. The experimental setup was meticulously designed to ensure optimal performance and eficient resource utilization.
StratifiedKFold cross-validation [
The training process spanned 15 epochs. A weight decay of 0.01 was applied as a regularization technique to penalize large weights. A custom linear learning rate scheduler was employed, adjusting the learning rate from an initial value of 2e-5 to a final value of 2e-6 over the total number of training epochs. Gradient accumulation steps were set to 4, efectively increasing the batch size without inflating the memory footprint by accumulating gradients over multiple steps before updating the model’s weights. The training batch size per device was dynamically set based on available GPU memory, managed by a custom callback designed to dynamically adjust the batch size used during training and evaluation based on the available GPU memory. This ensures eficient resource utilization and prevents memory-related issues during training, especially when dealing with varying data sizes and model complexities. Upon completion of training, the best model, as determined by the F1 score, was loaded.
Participants were provided with a JSON file containing all texts in the training dataset along with
their annotations. Each text is represented by a dictionary that includes the ID, tokenized text, binary
category, and span annotations. Span annotations are provided as a list of dictionaries, with each
dictionary representing an annotated span and detailing the span’s category and text, specified by the
start and end characters. The training subset, comprising 4000 records, was released with all annotations,
while the test subset, consisting of 1000 records, was released only with “id” and the “text” field [
Label CRITICAL CONSPIRACY
English As summarized in Table 1 and shown in Figures 1a and 1b, the binary classification task for distinguishing critical from conspiracy texts reveals an inherent class imbalance in both the English and Spanish datasets. The English dataset has 65.53% critical texts compared to 34.48% conspiracy texts, while the Spanish dataset shows a similar trend with 63.45% critical and 36.55% conspiracy texts. This imbalance poses a challenge for model training. The similarity in the proportions of critical and conspiracy texts across both languages suggested that the models trained on one dataset might be adaptable to the other with minimal adjustments. That was indeed the approach adopted for the shared task. (a) Distribution of span text categories for the English dataset.
(b) Distribution of span text categories for the Spanish dataset. The token classification task aims to identify key elements within oppositional narratives. As shown in Figures 2a and 2b and in Table 2, the distribution of these elements varies significantly between the English and Spanish datasets.
The oficial evaluation metrics were diferent for each subtask. For subtask 1, which involved distinguishing between critical and conspiracy narratives, the Matthews Correlation Coeficient (MCC) was used as the primary evaluation metric [16]. Additionally, the F1-macro, F1-conspiracy and F1-critical metrics were provided for subtask 1. For subtask 2, focusing on span-level detection of narrative elements, the macro-averaged span-F1 was used as the primary evaluation metric [17]. In addition, the span-P, span-R and micro-span-F1 metrics were reported for subtask 2.
The shared task organizers provided two baseline models for these tasks. For subtask 1, the baseline
is a standard BERT classifier [
5.1.1. English Results task 1 In our analysis of the English dataset, the first run exhibited good performance metrics. The Matthews Correlation Coeficient (MCC) was 0.7574, indicating a robust ability to distinguish between diferent narrative types. The F1-macro score of 0.8769 further supported the model’s high overall classification capability. Notably, the F1 score for critical thinking texts was 0.92, compared to 0.8338 for conspiracy texts. This disparity suggests that the model more efectively identified critical thinking. However, it is important to note that these results are below the baseline, which had an MCC of 0.7964 and an F1-macro score of 0.8975. This baseline is indeed a hard baseline and dificult to beat.
In the second run, the ensembled model demonstrated improved reliability, with the MCC rising to 0.7872. The F1-macro score also increased to 0.8917, indicating enhanced overall performance. The F1 scores for conspiracy and critical texts were 0.8536 and 0.9297, respectively, showing more balanced and accurate classifications. Despite these improvements, the model still did not surpass the baseline, highlighting the baseline’s strong performance and the challenges in achieving higher accuracy. 5.1.2. Spanish Results task 1 For the Spanish dataset, the first run showed moderate performance with an MCC of 0.6147, indicating a need for further improvement. The F1-macro score was 0.795, reflecting decent overall performance but highlighting areas for enhancement. The model struggled more with conspiratorial texts, achieving an F1 score of 0.7179 for conspiracy versus 0.872 for critical texts, likely due to the specific linguistic challenges presented by the Spanish language. The baseline for Spanish had an MCC of 0.6681 and an F1-macro score of 0.8339, indicating that the baseline was also strong for this language.
In the second run, there was a noticeable improvement in performance. The MCC increased to 0.6722, and the F1-macro score rose to 0.8293, indicating better overall performance. The F1 scores for conspiracy and critical texts improved significantly to 0.7699 and 0.8887, respectively. These improvements suggest that the ensembled model became more adept at handling linguistic features specific to Spanish, leading to more balanced and accurate classifications. This second run beat the strong baseline performance, reflecting significant progress.
5.2.1. English Results task 2 In the first run for the English dataset, the model achieved a span-P of 0.5832 and a span-R of 0.6856, indicating moderate precision but better recall. The span-F1 score was 0.6293, and the micro-span-F1 score was 0.6074, suggesting a balanced performance with a need for improvement in precision. The baseline for this task had a span-F1 score of 0.5323 and a micro-span-F1 of 0.4998, showing that our model performed significantly better than the strong baseline.
In the second run, there was a slight improvement in precision, with a span-P of 0.5859 and a span-R of 0.679. The span-F1 score remained relatively consistent at 0.6279, and the micro-span-F1 score increased marginally to 0.6120. These modest enhancements reflect steady progress in performance and show that our model maintained competitive performance with the baseline. 5.2.2. Spanish Results task 2 For the Spanish dataset, the first run had a span-P of 0.5997 and a span-R of 0.6193. The span-F1 score was 0.6089, and the micro-span-F1 score was 0.6051. The model performed significantly better at span detection in Spanish compared to English, possibly due to distinct narrative markers in the language. The baseline for this task had a span-F1 score of 0.4934 and a micro-span-F1 of 0.4952, indicating that our model significantly outperformed the baseline in both metrics. In the second run, the performance improved, with a span-P of 0.6159, a span-R of 0.6129, and a span-F1 of 0.6129, maintaining a significant advantage over the baseline.
Overall, retraining the model on the entire dataset for one epoch, using the augmented dataset without a validation and test set, resulted in improved performance during the second run.
In the future, we will thoroughly investigate why we achieved the best results in the competition in the second, more challenging task, but not in the first, theoretically easier one. By examining the confusion matrices for binary classification, we can pinpoint where the model’s predictions deviate from actual categories. For instance, in the Spanish dataset, the model misclassified "CRITICAL" texts as "CONSPIRACY" with a 5.52% error rate and "CONSPIRACY" texts as "CRITICAL" with a 31.4% error rate. In the English dataset, these error rates were 4.12% and 19.7%, respectively, indicating greater dificulty in detecting "CONSPIRACY" texts, especially in the Spanish dataset. Misclassifications may arise from language ambiguity, training data limitations, and cultural nuances. For token classification, both datasets exhibited signicfiant issues with spans that should not be annotated, often being misclassified into various categories. This could be due to model overconfidence, lack of enough training data, or to the subjective nature of the task.
The detailed analysis of the results highlights the significant progress made in distinguishing between conspiratorial and critical thinking narratives and detecting narrative elements within texts. The continuous improvements observed between runs emphasize the importance of data augmentation, model fine-tuning, and language-specific adaptations.
Future work should focus on refining these models further, exploring advanced augmentation techniques, and incorporating diverse datasets to improve generalizability. Additionally, developing specialized tools for intergroup conflict analysis and enhancing content moderation strategies will be crucial for addressing the challenges posed by misinformation and fostering a healthier information environment.
The work of Mariona Coll Ardanuy is funded by the research project FairTransNLP, grant PID2021124361OB-C31, funded by MCIN/AEI/10.13039/501100011033 and by ERDF, EU A way of making Europe. imbalanced datasets, in: J. C. Xavier-Junior, R. A. Rios (Eds.), Intelligent Systems. BRACIS 2022. Lecture Notes in Computer Science, volume 13653 of Lecture Notes in Computer Science, Springer, Cham, 2022. doi:10.1007/978-3-031-21686-2_43. [16] D. Chicco, N. Tötsch, G. Jurman, The Matthews correlation coeficient (MCC) is more reliable than balanced accuracy, bookmaker informedness, and markedness in two-class confusion matrix evaluation, BioData mining 14 (2021) 1–22. [17] G. Da San Martino, Y. Seunghak, A. Barrón-Cedeno, R. Petrov, P. Nakov, et al., Fine-grained analysis of propaganda in news article, in: Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP), Association for Computational Linguistics, 2019, pp. 5636–5646. [18] K.-H. Thung, C.-Y. Wee, A brief review on multi-task learning, Multimedia Tools and Applications 77 (2018) 29705–29725.