Conspiracy theories are complex narratives that attempt to explain the ultimate causes of significant events as cover plots orchestrated by secret, powerful, and malicious groups. To analyze texts afecting adversarial thinking containing conspiratorial or critical narratives, PAN 2024 introduces Adversarial Thinking Analysis: Conspiracy vs. Critical Thinking Narratives. This evaluation subtask comprises two sub-tasks: subtask 1 requires distinguishing texts questioning public health decisions without endorsing conspiracy theories from those attributing these decisions to malicious conspiracies; subtask 2 involves extracting, identifying, and classifying key elements of adversarial narratives. The hyperparameters of the RoBERTa and XLM-RoBERTa models are tuned to accomplish these tasks. After the training, these models are employed to make predictions and evaluate the test set. Ultimately, the following metrics are achieved in performance: in subtask 1, an MCC of 0.7758 for English texts and an MCC of 0.6871 for Spanish texts are obtained. In subtask 2, the span-F1 score reached 0.5666 for English and 0.4903 for Spanish texts.
multi-tasking approach is conducted. Subsequently, these models are used to predict and evaluate the test set.
Subtask 1 is approached as a binary classification task, and Subtask 2 is approached as a token-level classification subtask. The research methodology consists of three steps: 1) data preprocessing, 2) model training, and 3) prediction.
For subtask 1, in the first step, text preprocessing is performed. In the second step, due to the limited sample size, the cross-validation method is applied to determine the hyperparameters of the RoBERTa and XLM-RoBERTa models for subtask 1. In the third step, the trained models are used to classify the text and tested against the oficial test set.
For subtask 2, the first step is to transform the text into a format acceptable to the models. Then, the RoBERTa and XLM-RoBERTa models are used for training. In the third step, the trained models are used for text recognition, followed by testing on the oficial test set.
The model is trained on the training data provided by the evaluation party. The training data can be accessed through a JSON file, part of the PAN@CLEF2024 shared task Oppositional Thinking Analysis. This JSON file contains 4000 texts in English and Spanish, encompassing all texts and their respective annotations in the training dataset. Each text entry is in dictionary format, recording the text ID, tokenized text content, binary category label, and span annotations. The span annotations consist of a series of dictionaries detailing a specific annotation span, including its category, start and end character indices, and corresponding text snippet. This link allows researchers to request access to dataset 1 designed specifically for subtasks 1 and 2.
For subtask 1, a binary classification approach based on RoBERTa and XLM-RoBERTa is adopted to distinguish between two types of texts: Those questioning public health decisions without propagating conspiracy theories and those attributing these decisions to malicious conspiracies. Initially, for datasets in English and Spanish, the ’text,’ ’id,’ and ’category’ are extracted from each data entry and subsequently regarded as forming a new dataset.
For subtask 2, the BIO tagging method is employed to identify and classify narrative elements within the text. This approach can identify the starting and ending positions and hierarchical structure of each narrative element. The text is tokenized, and each token is labeled (where ’B’ denoted the beginning of a narrative element, ’I’ represented the inside of a narrative element, and ’O’ stood for outside a narrative element) based on a predefined set of tags. Subsequently, the annotated data is fed into RoBERTa and XLM-RoBERTa models for model training.
For subtask 1, to address the issue of limited data volume, a -fold cross-validation method is employed, with the value of set to 5, evenly dividing the dataset into five parts. During each round of crossvalidation, one subset is selected as the validation set, while the remaining four subsets are combined to form the training set. Conducting five such tests allows each sample in the dataset can be used for training and validation. Ultimately, based on the results of these five sets of models, the hyperparameter combination that yields a better average performance is selected as the hyperparameter setting for the model. Ultimately, 25 epochs are selected for the training cycle (epochs attempted: 10, 15, 20, 25, and 30). The learning rate of 1 − 5 is chosen (learning rates attempted: 1 − 5, 2 − 5, and 3 − 5). The batch size of 64 is selected (batch sizes attempted: 32, 64, and 128). Model training is executed on NVIDIA A800 TENSOR CORE GPU hardware, where the cross-entropy loss function is opted for to measure the discrepancy between predictions and accurate labels, and the Adam [5] optimizer is employed to adjust the parameters of the model.
For subtask 2, the -fold cross-validation method and multi-tasking approach are utilized, with the value set to 5, thereby allowing each sample in the dataset to serve as training and validation data. Ultimately, 20 epochs are chosen for the training cycle of the model (epochs attempted: 10, 15, 20, 25, and 30). The learning rate of 1 − 5 is selected for the model (learning rates attempted: 1 − 5, 2 − 5, and 3 − 5). The batch size of 64 is selected (batch sizes attempted: 32, 64, and 128). During the training process, the multi-tasking approach is employed, where each named entity is assigned to a task, and tasks share parameters through RoBERTa or XLM-RoBERTa as they are trained. The training is conducted on NVIDIA GeForce RTX 3090 hardware. Regarding the selection of the loss function, the cross-entropy loss function is employed to calculate the discrepancy between predicted and actual labels, and the parameters are adjusted using the Adam optimizer.
The oficial evaluation metric for subtask 1 (critical vs. conspiracy classification) is MCC, while the
oficial metric for subtask 2 (span-level detection of narrative elements) is macro-averaged span-F1.[
For subtask 1 in English using RoBERTa, the score does not exceed the baseline when compared with it. The reason may be that the parameters determined by the -fold cross-validation method are not efective, thus afecting the performance of the model. The score for subtask 2 in English using RoBERTa exceeds the baseline model, possibly because the -fold cross-validation method determines a hyperparameter combination with better average performance.
For the Spanish texts of subtask 1 and subtask 2, a multi-tasking approach is employed to train the XLM-RoBERTa model, with the anticipation that the scores will surpass the baseline. Upon comparing the outcomes with the baseline, the score for subtask 1 in Spanish exceeds the baseline by 0.019 points. For subtask 2 with Spanish texts, the score does not surpass the baseline. This outcome is not anticipated, which is the research focus of our further research.
This work is supported by the Natural Science Platforms and Projects of Guangdong Province Ordinary Universities (Key Field Special Projects) (No. 2023ZDZX1023) L. Zettlemoyer, V. Stoyanov, Unsupervised cross-lingual representation learning at scale, 2020. arXiv:1911.02116. [5] I. Loshchilov, F. Hutter, Decoupled weight decay regularization, 2019. arXiv:1711.05101. [6] M. Fröbe, M. Wiegmann, N. Kolyada, B. Grahm, T. Elstner, F. Loebe, M. Hagen, B. Stein, M. Potthast, Continuous Integration for Reproducible Shared Tasks with TIRA.io, in: J. Kamps, L. Goeuriot, F. Crestani, M. Maistro, H. Joho, B. Davis, C. Gurrin, U. Kruschwitz, A. Caputo (Eds.), Advances in Information Retrieval. 45th European Conference on IR Research (ECIR 2023), Lecture Notes in Computer Science, Springer, Berlin Heidelberg New York, 2023, pp. 236–241. URL: https://link. springer.com/chapter/10.1007/978-3-031-28241-6_20. doi:10.1007/978-3-031-28241-6_20.