Conspiracy theories serve as significant conduits for disseminating false information, leading to the misguidance of the public and impacting the ability to make informed decisions. Identifying such theories poses numerous challenges. For this, PAN 2024 has introduced two tasks: the binary classification task that distinguishes between conspiratorial and critical texts, and the task of detecting elements of oppositional narratives. In this paper, Covid-Twitter-BERT and BETO are fine-tuned to address these challenges. The assessment results indicate that the model achieved MCC scores of 0.8198 and 0.6192 on the English and Spanish test sets for subtask 1, respectively. For subtask 2, span-F1 scores of 0.5886 and 0.4994 were attained on the English and Spanish test sets, respectively.
The contest encompasses two distinct subtasks. Subtask 1 is a binary text classification task, which requires determining whether the text falls into the category of "critical" or "conspiracy." Subtask 2 is a token-level classification task, which involves identifying the goals, efects, agents, and groups-inconflict within the text. Both tasks include texts in both English and Spanish.
Numerous studies have demonstrated the outstanding performance of Transformer-based models [
Data processing was initially conducted for model fine-tuning. The datasets for this task are presented in two languages, English and Spanish. Each dataset consists of 4000 texts with their annotations. Each text is associated with a dictionary containing the ID, tokenized text, the binary category, and the span annotations. 1Span annotations comprise a list of dictionaries, with each dictionary corresponding to an annotated span, which includes the span’s category and text, identified by the start and end characters. The data for each language was partitioned into a training set and a validation set in a 9:1 ratio.
For subtask 1, the BERT model is fine-tuned using the AutoModelForSequenceClassification as the model architecture to construct a binary classifier. This architecture is designed for sequence classification tasks, capturing the sequential characteristics of text data, thereby enhancing the model’s generalization capability. The fully connected layer of the model outputs the probability distribution of the target categories, which is then transformed into the probabilities of each class using the softmax activation function. During fine-tuning, the Adam optimizer and the cross-entropy loss function are employed, with the number of epochs set to 100. The model that performs best on the validation set, according to the evaluation results of each epoch, is selected and saved.
The CT-BERT and BETO models were fine-tuned, with CT-BERT being applied to English texts and BETO to Spanish texts, followed by an evaluation of their performance on the validation set. During the experimentation, it was observed that CT-BERT, once trained on a Spanish dataset, also demonstrated efective processing capabilities on the English dataset. This led to the hypothesis that CT-BERT possesses multilingual learning capabilities. Consequently, an attempt was made to merge the English and Spanish training datasets into a single multilingual dataset for training CT-BERT. It was found that, for Task 1-Spanish, the CT-BERT model trained with the multilingual dataset showed specific improvements over the model trained with a single language dataset, with an approximate increase of 0.012 in MCC on the validation set.
Furthermore, the model’s performance in Spanish was relatively poor. A multi-model voting approach was also attempted to enhance the model’s accuracy in Spanish texts. The models used for voting include: • BETO • CT-BERT trained with a Spanish dataset • CT-BERT trained with a merged bilingual dataset 1https://pan.webis.de/clef24/pan24-web/oppositional-thinking-analysis.html#data
It was discovered that post-voting, the model’s predictive accuracy on Spanish texts significantly increased.
Table 1 shows the results of various models on validation set.
For subtask 2, based on the Baseline [
Table 3 shows the results of various models on validation set.
For subtask 1, on task1-English, results were submitted utilizing the CT-BERT model trained with the
English dataset. On task1-Spanish, two submissions were made: one from the CT-BERT model trained
with the Spanish dataset and another employing a multi-model voting approach. The oficial evaluation
metric for subtask 1 is the MCC [
On task1-English, the model’s performance surpassed the baseline, ranking fifth among all participants and outperforming the majority of competing methods. This suggests that the CT-BERT model is able to obtain higher-quality text encoding in this task, demonstrating its superior performance. On task1Spanish, the multi-model voting method showed an improvement over the single CT-BERT model prediction method, with an approximate increase of 0.03 in the MCC score. This suggests that the multi-model voting approach can enhance model performance to a certain degree. Additionally, a significant discrepancy was observed between the model’s performance on the test set and the validation set. This suggests that our model is sufering from overfitting, which may be due to the validation set being too small or the selected samples not being representative enough. To resolve this issue, we can ifne-tune the hyperparameters and retrain the model. Moreover, employing k-fold cross-validation can provide a more thorough assessment of the model’s performance, thereby reducing the risk of overfitting.
For subtask 2, on task2-English, predictions were made using the CT-BERT model, while on
task2Spanish, the BETO model was utilized. The oficial evaluation metric for subtask 2 is span-F1 [
On task2-English, the model demonstrated superior performance, achieving an approximate increase of 0.06 in span-F1 score compared to the baseline, securing the third place among all participants and surpassing the majority of the competing methods. On task2-Spanish, the model also exceeded the baseline, meeting the expected outcomes. These results indicate that the CT-BERT and BETO models are suitable for this token-level classification task and exhibit commendable performance.
In the present study, the text classification was realized by fine-tuning the BERT model, accomplishing all subtasks of the Oppositional Thinking Analysis 2024 task. This included distinguishing between texts of critical and conspiratorial nature and detecting elements within oppositional narratives. The results surpassed the baseline performance in the majority of the evaluated metrics. The top-performing model achieved an MCC score of 0.8198 on the task1-English test dataset and a span-F1 score of 0.5886 on task2-English, outperforming the majority of competing models. This suggests that the CT-BERT model is able to obtain higher-quality text encoding in this task, demonstrating its superior performance.
For future work, other outstanding models tailored to the task scenario can be explored, with the aim of leveraging ensemble learning to develop a robust learner and thereby enhance model performance. In addition, Analyzing the correlation between narrative elements and text categories statistically can reveal potential patterns and trends in text classification, providing additional features and information for the model, which may further improve the accuracy of classification.
This work is supported by the Social Science Foundation of Guangdong Province, China (No.GD24CZY02)