Conspiracy theories and critical thinking are two forms of oppositional thinking which are common especially on contentious issues and analyzing oppositional thinking entails scrutinizing narratives that question mainstream perspectives. Further, understanding the impact of different types of oppositional thinking on public opinion and behavior is crucial [1]. While critical thinking fosters constructive and democratic debate characterized by reasoned questioning without unfounded explanations, conspiratorial thinking can lead to misinformation and social conflict attributing the significant events to hidden malevolent forces [2]. In social and political discourse, conspiracy theories can have detrimental effects on an individual as well as on organisations or the entire society. These theories suggest that major social (encompass gatherings and activities involving people) and political events (activities related to government and leadership) with careful planning by powerful and malevolent entities can spread false information and stir social unrest. Conspiracy theories have been associated with violence, war, terrorism, prejudice, poor health choices, and denial of climate change [3]. In contrast, critical thinking involves analyzing and questioning decisions, particularly in areas like public health, without attributing events to hidden conspiracies.
Mislabeling critical discourse as conspiratorial can suppress healthy debate and alienate individuals who are merely questioning decisions and mislabeling critical conversation as conspiracy can stifle constructive disagreement and alienate people who are only challenging judgements [3,4]. This leads to a climate of mistrust and inhibits the exchange of open and sincere ideas. On the other hand, failing to identify and address conspiratorial narratives allows the spread of misinformation, leading to societal division and mistrust. Thus, accurate differentiation between the two forms of oppositional thinking by content moderation systems is essential to avoid marginalizing legitimate criticism and preventing individuals from being drawn into conspiracy communities by mistaking valid critique for conspiracy theories [5]. Additionally, distinguishing between these forms of oppositional thinking is essential for public discourse and social harmony.
Distinguishing between the oppositional forms of thinking is challenging due to the nuanced and overlapping content, the context-dependent nature of oppositional statements, and oppositional attitudes [6]. Further, differentiating between well-founded critical and unfounded conspiracy theories require advanced and context-aware NLP techniques and developing such techniques is crucial for improving the accuracy and fairness of content moderation systems. To address the challenges of distinguishing between the oppositional forms of thinking, "Oppositional thinking analysis: Conspiracy theories vs Critical thinking narratives" shared task organized at PAN 2024 [7], invites the research community to develop models to distinguish between conspiracy and critical thinking texts in English and Spanish languages. The shared task has two subtasks in English and Spanish languages and we -team MUCS participated in only Subtask-1: 'Distinguishing between critical and conspiracy texts'. This subtask having two categories -'CONSPIRACY' and 'CRITICAL', is modeled as a binary classification problem. In this paper, we describe various machine learning models trained with TF-IDF of char n-grams in the range (1,5) and transfer learning techniques using several BERT variants fine-tuned with the given Engish and Spanish datasets, to classify the given unlabeled English and Spanish text into one of the two categories -'CONSPIRACY' or 'CRITICAL'. The sample text from the given datasets for English and Spanish are shown in the Tables 1 and 2 respectively.
The rest of the paper is organized as follows: Section 2 describes the recent literature on the two forms of oppositional thinking and Section 3 focuses on the description of the proposed models followed by the experiments and results in Section 4. The paper concludes with future works in Section 5.
Conspiracy theories involve elaborate, unverified claims driven by cognitive biases and emotional needs, often rejecting official explanations without sufficient evidence. In contrast, critical thinking is characterized by objective analysis, logical reasoning, and the evaluation of evidence from multiple sources. Research highlights that while conspiracy beliefs are linked to cognitive biases and feelings of powerlessness, critical thinking fosters informed decision-making and intellectual humility [3].
To explore different strategies for the conspiracy textual content classification in social media, Moosleitner and Murauer [8] Giachanou et al. [2] performed a comparative analysis of various profiles and psychological and linguistic characteristics in social media texts of users who share posts about conspiracy theories. The authors then compared the effectiveness of these characteristics for predicting whether a user is a conspiracy propagator or not by proposing ConspiDetector, a model that is based on a Convolutional Neural Network (CNN) which combines word embeddings with psycho-linguistic characteristics extracted from the tweets of users to detect conspiracy propagators. Recordare et al. [11] implemented various machine learning classifiers (Logistic Regression (LR), k-Nearest Neighbours (kNN), Naive Bayes (NB), SVM, Decision Trees (DT), Random Forest (RF), Gradient Boosting: XGBoost and LightGBM, Quadratic Discriminant Analysis, MLP, Ridge Classifier, and Linear Discriminant Analysis, trained with Bidirectional Auto-Regressive Transformers (bart)-large-Multi-Genre natural language inference features for identifying users who propagate conspiracy theories based on a rich set of 871 features in English language. Among all the proposed models, LightGBM classifier outperformed other models with a macro F1 score of 0.87. To identify whether an article belongs to conspiracy theory or not in English language, Ghasemizade and Onaolapo [12] proposed machine learning classifiers (RF, SVM, k-NN, NB) trained with TF-IDF of word unigrams and deep learning model trained with padded and embedded text sequences. Using their respective tokenizers, tokenized and padded text sequences were taken as inputs to train the transformer models -BERT and RoBERTa. Their proposed RoBERTa model outperformed other models with a macro F1 score of 87%.
The above literature highlights extensive research efforts aimed at detecting conspiracy theories utilizing a range of machine learning, deep learning, and transfer learning models. These studies offer valuable insights into the detection of conspiracy theories. However, they do not specifically address the distinction between conspiracy theories and critical thinking within the framework of oppositional thinking. This gap suggests a need for further research to effectively distinguish between these concepts, encouraging the creation of new models for this specific type of application.
We have explored machine learning and transfer learning models for distinguishing between critical and conspiracy texts in English and Spanish and the steps involved in the construction of these models are explained in the following subsections.
The framework of machine learning model is visualized in Figure 1 and the steps included in building these classifiers are explained below:
Pre-processing is the preliminary step in building learning models and it involves cleaning and transforming the raw text data to a suitable format required for subsequent processing. Usually, text data contains noise in the form of: user mentions, hashtags, punctuation, digits, and hyperlinks, and eliminating this irrelevant information makes the data less complex and improves the performance of the classifier. Hence, in this work, this irrelevant information and stopwords are removed during preprocessing. Further, English and Spanish stopwords available at NLTK library1 are used as references to remove English and Spanish stopwords respectively from the given dataset.
The role of feature extraction is to extract relevant features from the given data to train the learning models. TF-IDF of char n-grams is used to represent English and Spanish text. Char n-grams are sequences of n consecutive characters in a word and char n-grams in the range (1,5) are obtained from the text and converted to TF-IDF vectors using TfidfVectorizer2 .
The performance of the learning model relies on the features and the classifier used to carry out classification. This work utilizes machine learning classifiers (MNB, LR, RF, and ensemble of LSVC, LR, and RF with majority voting), to distinguish between conspiracy and critical texts in English and Spanish language. A brief description of the machine learning classifiers is given below:
• MNB -is a probability-based classifier suitable for text classification with discrete characteristics like word frequency counts [13]. • LinearSVC -used from the Scikit-learn library3 attempts to maximize the distance between classified samples by finding a hyperplane. • LR -is used to predict the probability of certain classes based on dependent variables and is suitable for binary classification task. Further, regularisation approaches in LR classifiers are useful for reducing overfitting in high dimensional space [14]. • RF -is one of the supervised learning algorithms which is flexible and can be adapted easily to different situations but it is necessary to build a minimum number of trees in order to classify the data [15]. • Ensemble learning -is a strategy for building a new classifier from several heterogeneous base classifiers taking benefit of the strength of one classifier to overcome the weakness of another classifier to get better performance for the classification task [16]. In this work, three machine learning classifiers (LSVC, LR, and RF) are ensembled with hard voting to distinguish between critical and conspiracy texts.
The hyperparameters and their values used in the machine learning models are shown in Table 3.
The technique of transfer learning within the broader field of machine learning utilizes knowledge gained from one task to improve the performance of another related task. This is realized with the help of pretrained transformer models which are trained on large unlabeled data and are widely accessible and applicable to various tasks. The pretrained transformer models are fine-tuned with the given dataset to fit the models to a particular task or domain. The framework of the proposed transfer learning model is shown in Figure 2.
The text is pre-processed to clean and transform the raw text into a consistent format by converting numeric information to corresponding words, and removing URLs, user mentions, hash tags, and special characters. Preprocessing is applied to the sentences of the given text to retain the sentence structure in the text and the preprocessed text is used to fine-tune the transformer models. A brief description of the transformer models used in this study to fine-tune are given below: • BERT_base 4 is a conceptually simple and empirically powerful pretrained language model using a Masked Language Modeling (MLM) objective trained on Toronto Book Corpus and Wikipedia and exclusively used for tasks involving English texts. • English_BERT 5 -is a bilingual Legal BERT model trained with 2,000 Dutch and 6,000 English legal documents amounting to 12 GB legal text from various areas belonging to legal domain such as legislation and court cases. This domain specific BERT has resulted in improved performance compared to using standard BERT models for legal tasks. • CT_BERT_v2 6 -is a BERT-large-uncased model pretrained on a corpus of messages from Twitter about COVID-19. This model is identical to covid-twitter-bert but trained on more data (40.7M sentences and 633M tokens) resulting in higher performances for many downstream applications. • EN_RoBERTa 7 -is a large multi-lingual language model trained on 12.5TB of filtered Common-Crawl data. Based on Facebook's RoBERTa model, this model is fine-tuned with the conll2003 dataset in English. • ES_BERT 8 -BETO: Spanish BERT is trained on the Spanish edition of Wikipedia, the OPUS Project, and Spanish books and news articles, and is exclusively used for tasks involving Spanish texts. • Distil_SpanBERT 9 -is a distilled version of SpanishBERT, trained on Spanish text sources, and is also exclusively used for tasks involving Spanish texts, but is optimized for efficiency and speed. • Spanish_BERT 10 -is a sentence-transformers model which maps sentences and paragraphs to a 768 dimensional dense vector space and can be used for tasks like clustering or semantic search. • ES_RoBERTa 11 -a variant of RoBERTa is a BERT-based model, specifically tailored for the Spanish language. It is trained on large Spanish text corpora to understand and generate contextually relevant representations of words and sentences.
We employed the above mentioned BERT variants from the Hugging Face library. The hyperparameter and their values used in the above transfer learning models are shown in Table 4. These BERT variants are fine-tuned with the pre-processed Train set and is used to train transformer classifier (ClassificationModel) to distinguish between conspiracy and critical texts in English and Spanish language.
The datasets provided by the organizers of the shared task consisted of only Train set and are highly imbalanced. Statistics of the datasets are as follows:
• English dataset: 2,621 samples belong to 'CRITICAL' class and 1,379 samples belong to 'CON-SPIRACY' class. • Spanish dataset: 2,538 samples belong to 'CRITICAL' class and 1,462 samples belong to 'CON-SPIRACY' class.
In this paper, we -team MUCS, describe the models submitted to Subtask-1: 'Distinguishing between critical and conspiracy texts' of the shared task "Oppositional thinking analysis: Conspiracy theories vs critical thinking narratives" at 'PAN 2024', to distinguishing between critical and conspiracy texts in English and Spanish. Experiments are carried out with TF-IDF of char n-grams in the range (1,5) to train several machine learning classifiers and several BERT variants are fine-tuned with the English and Spanish Train sets in transfer learning models, to distinguish the given unlabeled English and Spanish text as 'CONSPIRACY' or 'CRITICAL'. As the shared task participants were allowed to submit the predictions of only two models on the Test set, we fine-tuned English_BERT and EN_RoBERTa with the complete English Train set and Spanish_BERT and ES_RoBERTa with the complete Spanish Train set and the predictions of these models on English and Spanish Test sets were submitted to the organizers for evaluation. Among these two models, English_BERT and Spanish_BERT obtained MCC scores of 0.7162 and 0.6293 for English and Spanish texts securing 61 st and 36 th position respectively. As the given datasets are imbalanced, suitable text augmentation techniques followed by efficient text representation methods and context-aware models to distinguish between the two forms of oppositional thinking will be explored further.
As the datasets consists of only Train sets, 33% of the Train sets at random are considered as Validation sets to evaluate the performances of the proposed models for both the languages and the remaining as the Train sets. Experiments were carried out by training various machine learning models using TF-IDF of char n-grams in the range (1,5) and by fine-tuning various BERT variants mentioned above, to distinguish between conspiracy and critical texts in English and Spanish. The performances of the proposed models were evaluated on the Validation set based on macro F1 score and the performances are shown in Tables 5 and 6 for English and Spanish datasets respectively.
The results shown in Tables 5 and 6 illustrate that transfer learning models have performed better than machine learning models. As the shared task participants were allowed to submit the predictions of only two models on the Test sets, we fine-tuned English_BERT and EN_RoBERTa with the complete English Train set and Spanish _BERT and ES_RoBERTa with the complete Spanish Train set and the predictions of these models on English and Spanish Test sets submitted to the organizers were evaluated based on MCC scores. MCC is a metric used to evaluate the quality of binary classifications especially with imbalanced datasets. The MCC value ranges from -1 to +1, where +1 indicates perfect prediction, 0 indicates no better than random prediction, and -1 indicates total disagreement between prediction and observation. MCC provides a balanced and comprehensive measure of model's performance, considering all types of classification errors. MCC scores are calculated using the following formula:
where:
• 𝑇 𝑃 (True Positives) -number of correct positive predictions,
• 𝑇 𝑁 (True Negatives) -number of correct negative predictions,
• 𝐹 𝑃 (False Positives) -number of incorrect positive predictions, • 𝐹 𝑁 (False Negatives) -number of incorrect negative predictions.
Among the two transfer learning models submitted to the shared task, English_BERT and Span-ish_BERT models obtained MCC scores of 0.7162 and 0.6293 for English and Spanish texts securing 61 st and 36 th position respectively. The performances of these models is shown in the Table 7. The low performances of fine-tuned English_BERT and Spanish_BERT models could be attributed to the following reasons:
• The given datasets are highly imbalanced with approximately 2/3 of the total samples belonging to 'CRITICAL' class and 1/3 belonging to 'CONSPIRACY' class in both the languages. The highly imbalanced data will significantly impact the models' performance with a bias towards majority class. • English_BERT designed for both Dutch and English may potentially reduce its effectiveness for purely English tasks. Further, domain-specific models like English_BERT might not generalize well outside their specialized contexts. • Spanish_BERT model might be more suited for sentence similarity tasks rather than classification. Further, differences in pre-training data, fine-tuning processes, and the values of hyperparameters used in the models could also contribute to the disparity in the performances of the proposed models.