The proliferation of misinformation and conspiracy theories needs robust methods to differentiate legitimate critical discourse from harmful conspiratorial narratives. This study investigates discerning critical messages from conspiracy theories within COVID-19 discussions on Telegram. Preserving information integrity on social media impacts vital public discourse on health, politics, and science.
The research employs two distinct approaches: linguistic style classification and contextual knowledge classification. The former leverages a diverse ensemble of Small Language Models (SLMs), Large Language Models (LLMs), and State-Space Models (SSMs), while the latter harnesses the capabilities of the Claude 2.0 Opus model for contextual analysis.
Empirical evaluations demonstrate that the SLM models using Matryoshka embedding and Mamba (SSM) models exhibit superior performance for the English language dataset, achieving a Matthews Correlation Coefficient (MCC) of 0.793. For the Spanish dataset, the Spanish BERT baseline (SLM) attains an MCC of 0.699. Notably, a multilingual model trained on a balanced combination of English and Spanish data outperforms its monolingual counterparts, with the multilingual-e5-large model (LLM) achieving an MCC of 0.768 for English and 0.725 for Spanish. This finding underscores the potential of multilingual models to mitigate the "curse of multilinguality, " where performance often degrades on low-resource languages. However, the suboptimal performance of the Claude 2.0 Opus model, exhibiting a tendency to classify texts as conspiracy-related, highlights inherent biases that require further investigation.
Overall, this study contributes to the development of advanced models that can effectively differentiate critical thinking from conspiratorial narratives in various linguistic contexts. Future research should prioritize identifying and addressing biases in large language models to ensure fair treatment of diverse perspectives, as well as to preserve freedom of expression and ensure fair representation of narratives.
The proliferation of misinformation and conspiracy theories has become a significant challenge in today's digital age, impacting vital aspects of public discourse such as health, politics, and scientific discourse [1]. Conspiracy beliefs can shape human behaviour and decision-making processes, making understanding the cognitive styles and personality traits associated with such beliefs is crucial. Extensive psychological research has identified numerous predictors of conspiracy beliefs, including personality factors like low agreeableness and high openness to experience [1]. Moreover, studies on cognitive styles have revealed a correlation between belief in conspiracy theories and lower analytic thinking coupled with higher intuitive thinking [2].
Understanding these human aspects of conspiracy beliefs is not merely an academic exercise; it has far-reaching implications. This knowledge can inform behavioural interventions to mitigate the spread of misinformation [3]. Furthermore, integrating cognitive and personality factors into natural language processing (NLP) models can enhance their accuracy in distinguishing between critical and conspiratorial narratives, ultimately improving their performance and reliability.
In the realm of automatic content moderation, the challenge of distinguishing between conspiracy theories and critical thinking in NLP models has emerged as a vital area of study. The prevalence of conspiratorial content has escalated the need for robust methodologies that can accurately differentiate between legitimate critical discourse and harmful conspiracy narratives. Maintaining the integrity of information shared across social media platforms and other digital forums is crucial for preserving the credibility of public discourse.
While the focus on this topic remains relatively limited, related studies provide valuable insights into methodologies and applications in adjacent areas. For instance, a significant contribution by [4] presents a framework for detecting conspiracy theories on Twitter using a novel recurrent model called BORJIS, highlighting the efficacy and challenges of NLP techniques in identifying that conspiratorial content is often found within vast amounts of social media data. Similarly, [5] explored fake news detection related to COVID-19 and 5G conspiracy theories using BERT embeddings and Graph Neural Networks, showcasing advanced NLP techniques for distinguishing misinformation from legitimate critical analysis.
Other studies follow a different approach focusing on tracking the spread across social networks, such as the FacTeR-Check semi-automated fact-checking tool that uses semantic similarity and natural language inference (NLI) to monitor the evolution of misinformation or disinformation on online social networks [3]. Additionally, the use of camouflage for content evasion has also been reported, and works have developed multilingual NER NLP models to counter these strategies, like the "pyleetspeak" tool for simulating word camouflage and a NER Transformer model for its detection [6].
Furthermore, the research conducted by [7] explores the potential of NLP techniques in fostering critical thinking skills within educational settings. It offers valuable insights into the systematic instruction and assessment of critical thinking, specifically in comparison to conspiratorial thinking. Finally, [8] emphasize the importance of explicit theorization in developing models that can accurately differentiate between critical and conspiratorial thinking in their paper on gender bias in NLP research.
While significant progress has been made in the field, there is still much to explore in analyzing oppositional thinking using NLP. This research article addresses this issue in both English and Spanish, contributing to the development of more sophisticated NLP systems for real-world scenarios.
The competition, titled "Oppositional Thinking Analysis: Conspiracy vs Critical Narratives" is part of the PAN at CLEF 2024 event [9,10]. Our focus is on the first subtask, which involves analyzing texts from the Telegram platform related to the COVID-19 pandemic. The objective is to perform a binary classification to differentiate between two types of narratives:
• Critical comment: Messages that question major decisions in the public health domain without promoting a conspiracist mentality. These are critical opinions based on information that may not be commonly accepted but do not imply secret plots or malevolent intentions. • Conspiracy comment: Messages that portray the pandemic or public health decisions as results of malevolent conspiracies by secret, influential groups. These messages often encourage distrust based on unverified or poorly explained evidence.
The official evaluation metric for this subtask is the Matthews Correlation Coefficient (MCC). MCC is a measure of the quality of binary classifications, providing a balanced evaluation even when the
Our methodology is based on two main approaches: one that classifies texts according to their linguistic style and content, which we refer to as the Linguistic Style Classification Approach; and another that uses the input text, combined with contextual knowledge and reasoning from large language models (LLMs), referred to as the Contextual Knowledge Classification Approach.
The dataset comprises texts in both English and Spanish, categorized into CRITICAL and CONSPIRACY narratives. The English dataset consists of 2,621 CRITICAL texts and 1,379 CONSPIRACY texts. The Spanish dataset includes 2,538 CRITICAL texts and 1,462 CONSPIRACY texts. Both datasets were divided into training (80%) and validation (20%) sets using a random seed of 42. The preprocessing involved analyzing the prevalence of URLs, emojis, and text length distributions. URLs were removed to standardize the text data. The text length distributions for the English dataset were found to be 743±740 characters for CONSPIRACY and 476±479 characters for CRITICAL. For the Spanish dataset, the distributions were 1112±946 characters for CONSPIRACY and 641±577 characters for CRITICAL. These distributions are illustrated in Figure 1.
We employed a diverse range of models, both monolingual and multilingual. Except for Mamba, all models are based on the Transformer architecture. The significance of Transformer models lies in their attention mechanism, which allows them to efficiently handle dependencies in long sequences and capture intricate patterns within the data. According to Vaswani et al. [11], the self-attention mechanism of Transformers enables them to dynamically weigh the importance of different tokens in a sequence, making them highly effective for various NLP tasks. Mamba, in contrast, is an advanced state-space model (SSM) designed for efficient handling of complex sequences with large datasets, as detailed by Gu and Dao [12].
Below, we list the models used in our research along with brief descriptions: Monolingual
• BERT-base-uncased [13]: A foundational model that effectively applies Transformers at scale, expanding our understanding of linguistic context. We selected the largest variant to ensure a comprehensive analysis and to compare historical model design evolution. • DistilBERT1 : A compact version of BERT by Hugging Face, offering a smaller and faster alternative while maintaining similar performance. Suitable for various NLP tasks. • Nomic: Nomic Embed [14] innovates in embedding techniques to provide dynamic, contextaware representations, surpassing leading models as of February 2024. With a compact size, low memory usage, and advanced training methods, Nomic Embed efficiently processes up to 8192 tokens, making it ideal for analyzing extensive online materials. • DistilRoBERTa [15]: A faster and smaller version of RoBERTa, trained on the same corpus in a self-supervised manner using BERT as a teacher. • twitter-roberta-base-sentiment-latest [16]: A RoBERTa-base model fine-tuned for sentiment analysis using tweets from January 2018 to December 2021, benchmarked with TweetEval. • all-MiniLM-L6-v22 : A Transformer model trained with contrastive loss on 1B sentence pairs to encode sentences and short paragraphs into a dense vector space of 384 dimensions, suitable for tasks like clustering or semantic search. • mxbai-embed-large-v1 [17]: A powerful English embedding model known for its efficient size and high performance. Using Matryoshka Embedding [18], it trains hidden layers to generate high-quality embeddings independently of higher layers, reducing both the number of layers and embedding dimensions. Ranked in the top 25 on the MTEB leaderboard 3 for sentence embedding tasks, it outperforms commercial models like OpenAI's text-embedding-3-large, making it a top choice for our research. • Mamba4 [12]: An advanced state-space model designed for efficient handling of complex sequences with large datasets. It uses a selection mechanism to decide whether to propagate or discard information based on token relevance, providing a viable method for assessing intricate controversial and critical comments on social media. • dccuchile/bert-base-spanish-wwm-uncased [19]: Also known as BETO, this is a BERT model trained on a large Spanish corpus using a vocabulary of about 31k BPE subwords constructed with SentencePiece.
• XLM-RoBERTa: A scaled cross-lingual multilingual sentence encoder version of the RoBERTa model, trained on 2.5TB of data across 100 languages filtered from Common Crawl. • LLAMA 2: A family of pre-trained and fine-tuned large language models (LLMs) by Meta AI, useful for various research and commercial purposes. • multilingual-e5: Developed at Microsoft, this sophisticated embedding model excels in tasks requiring robust text representation, such as information retrieval, semantic textual similarity, and text reranking. Initialized from xlm-roberta-large, it is continually trained on a mixture of multilingual datasets, supporting 100 languages from xlm-roberta with potential performance degradation for low-resource languages.
In this style-based strategy, all these models are employed as the encoder body of the texts to which a layer of 1024 classifier neurons is added.
For hyperparameter tuning, we used Bayesian optimization, which leverages prior evaluations to guide its search process, enhancing model performance. Table 1 lists the explored hyperparameters, including their ranges and sampling distributions. We analyzed the importance and correlation of hyperparameters with the Matthews Correlation Coefficient (MCC). Correlation measures the linear relationship between hyperparameters and MCC, indicating how changes in hyperparameters affect performance.
Additionally, we calculated an importance metric exploiting the feature importance of a random forest model, based on the idea that the more important features appear more often in the trees of the forest. Hyperparameters served as input features, with MCC as the target output. This provided feature importance values, showing each hyperparameter's contribution to predicting performance. These analyses offer insights into how hyperparameters influence model performance.
The experimental tracking can be consulted in Weight and Biases5
For this approach, we utilized the Claude 2.0 Opus model for zero-shot classification. This approach relies on prompt engineering using tempeareture equals to 1 and without fine-tuning the model, leveraging the extensive knowledge of language and context up to early 2023. Additionally, its multilingual capability is well-suited to this task, as approximately 10% of the data used was non-English, according to Anthropic. This model was accessed via Anthropic's public API before May 6. Below is the final prompt used for the Zero-shot classification:
Claude 2.0 Opus Prompt Your role is to analyze text inputs to identify whether they represent critical commentary or conspiracy theories, each with distinct characteristics: Critical Commentary: Definition: Critical messages that question major decisions in the public health domain, but do not promote a conspiracist mentality. It is an opinion, it may not be correct but do not consider that the revendication belongs to a secret or a plot against the population in terms of influential groups. It can be a critic based on information that may not be the common opinion, and it could be wrong, but it is expressing a point of view that another can criticize. Characteristics: Applicability: Applies even when the topic might be susceptible to conspiratorial interpretations.
Definition: Messages that view the pandemic or public health decisions as a result of a malev-olent conspiracy by secret, influential groups. It can be an opinion but the main problem is that it tries to convince you to distrust based on evidences that are not well trusted or explained and leave open the door to be distrustful instead of being critical based on information that may not be the common opinion. Characteristics:
• Suspicion and Paranoia: Thrives on distrust of official narratives and institutions.
• Simplistic Explanations: Oversimplifies complexities by attributing them to the actions of a few. • Resistance to Evidence: Dismisses contrary evidence as part of the cover-up.
You are required to utilize web browser research extensively to verify claims and gather context before making your classification. Be sure to adhere strictly to the output format, especially in reporting URLs used in your research to ensure transparency and accountability. Additional Instruction:
• Always use a web browser to search for information related to the text. Your classification should be informed by credible online sources. Include URLs of these sources in your explanation to validate your findings and reasoning. • Maintain neutrality in your classification process. Do not classify a text as "CONSPIR-ACY" solely because the topic is related to commonly misunderstood or hot-button issues. Instead, use clear evidence from the text and supporting information from web searches to distinguish between critical perspectives and actual conspiracy theories. Include URLs of these sources in your explanation to validate your findings and reasoning.
• Classify the narrative of the text based on the categories above.
• Determine the main topics of the text in 2-3 words.
• Assign a Confidence Score from 0 to 1, indicating the certainty of your classification.
• Your explanation must reflect how the information sourced online influenced your classification and must include URLs for verification.
Output Format: (It is crucial that the output strictly follows this format) { "Prediction": "CATEGORY_NAME", "Confidence": [Confidence Score], "Topic": ["topic1", "topic2"], "Reason": "A concise explanation based on the characteristics with URLs of the sources used." } It is crucial that the output strictly follows this format.
This section presents the results obtained from developing and evaluating various models. As shown in Table 2, the best monolingual model for English are mxbai-embed-large-v1 and Mamba 370m model, achieving an MCC of 0.793, while the best model for Spanish is bert-base-spanish-wwm-uncased, with an MCC of 0.699. As shown in Table 2, the superior performance of the mxbai-embed-large-v1 model underscores its effectiveness in encoding texts into embeddings. This model employs the Matryoshka Embedding technique [18], where each layer is trained to produce high-quality embeddings independently, thus enhancing the model's overall performance. This approach contrasts sharply with the performance of larger models such as Llama 2 [20], which, despite having over 7 billion parameters, underperforms when a classifier head is added. This observation corroborates the notion that model size does not necessarily correlate with task-specific performance. Optimizing the model architecture to improve linguistic encoding, as demonstrated by mxbai-embed-large-v1, proves more beneficial than merely increasing the number of parameters.
Additionally, the performance of the Mamba 370m model, which matches mxbai-embed-large-v1 with an MCC of 0.793, highlights the potential of alternative architectures beyond Transformers. The Mamba model, with its state-space approach and selective propagation mechanism, presents a compelling case for further exploration of non-Transformer architectures in NLP tasks.
The horrible performance of the large language model Claude 2.0 Opus6 in English, which is typically a benchmark model for complex reasoning tasks, warrants further investigation. Despite its state-of-the-art status in reasoning datasets, Claude 2.0 Opus showed a tendency to classify texts as conspiracy-related. This bias was evident even when the model provided reasoning for its classifications, suggesting a predisposition influenced by the sensitive nature of the subject matter. This finding highlights the need for ongoing research into model biases and their impact on classification tasks, particularly for topics with significant socio-cultural implications such as conspiracy theories.
In the Spanish dataset (see Table 3, the bert-base-spanish-wwm-uncased model achieved lower performance compared to its English counterparts, indicating potential limitations in the Spanish training data or model architecture. However, when using the multilingual model multilingual-e5-large, which was trained on Spanish data alone, still not surpassing the monolingual model. This suggests that in this context, monolingual models might be more effective than multilingual ones just using one language data for training. Interestingly, when the multilingual model was trained on both English and Spanish datasets, it achieved an MCC of 0.725, as shown in Table 4. This indicates that multilingual models can leverage larger and more diverse datasets to enhance their understanding of the task. The ability of multilingual models to generalize across languages is particularly evident when they are exposed to substantial amounts of well-represented data, demonstrating their potential to exploit linguistic diversity for improved performance.
Overall, we select the following models for the two trial of the competition:
• RUN 1 -Monolingual approach consists of Mamba 370m model for English and bert-base-spanishwwm-uncased model for Spanish. • RUN 2 -Multilingual approach consists of multilingual model multilingual-e5-large, trained in both languages together.
The results of our hyperparameter tuning highlight the significant influence of learning rate (lr) on model performance, with an importance score of 0.531 and a negative correlation of -0.535 with MCC. This suggests that optimizing the learning rate is crucial for achieving higher performance, as inappropriate values can lead to suboptimal results. Runtime also showed considerable importance (0.196) and a positive correlation (0.410), indicating that longer training times generally improve model performance.
Weight decay and sigmoid focal loss parameters, while less influential than the learning rate, still play vital roles. The weight decay parameter had an importance of 0.163 and a negative correlation of -0.323, suggesting that higher weight decay might adversely affect the model. Sigmoid focal loss [21] parameters (alpha and gamma) demonstrated moderate importance, with alpha showing a positive correlation (0.156) and gamma a negative one (-0.325). The focal loss function, designed to address class imbalance, is given by:
where 𝑝 𝑡 is the model's estimated probability for the true class label, 𝛼 𝑡 is a weighting factor for class imbalance, and 𝛾 is a focusing parameter that adjusts the rate at which easy examples are downweighted. This indicates a complex relationship where these parameters can be fine-tuned to balance the model's sensitivity to class imbalances effectively.
Other parameters, such as epochs, batch size, and accumulation steps, showed lower importance scores. Interestingly, batch size had a positive correlation with MCC (0.287), indicating that larger batch sizes might contribute to better performance. However, the relatively low importance scores for these parameters suggest that while they do influence performance, their impact is less critical compared to the learning rate and regularization parameters.
Table 5 presents the test results, demonstrating that Run 2, which employs a single multilingual model, outperforms the monolingual models. Notably, in both languages, the baseline performance of BERT (MCC 0.7964) is exceeded. Out of 82 teams, only 17 surpass this threshold. Particularly striking is the performance in Spanish, where only 13 teams exceed the MCC threshold of 0.6681, placing us in the top three. These results highlight the potential advantages of multilingual models in achieving robust performance across languages. The improved performance in Run 2 suggests that training a multilingual model on data from both languages mitigates the so-called "curse of multilinguality", where multilingual models often struggle to distribute their knowledge equally across all languages. This phenomenon has been documented in the literature, where multilingual models tend to underperform on low-resource languages due to an imbalance in data distribution and representation [22,23]. Our findings supports that providing a balanced dataset across languages can significantly enhance the fairness and effectiveness of multilingual models, as other works have applied these to counter content evasion on social media platforms [6,24].
Furthermore, this study underscores the importance of addressing biases in large language models (LLMs). The bias observed in the Claude 2.0 Opus model, which showed a tendency to classify texts as conspiracy-related, raises critical questions about the ethical deployment of AI technologies. Such biases can have profound implications for freedom of expression and the equitable treatment of diverse perspectives. Future research should focus on developing techniques to identify and mitigate these biases, ensuring that LLMs operate fairly across different socio-cultural contexts.
In conclusion, our findings support the utility of multilingual models in handling diverse linguistic data, provided that training data is well-distributed across languages. The competition has allow us to conduct a research that demonstrate the potential of such models in achieving high performance and also emphasizes the necessity of continuous efforts to address and mitigate inherent biases in AI systems. Moving forward, it is essential to explore advanced methodologies for bias detection and mitigation, which will be crucial for the ethical and effective application of AI technologies in realworld scenarios.