In recent years, we have seen an increase in conspiracy theories on social media. Users create these ideas mainly due to ignorance and distrust of government indices. These types of problems became prominent during the Covid-19 pandemic, representing a serious public health issue. This year, PAN 2024 has organized the task "Oppositional thinking analysis: Conspiracy theories vs critical thinking narratives", to present at CLEF 2024, in which we participated. For this task, we present a solution using a BERT configuration, with a preprocessing layer that has previously given us good results for various types of classification. We also tried combinations of n-grams of words (2-3-4) and characters (2-3-4-5).
With the evolution of the Internet comes the evolution of social media, now one of the primary means of communication and dissemination, where millions of people worldwide can exchange their opinions in real time. With the onset of the Covid-19 pandemic, millions of stories were created by social media users, primarily driven by fear or societal ignorance. Conspiracy theories on platforms such as Facebook, Twitter/X, Telegram and YouTube can be highly persuasive, exploiting users psychological and social vulnerabilities.
The lack of rigorous information verification and the ease with which these theories can be shared
and amplified contribute to their persistence and spread. These narratives exist due to a combination
of psychological, social, and technological factors when used on these platforms. Their impact can
be significant, and addressing them requires an approach that includes education, regulation, and
communication [
There is a study by Hendari et al. [
A notable study using BERT is by Guo et al., which discusses the use of transformers [
Due to the COVID-19 lockdown, Ana López tells us how the trafic on social networks like WhatsApp
and Telegram grew [
In the research of Herasimenka et al., it is shown how certain communities manage to share
misinformation via Telegram, often through links to deceptive sources [
This year, PAN 2024 has organized a series of tasks called "Oppositional Thinking Analysis:
Conspiracy Theories vs. Critical Thinking Narratives" [
For this work, we used a combination of BERT models to perform the binary classification. We have employed these combinations for other tasks and wanted to see how they would perform in this context, as we have primarily used them for tweets analysis as well. As part of our experiments, we utilized certain variations of BERT, such as Bertweet and DistilBERT. 2.1. Preprocessing steps In each research project related to natural language processing that we conduct, we recommend having a data preprocessing layer. Often, this step is essential for obtaining better results that the training model cannot achieve on its own to improve accuracy. Data preprocessing includes several essential tasks such as data cleaning, normalization, tokenization, and noise removal. For this research, we performed the following steps: • Lowercase. We decided to use all lowercase text to standardize the text. • Links. The links in the messages were replaced with the label ’link’. • User Mentions. The messages in this dataset contain the ’@’ sign followed by a space and then the username. Were replaced with the tag ’usermention’. • Hashtags. Can identify them because they start with the ’#’ sign, followed by a space and then the word. Were replaced with the tag ’hashtag’. • Emojis. In previous experiments, emojis have shown to improve classifier learning, so we did not make any changes to them.
• Other symbols. All symbols not registered in the ASCII standard were removed from the dataset.
Due to our previous research experience with social media, we decided also to try a methodology
based on N-grams. Previously, we obtained interesting results despite of its’ simplicity [
Given that the initial results could be improved, we decided to use BERT and some of its variants
for the task. One of these variants is DistilBERT, which is a lighter and more eficient version of BERT.
However, a significant issue with this version is that it was primarily trained on English data, which
may limit its efectiveness in contexts requiring data analysis in other languages [
Similarly, another model we considered was RoBERTa. Although RoBERTa is known for being an
extremely robust and powerful model, it also shares the limitation of being trained primarily on English
data, which could afect its performance in multilingual applications [
To evaluate and compare the performance of these models, we decided to train and test BERT, DistilBERT, and RoBERTa under the same conditions. Specifically, we configured the experiments with a batch size of 16 and trained the models for 9 epochs. This setup allowed us to maintain a balanced use of GPU resources and training time. It is important to note that while RoBERTa is more robust and generally requires more time to fully train due to its complexity and size, this approach allowed us to eficiently manage time and available resources without overwhelming the capabilities of the GPUs used.
Based on the experiments, we decided to use BERT as the model to execute this task. The model performed very close to our expectations, and we believe that with a bit more training data, it could have achieved even better results. DistilBERT ended up in last place. While it is quick in training and execution, it may be suitable for projects that involve limited storage and technological capabilities.
The implementation and analysis of models such as BERT, DistilBERT, and RoBERTa were highlighted, with a focus on evaluating their performance under comparable conditions. The results indicated that BERT was the most efective model for the task at hand, showing results close to expectations. While DistilBERT, although fast, demonstrated limitations in contexts requiring deep analysis and multilingual data. These findings underscore the importance of adapting models to the specific characteristics of each task and linguistic platform, therefore optimizing the accuracy and relevance of the results obtained.
The rise of conspiracy theories on social media poses a significant challenge to society, especially during crises like the COVID-19 pandemic. The spread of misinformation can erode trust in institutions, polarize society, and lead to dangerous behaviors. Engaging in this task not only has the potential to advance academic research in natural language processing but also contributes significantly to combating misinformation in the public sphere. We believe that with a meticulous, evidence-based approach, we can develop efective tools to distinguish between conspiracy theories and critical thinking narratives, thereby helping to mitigate the negative impact of misinformation on society.
We would like to promote the use of BERT and RoBERTa for text classification in Spanish. This could lead to discoveries and improvements that will benefit not only Spanish speakers but also other languages that share similar linguistic features. We are excited about the possibilities this initiative presents and are confident that our contributions will be valuable in the global efort to understand and counter misinformation on social media.