English. Automatic Conspiracy Theory Identification (ACTI) is a new shared task proposed for the first time at the EVALITA 2023 evaluation campaign. ACTI is based on a new, manually labeled dataset of comments scraped from conspiratorial Telegram channels and consists of two subtasks: (1) identifying conspiratorial content (conspiratorial content classification); and (2) classifying content into specific conspiracy theories (conspiratorial category classification). A total of 15 teams participated in the task with 81 submissions. In this task summary, we discuss the data and task, and outline the bestperforming approaches that are largely based on large language models. We conclude with a brief discussion of the application of large language models to counter the spread of misinformation on online platforms.
fundamental technology: ways to identify conspiratorial
content accurately and at scale across various languages
and cultural contexts [
In this context, we propose the Automatic Conspiracy Theory Identification (ACTI) task. Considering a dataset with over 25 thousand posts in Italian extracted from five Telegram channels, the ACTI consists of two subtasks: (i) a binary classification task where the goal is to determine if a given text piece is conspiratorial or not; and (ii) a multi-class classification task to recognize specific conspiracy theories.
From ancient tales of secret societies, [
Fast-forward to the digital age, the Internet has
emerged as the prominent medium through which
individuals are exposed to conspiracy theories [
The QAnon conspiracy, which gained momentum during
the Trump administration, was pivotal in planning the A: Conspiratorial Content Classification. The first
2021 invasion of the US Capitol [
Mainstream platforms limit the difusion of conspir- (ii) interpret events in a way that supports the narrative
atorial content through interventions that range from of a conspiracy theory.
banning online communities [
Sub-task: A B: Conspiracy Category Classification. The second subtask is determining which conspiracy theory a post belongs to. In particular, we consider four possible conspiracy theories. l e b a L Cospiracy • COVID-19: Text concerning vaccine production, Original 5G, and non-pharmacological interventions as a Augmented tool of control over people. Texts denying the 100 101 102 103 pandemic was a real event or minimizing its im- Count portance. Sub-task: B • QAnon: Texts associated with the QAnon theory.
According to QAnon, a group of Satanic canni- COVID balist sex abusers conspired against former U.S. l QAnon President Donald Trump during his term in ofice. eab This theory extended far over its original scope LFlat-Earth embodying other beliefs that support (among the Russia Original others) the idea that women are enemies (hate 100 101 102 103 against women) and that a powerful elite (led by Count public figures like Pope Francis, Queen Elizabeth, and Hillary Clinton ) is trying to organize a New Figure 1: Distribution of labels for Subtask A and Subtask B.
World Order. • Flat-Earth: Texts associated with the claim that the earth is flat and that influential organizations hide this fact from laypeople. Usually, the flatearth conspiracy theory is supported by pseudo- 3.1. Annotation Process scientific evidence. The data collection process for our study on conspirato• Pro-Russia : Texts associated with conspirato- rial content in online channels involved several steps to rial beliefs promoting Russian interests, e.g., that ensure the accuracy and relevance of the collected data. nazists control Ukraine’s governments and army. One of the main challenges we encountered was the presence of non-conspiratorial content within the channels. 3. Data Collection While some comments discussed conspiratorial topics, others contained valid points or critiques regarding conTo gather the necessary data for the ACTI task, we em- spiratorial perspectives. Additionally, some comments ploy a customized web crawler using the Selenium and were deemed meaningless and needed to be filtered out BeautifulSoup libraries in Python. Our web crawler tar- to maintain the integrity of the dataset. gets specific sources known for hosting conspiratorial To address this, we employed two human annotacontent on the Telegram platform. tors who were responsible for labeling the comments
Specifically, we focus on a selection of Telegram chan- according to three categories: “Not Relevant,” “Nonnels that gained notoriety for promoting far-right ideolo- Conspiratorial,” and “Conspiratorial.” The “Not Relevant” gies and disseminating conspiracy theories. The channels label was assigned to comments that did not contribute to we collect data from include: Qlobal-Change Italia, Basta the discussion, while the “Non-Conspiratorial” label was Dittatura, Studi Scientifici Vaccini, Terra Piatta, and Den- used for comments that did not involve conspiratorial tro La Notizia. For example, the channel “Basta Dittatura” content. The “Conspiratorial” label indicated comments has been actively involved in various events, including that contained or supported conspiratorial discussions. the siege of a trade union headquarters, indicating its For the comments labeled as “Conspiratorial,” we furstrong afiliation with conspiratorial movements. ther categorized them into four subcategories: “QAnon”,
Our data collection process spanned from January 1, “Covid19”, “Russia”, and “Flat-Earth”. These subcategories
2020, to June 30, 2020, during which we capture and allowed us to analyze specific conspiracy theories in
retain comments written in Italian. To ensure suficient greater detail. The definitions of conspiratorial content
text for analysis, we filtered out comments with less than are based on established studies in the field [
Our data collection process yielded 2,301 comments for the first subtask and 1,110 comments for the second subtask. This resulted in a curated dataset that provides a solid foundation for research on conspiratorial content in online discussions.
B because of the distribution of the labels provided by the annotators. In particular
proach to provide a baseline for both subtasks. Specifically, the baselines for subtasks A and B are a Random Forest trained on a bag-of-words representation of the comments. In particular, we trained the random forest with 500 estimators and validated it using a five-fold cross-validation. These baselines achieve 0.63 accuracy for the first and 0.68 for the second subtask, respectively.
participated in the two tasks. Specifically, eight teams submitted for the conspiracy content classification and seven for the conspiracy category classification. In total, we obtain 81 submissions. In Tables 1 and 2, we show the results for both submissions.
Table 2 reports the results of the Conspiracy Category Classification task, which received 41 submissions in total. Once again, the “UPB” team from the University Politehnica of Bucharest achieved the highest F1-score
B, ranking fifth with a score of 0.85 F1-score.
In other EVALITA tasks, ExtremITA showed significant variability in predictive capacity. It ranked first in eight out of twenty-five tasks, but in the remaining subtasks, it performed poorly, ranking between fifth and eleventh. These results confirm LLMs’ high potential and applicability in real-world scenarios. However, the high variability of results shows that LLMs need help improving over models fine-tuned on specific tasks. Future research should focus on refining prompting techniques to improve the predictive capacity of LLMs at the singletask level. 6.2. Augmenting Data with LLMs (0.91). Interestingly, the data augmentation process used for subtask A did increase the model performance. Indeed, the participants submitted the same transformerbased model trained with contrastive learning, excluding the data augmentation block. The second-best performing team from Tor Vergata University (Michael Vitali) achieved an F1-Score of 0.89. They fine-tuned two BERT models, one in Italian and one multilingual, and combined them in an ensemble. Numerous teams performed well in this task, achieving F1-scores beyond 0.80. Only one participant obtained a result slightly inferior to the provided baseline.
The winning team of subtasks A and B (“UPB”) used an approach based on data augmentation via Large Language Models and the training of sentence transformers with contrastive learning. This approach tackles the challenge of the acquisition of conspiratorial data. Indeed, collecting and labeling conspiratorial data requires substantial eforts by domain specialists. This approach tested the possibility of leveraging LLMs to generate synthetic data and use it to train systems for automatically detecting conspiratorial content based. However, it is essential to note that validating the quality of data gener6. Discussion ated by LLM is an open issue within the NLP community. While LLMs can efectively produce synthetic content, A comprehensive analysis of the submitted systems re- assessing its authenticity and alignment with real-world veals that most participants opted for LLMs-based mod- conspiratorial beliefs is crucial. The lower performance els. Within this context, we emphasize two distinct ap- of the model augmented with synthetic data suggests proaches the participants employ: (i) prompting and (ii) that the quality of the generated data drastically impacts data augmentation. Upon thorough analysis, we find that the overall model performance. Therefore, human evaluwhile prompting does result in positive outcomes, the ation is mandatory to evaluate the efectiveness of these predictive capabilities of zero-shot LLMs are still inferior approaches. to systems that have been finetuned for a specific task.
Prompting consists of providing information to a trained reached its end in light of the increasing use of LLMs.
model to predict output labels for a task. It is a task- However, based on the outcomes presented in this report,
agnostic approach, making it versatile and widely appli- it becomes evident that the answer remains negative. The
cable [
The power and flexibility of prompting LLMs are well and tools specifically for the Italian language. This fact is exemplified by team ExtremITA’s approach: adopting a exemplified by transformer-based models’ difering rankLarge Language Model (LLM) to address all EVALITA ings, demonstrating the evaluation campaign’s diversity tasks simultaneously. For the ACTI task, ExtremITA is and significance. However, the performance achieved by prompted with simple questions such as “Does this text LLMs is undoubtedly pushing the limits of some tasks, talk about a conspiracy? Answer yes or no” and “Which especially text classification tasks. In conclusion, while conspiracy theory is discussed in this text: Covid, QAnon, LLMs have shown great potential, EVALITA remains an Flat Earth, or Russia?” This approach ranks second in essential platform for improving language tools for the subtask A of ACTI, achieving a score of 0.86 F1-score. Italian language.