English. In this paper we describe the methodology that we have implemented to solve the subtask A of the Automatic Conspiracy Detection (ACTI) task (EVALITA 2023). We have developed diferent classifiers and then used a majority voting approach to obtain the final prediction. The implemented classifiers can be distinguished into three diferent types: Machine Learning models trained on a document-term matrix (Support Vector Machines, Random Forest and Multinomial Näive Bayes), Neural Network models trained on the text sequences (Long Short Term Memory), and Machine Learning models trained on a set of linguistic features derived from the text. While the single models were prone to overfitting, the classification obtained with the voting approach appears to be more stable and showed an adequate performance on the oficial test set of the contest. Italian. In questo articolo descriviamo la metodologia implementata per risolvere il subtask A del task Automatic Conspiracy Detection (ACTI) task (EVALITA 2023). Abbiamo sviluppato diversi classificatori, per poi ottenere una classificazione finale tramite un meccanismo di voto a maggioranza. I classificatori implementati possono essere distinti in tre gruppi: modelli di tipo Machine Learning allenati su una rappresentazione documento-termine del testo (Support Vector Machines, Random Forest and Multinomial Näive Bayes), modelli di tipo Neural Network allenati sulle sequenze di testo (Long Short Term Memory) e modelli di tipo Machine Learning allenati su un insieme di caratteristiche linguistiche ricavate dal testo. Mentre i singoli modelli sono molto suscettibili all'overfitting, la classificazione ottenuta con il meccanismo di voto sembra essere più stabile ed ha mostrato una performance adeguata sul test set uficiale del contest.
does not only consider the presence of single words, but
With the proliferation of misinformation across multi- they are also able to estimate patterns and relationships
ple platforms and channels, detecting conspiracy theo- among sequences of tokens in a sentence.
ries and fake news has become a crucial task to ensure ML classifiers can also be trained on sets of linguistic
public safety and preserve democratic discourse. There features that can be predominantly present in fake
conhave been eforts to develop Natural Language Process- tent [
Diferent kinds of models are able to capture difer- to increase persuasiveness of electronic communication ent aspects of a text and define suitable predictors for [11]. conspiracy messages. Fake news corpus also tends to display also grammati
Machine Learning (ML) models based on a matrix rep- cal mistakes and inconsistencies, problems with sentence
resentation of terms in documents (document-term ma- structure and an incorrect use of punctuation [
Therefore developing automated models to identify the The ANN model implemented is a Long Short-Term Memspreading of such dangerous ideas is of the utmost im- ory (LSTM) with the structure as described in table 1. portance.
Automated Methods for Identification of Toxic Content. Recent advances of natural language processing [17, 18, 19, 20] provides tools to address questions re- The latter model is trained on a set of syntactical features lated to the identification of troublesome content online. that have been extracted from the texts. The defined feaClearly traditional methods utilizing surface level fea- tures were: the number of tokens and the percentages of tures [21, 22] have opened the path to automatic detec- punctuation, emoji, uppercase words, adjectives, adverbs, tion of hate speech and fake news. Word embeddings and interjections in the text. All the percentages had as [23] and recurrent neural networks [24, 25] significantly denominators the sum of the number of tokens, punctuaincreased the prediction abilities of models to identify tion signs, and emoji. The emoji were identified in the troublesome content. Finally, more recent systems based text with the emoji library. The Italian flag emoji was not on transformers architectures [17, 18] have improved recognized which this library and then identified in a secprediction accuracy among numerous tasks in diferent ond step. The Part Of Speech (POS) tagging used to count ifelds spanning from politics [ 26, 27], conflict prediction the number of adjectives, adverbs and interjections was [28], and, of course, hate speech detection [29, 30, 31]. performed with the Spacy’s pipeline "it_core_news_sm".
We have also tried to identify the percentage of misspelled words with the phunspell library. However, the 3. Description of the System result was not entirely accurate and we decided to discard this feature for this reason.
We have first tested the significance of each feature with a probit logistic regression model and then used the significant ones as predictors in a RF model with maximum depth equal to 13.
In order to evaluate diferent aspects of the text, diferent models have to been tested on a training-validation (70%-30%) split on the labeled training dataset. Since its dimension for task A is 1842 records, models shown a high tendency to overfit. Once the more performing models and their parameters were selected, the final models were trained again on the entire training set in order 4. Results and Discussion to predict labels on the test set (460 records).
In order to limit this problem and build a more sta- The syntactical features used in the RF model are the ble system, we have defined a voting ensemble machine ones that were significant in the logistic model (see table learning model combining the predictions from diferent 2). models able to capture diferent features from the text.
Therefore, the final ensemble classification represents the majority voting of the single predictions from each model.
formed better and have been selected as part of the
voting approach were: Support Vector Machines (SVM) with
The logistic regression showed that a rise in the
number of tokens and the proportion of punctuation and of
uppercase words in the text is slightly increasing the
probability of the text being of conspiracy type. These
associations were in line with what was found in the
literature [
does not seem to give better results than fitting the model on the text alone. SVM and MNB models gave better results with the original tokens, while RF works better with lemmatized tokens. On the train-validation split, RF and SVM obtained higher precisions, whereas MNB higher recall.
The LSTM model was highly prone to overfitting starting from the second epoch. The f-1 scores of the tested models on the final test set are reported in table 3.
The model based on linguistic features should be improved by further detailing of the POS categories (adjectives, adverbs and interjections) in subcategories to better identify key predictors for fake news. A deeper analysis should be performed on emojis too, since they are known to play a role in the spread of misinformation by appealing to emotions [10]. The proportion of emojis in the text seems to be associated with an increased probability of the text not being of conspiracy type probably because emoji can express diferent kinds of feelings and should be divided into subtypes too.
Moreover, explainable AI (XAI) techniques could be further identified to underline specific tokens or other elements in the text influencing the model to classify a text as conspiracy or not.
The corpus provided for the ACTI task would have benefited from a data augmentation technique [ 33] or an integration from other corpora. However, we have not worked on this side, nor tested techniques of bootstrap or boosting. In addition, we have not tried to use more sophisticated models like the ones exploiting attention mechanism, both because of the size of the training set and because the fine-tuning of large language models would have required large GPU resources.
As we explore the potential benefits and limitations of using artificial intelligence (AI) to detect and categorize online conspiracy theories, it is important to consider the broader implications and risks involved in this area of research. While AI holds great promise for helping us better understand the spread of misinformation online and assist human moderators in identifying problematic content, it must not be deployed without considering its limitations in terms of accuracy of the results. Moreover, we need to remember that models trained on textual data can be potentially at risk of privacy leakage by adversarial attacks [34]. Model RF MNB SVM rbf LSTM RF with linguistic features Majority voting