With the amount of information shared on the web increasing on a daily basis, we are prone to face more misinformation online. This is especially true on social media websites, where users have good amount of freedom to share their opinion. During the COVID-19 pandemic, numerous conspiracy theories were shared on Twitter. In this “FakeNews Detection” task, the goal is to detect COVID-19-related conspiracy theories using tweet text and user interaction graph. We tackled this challenge using Transformer-based models (CT-BERT) and node embedding techniques (node2vec) with classification objective models. Our best model obtains a MCC score of 0.719 on the test data.
Transformer-based [
The second task requires methods that leverage graph data. Indeed, the provided data is
composed of a graph of interaction between Twitter users, as well as some information about
the user itself. More information about the data can be found in the task description paper
[
In order to tackle this challenge, we studied text-classification transformer-models for tasks 1 and 3, and node-classification models for tasks 2 and 3. Our approach leverages multiple CTBERT models for text-classification and node2vec in combination with simple classifiers (MLP, RF) for node-classification. We also experimented with GNN without much success and we do not report these results. In all experiments, we split the data into 5 stratified cross-validation sets.
First, we used some basic pre-processing on the text data. We replaced all emojis with their textual meaning using the emoji Python library.2 We also removed the hashtag (‘#’ character) from the tweets. Next, we approached task 1 as a multi-label 3-way classification problem. We trained 5 CT-BERT models, one for each cross-validation fold, using a custom loss function. The last layer of our models has 27 dimensions, three for each of the 9 conspiracy theories (discuss, support, not related). We build 9 diferent Cross-Entropy losses, each measuring the performance of the model at detecting one conspiracy theory. These Cross-Entropy losses are weighted independently, proportionally to the inverse of the frequency of the respective class in the training data. Then, the final loss is the unweighted sum of the 9 diferent losses.
We used a node2vec model to generate node embeddings, and then used standard machinelearning classifiers to perform the node classification.
We first build the graph from the user-interaction data, using the networkx Python
library [
Next, we train some popular machine-learning classifier algorithms available on the
scikitlearn Python library [
We used both graph and textual data for the third task of the challenge. We trained a classifier from the concatenation of both text and graph features, without any form of feature space normalization. Those graph features are the 32-dimension vector from the node embeddings, and the textual features are the 27-dimension vector output of the task 1 model. The training loss is similar to the one described in Section 3.1.
In this section, we first describe each run, and then analyse the main takeaways from the results. We present our results for this challenge in Table 1.
For the first task, we performed an ensembling of the 5 models trained on each fold of the cross validation split (run 001). This ensembling was done with majority voting. In case of a draw between two classes, 1 would have priority over 2 and 3, and 3 over 2. This order follows the proportion of samples we have in the data. Run 002 is the single best model of the 5 models. Run 003 is the same ensembling as run 001 but with the ’cannot determine’ labeling if there is less than 4 models out of 5 in agreement. The ensembling (run 001) obtained the best results.
The second task was more challenging and results are lower compared to task 1. We propose multiple ensembling methods from the models trained on each fold of the cross-validation split. 5List of tested classifiers: KNeighborsClassifier, GaussianProcessClassifier, DecisionTreeClassifier, RandomForestClassifier, AdaBoostClassifier, GaussianNB, QuadraticDiscriminantAnalysis and GradientBoostingClassifier 6The hyper-parameters for the MLP model are: layers of 32-16-8-1, dropout p=0.1, ReLU activation function We trained 5 MLP models and 5 RF models. Run 101, 103, and 105 are ensembling of those models. Run 106 is the same as run 101 with the ’cannot determine’ labeling logic as in run 003. Run 102 is the best single MLP model from run 101. This run obtained the best results, out-performing the ensembling (run 101). We also submitted our run 101 as a sixth run (106) because we did not use any pre-trained model.
For the last task, we first propose the same models as the first task, without using the graph data. Runs 201, 202 and 203 are the same as runs 001, 002 and 003. Results are slightly diferent because the test data is diferent from task 1 and 3. Then, we also propose models using both text and graph data, using the approach described in Section 3.3. Run 204 is an ensembling of the MLP models trained on each fold of the cross validation split and run 205 is the single best MLP model. The best performing model for this third task is the ensembling of CT-BERT models, only on text data (same as run 001).
A first takeaway from these results is that the CT-BERT model is suited for text classification
tasks and obtains very good results, even if slightly lower than in 2021 [
Comparing these results with last year challenge, we did not see a major improvement in the overall score, even though we had access to more data this year. Regarding the results for each conspiracy theory, the best results are obtained for the ‘Harmful Radiation/Influence’ (0.830), ‘New World Order’ (0.830), and ‘Population reduction’ (0.876) conspiracies. The worst result is obtained for the conspiracy theory ‘Antivax’ (0.563). More data does not seem to correlate to better results for each conspiracy as well, since ‘Antivax’ has almost four times more data than ‘Harmful Radiation/influence’ and still perform significantly worse. This is partially due to the weighting of the loss function, which emphasizes the classes with a smaller number of samples.
In this paper, we propose a transformer-based method to detect COVID-19-related conspiracy theories in tweets, composed of an ensembling of CT-BERT models. We also propose a node embedding-based techniques to detect misinformation spreader in the user-interaction graph, using node2vec and an MLP classification head. Our best model obtains a MCC score of 0.719 on the test data.
This work has been partially supported by CHIST-ERA within the CIMPLE project (CHIST-ERA19-XAI-003).