In this paper, we present our participation to the MediaEval-2021 challenge on fake news detection about coronavirus related Tweets. It consists in three subtasks that can be seen as multi-labels classiifcation problems we solved with transformer-based models. We show that each task can be solved independantly with mutiple monotasks models or jointly with an unique multitasks model. Moreover, we propose a prompt-based model that has been finetuned to generate classifications from a pre-trained model based on DistilGPT-2. Our experimental results show the multitask model to be the best to solve the three tasks.
With the worldwide spread in the last few years of the Sars-Cov-2 virus, also known as Coronavirus, fear and concern has grown. While traditional media often relies on scientifically-vetted sources to bring information to concerned readers or viewers, social media is not subjected to this obligation to fact check. Therefore, a plethora of messages of various degrees of truthfulness has emerged across social media platforms, such as Tweeter. This network has been used as a soapbox for a multitude extreme political theories revolving around the Coronavirus epidemic, as well as for the debunking of said conspiracies, making it harder to untangle the conspiracy theories from the real facts.
This paper describes the systems 1 that we developed for
MediaEval 2021 Fake News detection challenge. The main objective of
the present task is to classify tweets according to whether they
are relaying conspiracy theories, and what specific conspiracy is
evoked. Several sub-tasks contribute to refining the task[
For each of the three aforementioned tasks, we propose a
classiifcation solution relying on the fine-tuning of transformer-based
models e.g., [
We explain hereunder the diferent models we devised to address to detect conspiracy theories in tweets. We first describe the separated task learning framework in Sec. 2.1, than we introduce the Promptbased model in Sec. 2.3. Finally, Sec. 2.1 is dedicated to the multi-task setup where all the three diferent task are performed at once. 2.1
The tasks can be solved with multi-labels model to classify a tweet as a vector of independent probabilities for each label thanks to a Sigmoid activation function. The labels of the task 2 are already well encoded as a binary matrix, whereas for the tasks 1 and 3, the original categorical labels have been converted to binary targets with one-hot-encoding.
For each task, a separate instance of several BERT-based models
are fine-tuned. We tested bert-tiny [
The multitask model uses the same backbone as the first approach with separated tasks. Each set of labels (for tasks 1, 2 and 3) are concatenated into a single set of labels. The idea behind this multitask approach is to learn one general model that can be used to perform the diferent tasks taking advantage of the relation between the tasks e.g., the existence of a conspiracy for task-1 or the existence of a particular conspiracy theory for task-3 (fine-grained version of task-1 and task-2). Thus by properly performing task-3, we expect the multitask model to be better in task-1 and task-2 as they are more general than task-3. 2https://huggingface.co/ans/vaccinating-covid-tweets C.-B. El Vaig, T. Girault, C. Mallart, D.-H. Nguyen 2.3
According to the prompt-based learning approach, samples are made into templates, containing the text of the tweet, a label for the tweet, and a binary classification of whether the tweet is related to its label, as in the following : Tweet : Media succeeded in creating this Covid 19 hoax... Label : Promotes/Supports Conspiracy Classification : true
As shown in the previous template, we formulated this problem as a binary classification task. For each tweet, one template is created for each possible label.
The language model is then trained to output the final word, chosen in a list that consists of the words ["false", "unlinked", "unrelated", "true", "related", "linked"]. This word should be consistent with the previous prompt, which is the text and the label, and therefore correctly learn the type of tweets associated with each label. At inference time, three templates are again created for one tweet : the text of the tweet, followed by each of the possible three labels. The chosen label is the one where the model has the highest probability to output "true", "related" or "linked" as the following word.
This prompt-based model was implemented with the use of the
OpenPrompt library [
Table 1 shows the results on task-1 of the diferent finetuned pretrained models, as well as the prompt-based approach. The best scores (MCC, macro-f1 and micro-f1) are obtained with the covidtweet model which has been clearly optimized on a corpus adapted to our task. The toxic-bert reached comparable results but it appears that pretraining on toxic language is not really transferable to the conspiracy language. Surprisingly, smaller models such as bertsmall and bert-tiny were able to achieve quite competitive results.
Prompt-based learning with DistilGPT-2 does not outperform traditional fine-tuning on the task 1. However, we would expect that this approach would benefits from expanding to multitask where the labels share semantic properties and the number of example per label is low.
As expected, the multitask approach outperforms the individual models on all three tasks, as displayed in Table 2. The detailed results for each label are given in table 3.
In this work, we experimented transformer-based models to detect
conspiracy theories in tweets. The three tasks have been solved with
multilabels classifiers relying on the pretrained models. We showed
that it is better to train the models jointly on multiple tasks rather
than independently. Meanwhile, the oficial MCC scores, while
good, still show that there is large space for progress, especially
for the task-1 which has only three labels. The other tasks are
more challenging due to diversity of labels and the small size of the
dataset. The idea of using the prompt-based model shows promising
results for task-1, but due to lack of time and resources we would
rather focus on the other experiments. As shown in [