This paper describes the systems that we developed for the textbased and structure-based MediaEval 2020 Fake News detection challenge. These two subtasks and the datasets are detailed in [10] and [12].

Text classification is a common NLP task [6]. Although simple machine learning approaches have shown promising results for fake news detection [8], the recent transformer-based architectures, such as BeRT [2], have set new standards. Several large pre-trained transformer models are now available; they are known to yield stateof-the-art results on many NLP tasks including text classification [16, inter alia]. We rely on one of these pre-trained models to build our systems. In order to improve this standard approach, we have investigated the interest of augmenting the dataset artificially by generating tweets with fine-tuned generative models (one for each class). These approaches and results are detailed in Sec. 2.

Similarly, classification of data represented as a graph, and in particular node classification, is not new but the recent trend is to use deep learning [5]. Yet, for the specific domain of fake news detection, other approaches are possible. In particular, it has been shown that the fake news are propagated differently (and faster) than legit news [15]. The use of node reputation and link-based analysis, as it is done in the detection of spam web pages from the Web graph (such as TrustRank [4], an adaptation of PageRank [1]) is another inspiration for our approaches. Our two approaches are further detailed in Sec. 3.

to the fixed string 'URL'. Twitter usernames are removed if they appear once, others are kept and the @ removed. The intuition is that some often cited users may be associated to a specific class. Hashtags are kept (with # removed), and decomposed when they contain a mix of capital and small letters (eg. #CovidHoax is changed in CovidHoax Covid Hoax).

For this task we wanted to investigate the use of generative models in order to artificially augment and balance the datasets. Indeed, the performance of neural language models based on transformers [14] makes this task realistic. To do so, we use GPT2 (Generative Pre-Trained Transformers), a model built from stacked transformers (precisely, decoders) trained on a large corpus by auto-regression [11]. Three GPT2 models -one for each class -are fine-tuned (from the 355M-parameter pre-trained model) with the tweets of the dev set. The amount of tweets available is very small; we stopped the iterations when perplexity reached 0.5. The way this stopping criterion impacts the results would need further investigations, which were not possible due to the limited time of the challenge. For the generation, we randomly picked up tweets and kept the two first words to serve as bootstrap. The temperature, which controls the creativity of the model, was set at 0.7. Here again, we had no time to investigate the impact of this parameter. Approximately 20,000 tweets were generated for each class. Here are some tweets generated for the class '5G conspiracy':

Crude and unproductive! Turn off the 5G in your area and see if that helps. Covid19 is not funny. I hope that the Wuhan government puts an end to this immediately. "Immigrants are the cause of 5G towers, they're the cause of the coronavirus outbreak, they're the covid-19 victims, the 5G towers are the weapon which will eradicate the world population, 5G lays the microchips for the virus, i read somewhere that the 5G was debuting prior to the introduction of the COVID-19 virus to negate some of the hype around COVID-19

The results of our models are given in Tab. 1. When available, in addition to the official score on the test set, we provide Matthews correlation coefficient (MCC), micro-F1 (accuracy) and macro-F1 on the dev data (80% for training, 20% for validation). Note that due to the cost of the artificial example generation and the small amount of data, the GPT2 models are fine-tuned on all the available dev data; we do not have reliable results for models 3 and 4 (generated tweets added to the training set can be very similar to those in the validation set).

From the results, we see that fine-tuning the representation (model 2 vs. model 1) is beneficial. Unfortunately, the artificially generated tweets (model 3 and 4) do not yield the expected improvement. From the confusion matrices, one can see that the class 'other conspiracy' has the poorest results, with tweets being equally labeled as '5G', 'non' or 'other'.

For the second sub-task, we have proposed two models, based on two different sets of features. They are described in the following subsections, as well as the machine learning algorithms adopted and their results.

This set of features aims at taking into account if one of the users posting or propagating the news has already be seen. Each user is indeed associated with a score for each possible label, computed from the numbers of training samples of each class it was associated with. We also take into account the scores of the neighbors of this user, their own neighbors, and so on... In practice, this is implemented with the PageRank algorithm [1] on the undirected graph with a dumping factor set to 0.8 (optimized by cross-validation). Finally, each sample ends up with one value for each class; these three scores are the features used by the classifier.

Several learning algorithms have been tested (logistic regression, random forests, SVM; as implemented in scikit learn [9]). The optimal settings for their hyper-parameters are grid-searched using 20% of the dev set as validation set. The weight of each sample is adapted according to the inverse of its class proportion ('balanced' strategy). With their optimal settings, the different learning algorithms finally show little differences. For this set of features, the submitted run was produced with a random forest (1,000 trees with a maximal depth set to 5, Out-of-Bag weights used in the prediction).

This set of features is built by considering how the tweet is propagated (without considering the users' reputations). These features can be used even if every involved user has never been seen before and is not connected any known user. The features include (with n 0 the first user tweeting the piece of news): number of nodes in the propagation graph; total number of friends and followers (for all nodes implied), as well as the median, 25% percentile, 75% percentile of followers; number of followers and friends of n 0 ; difference between the number of followers and friends of n 0 ; maximal, minimal, average, median, 25% percentile, 75% percentile of retweet time; times to reach at least 100, 1,000, 10,000 followers and so on up to 200,000 followers. With this set of features, a SVM has been used with the following parameters: standardized features (removed mean and scaled to unit variance), RBF kernel, C=0.9, gamma automatically set with the 'scale' heuristics.

The results of the systems are given in Tab. 1. The cross-validation and official results are consistent; they both show the advantage of the reputation-based approach, especially when considering micro-F1. The difference between cross-validation and official test score may be explained by a lower amount of already seen nodes in the test set, compared to what was generated by cross-validation. A system exploiting all the proposed features (propagation + reputation) was also tested but obtained no statistical difference with the reputation only features.

For both models, the 'other conspiracy' class is again the most error-prone (proportionally), with an equal amount of the its tweets being classified in the three classes. Overall, for both feature sets, many errors are caused by confusion between the 5G and non 5G conspiracy tweets.

For the detection of fake news based on the text, we have adopted a state-of-the-art approach based on RoBerta. The scores obtained show that there exists a large margin for progress, especially when dealing with close classes (5G vs. other conspiracies). The idea of incorporating artificially generated examples did not result in better performance and still needs some work. First, we may find better ways to set the training and generation hyper-parameters. Secondly, we plan to investigate the use of generative model to expand the sample at inference time.

For the detection based on the structure, we have shown that simple approaches like reputation already offered promising results, even on small datasets with many unseen-before nodes. In addition to this type of approach, we want to explore more recent node representation techniques that make it possible to use deep learning, such as node2vec [3] or subsequent variants.