The FakeNews: Corona Virus and Conspiracies Multimedia Analysis task, running for the second time as part of MediaEval 2021, focuses on the classification of tweet texts aiming detection of fastspreading misinformation. Task of this year extends the number of target conspiracy theories and introduces new challenges in terms of analysis complexity of the imbalanced dataset. This paper describes the task, including use case and motivation, challenges, the dataset with ground truth, the required participant runs, and the evaluation metrics.
During the development of the COVID-crisis, a lot of new
COVIDrelated conspiracy theories have arise. Despite eforts of the major
social networks, mass-spread fake facts, irrational theories and
news-like posts are widely presented in the online media sources.
Rumors and other fast-spreading inaccurate, counterfactual, or
intentionally misleading information can quickly permeate public
consciousness and have severe real-world implications. Public
attention to the problem have already allowed content moderation
and partial limitation of freedom of speech in order to prevent
manipulation of COVID-related public opinion. Thus, fake news and
intentional missinformation are still among the top global risks in
the 21st century [
Similar to text-only classification challenges, e.g., [
The task is intended to be of interest to researchers in the areas of online news, social media, multimedia analysis, multimedia information retrieval, natural language processing, and meaning understanding and situational awareness.
Our datasets creation can roughly be divided into four steps. First, We used Twitters’ search API between January 17, 2020 and Jun made publicly available and they are sent to the members of the research team via the direct emails.
After the challenge, the annotated datasets containing only tweet IDs, but not the tweet text itself will be made publicly available. These publicly available datasets will be shufled and supplied by the additional content to prevent linking to the full-text datasets was used during the challenge by the researcher team. An additional tweet content download script will be provided to obtain the tweets from their ids via the corresponding Twitter API using a usersupplied API access keys. 3
The oficially reported metric used for evaluating the multi-class
classification performance is the multi-class generalization of the
Matthews correlation coeficient (MCC, Rk-statistic) [
In case of equal metric values, we use the timestamp of the oficial run submission to rank the teams. For the evaluation, the participants must submit at least one run for at least one subtask defined below. Additionally, the participants optionally can submit four more runs for any of the described subtasks, i.e., participants can submit up to 15 runs in total.
participants receive a dataset consisting of tweet text blocks in English related to COVID-19 and various conspiracy theories. The participants are encouraged to build a multi-class classifier that can lfag whether a tweet promotes/supports or discusses at least one (or many) of the conspiracy theories. In the case if the particular tweet promotes/supports one conspiracy theory and just discusses another, the result of the detection for the particular tweet is expected to be equal to "stronger" class: promote/support in the given sample.
task, the participants receive a dataset consisting of tweet text blocks in English related to COVID-19 and various conspiracy theories. The main goal of this subtask is to build a detector that can detect whether a text in any form mentions or refers to any of the predefined conspiracy topics.
Detection: In this subtask, the participants receive a dataset consisting of tweet text blocks in English related to COVID-19 and various conspiracy theories. The goal of this subtask is to build a complex multi-labelling multi-class detector that for each topic from a list of predefined conspiracy topics can predict whether a tweet promotes/supports or just discusses that particular topic.
All the subtask, in which the team has decided to participate, requires one mandatory and four optional runs to be submitted. The required mandatory run implements a pure NLP classification of tweets based only on tweet text content without using any additional sources of data. Optional runs gradually extend the amount and types of allowed additional information by implementing classification based on tweet text analysis in combination with pretrained models and classification using any automatically scraped data from any external sources. Manual annotation of tweets or any externally scraped data is not allowed in any run.
In the submitted runs participants are allowed to use an
additional Cannot Determine class. This additional class represents
cases, when the output of the classifier is not reliable. This
additional class is important for evaluation of multi-class classifiers. The
efect of using Cannot Determine class is described in the related
literature [
With respect to the subtasks evaluation, the following methodology is used. Text-Based Misinformation Detection subtask is evaluated with Rk-statistic directly. Text-Based Conspiracy
mation and Conspiracies Detection subtasks are evaluated with the two-steps evaluation procedure. First, evaluation of each conspiracy theory individually and independently is performed using Rk-statistic. Then all the computed Rk-statistic values across all the conspiracy theories are averaged and the resulting averaged value is used to compare results of diferent teams. Finally, results in each conspiracy theory group are evaluated independently, but this evaluation is auxiliary and do not afect the final teams ranking. 4
The task itself can be seen as very atypical and challenging due to a fairly limited amount of information available to support the tweet classification process. This reflects the real-world conditions in which online social media analysis systems are deployed. Thus, this task is a practical attempt to make a step towards building a usable multi-modal social network analysis system that is able to combine isolated data source properties with inter-source relations. Due to the importance of the use case, we hope to motivate researchers from diferent research fields to present their approaches, thereby performing research that can help society to fight against malicious manipulations of social networks and threats to society in general. We hope that the FakeNews task can help to raise awareness of the topic, but also provide an interesting and meaningful use case to researchers interested in this application.
This work was funded by the Norwegian Research Council under contracts #272019 and #303404 and has benefited from the Experimental Infrastructure for Exploration of Exascale Computing (eX3), which is financially supported by the Research Council of Norway under contract 270053. We also acknowledge support from Michael Kreil in the collection of Twitter data.