The uncontrolled growth of fake news creation and dissemination we observed in recent years causes continuous threats to democracy, justice, and public trust. This problem has signi cantly driven the e ort of both academia and industries for developing more accurate fake news detection strategies. Early detection of fake news is crucial, however the availability of information about news propagation is limited. Moreover, it has been shown that people tend to believe more fake news due to their features [13]. In this paper, we present our framework for fake news detection and we discuss in detail a solution based on deep learning methodologies we implemented by leveraging Google Bert features. Our experiments conducted on two well-known and widely used real-world datasets suggest that our method can outperform the stateof-the-art approaches and allows fake news accurate detection, even in the case of limited content information.
Social media are nowadays the main medium for large-scale information sharing
and communication and they can be considered the main drivers of the Big Data
revolution we observed in recent years[
Fake news detection dates back long time ago[
Copyright c 2020 for this paper by its authors. Use permitted under Creative
Commons License Attribution 4.0 International (CC BY 4.0). This volume is published
and copyrighted by its editors. SEBD 2020, June 21-24, 2020, Villasimius, Italy.
Indeed, the term fake news has grown in popularity in recent years, especially
after the 2016 United States elections but there is still no standard de nition
of fake news [
The content of fake news exhibits heterogeneous topics, styles and media
platforms, it aims to mystify truth by diverse linguistic styles while insulting true
news. Fake news are generally related to newly emerging, time-critical events,
which may not have been properly veri ed by existing knowledge bases due to
the lack of con rmed evidence or claims. Thus, fake news detection on social
media poses peculiar challenges due to the inherent nature of social networks
that requires both the analysis of their content [
Our approach in a nutshell. Fake news detection problem can be formalized as a classi cation task thus requiring features extraction and model construction. The detection phase is a crucial task as it is devoted to guarantee users to receive authentic information. We will focus on nding clues from news contents. Our goal is to improve the existing approaches de ned so far when fake news is intentionally written to mislead users by mimicking true news. More in detail, traditional approaches are based on veri cation by human editors and expert journalists but do not scale to the volume of news content that is generated in online social networks. As a matter of fact, the huge amount of data to be analyzed calls for the development of new computational techniques. It is worth noticing that, such computational techniques, even if the news is detected as fake, require some sort of expert veri cation before being blocked. In our framework, we perform an accurate pre-processing of news data and then we apply three di erent approaches. The rst approach is based on classical classi cation approaches. We also implemented a deep learning approach that leverages neural network features for fake news detection. Finally, for the sake of completeness we implemented some multimedia approaches in order to take into account misleading images. Due to space limitation, we discuss in this paper the deep learning approach. 2
Our framework is based on news ow processing and data management in a preprocessing block which performs ltering and aggregation operation over the news content. Moreover, ltered data are processed by two independent blocks: the rst one performs natural language processing over data while the second one performs a multimedia analysis. The overall process we execute for fake news detection is depicted in Figure 1. In the following, we describe each module in more detail:
Data Ingestion Module. This module take care of data collection tasks. Data can be highly heterogeneous: social network data, multimedia data and news data. We collect the news text and eventual related contents and images.
Pre-processing Module. This component is devoted to the acquisition of the incoming data ow. It performs ltering, data aggregation, data cleaning and enrichment operations.
NLP Processing Module. It performs the crucial task of generating a binary classi cation of the news articles, i.e., whether they are fake or reliable news. It is split in two submodules. The Machine Learning module performs classi cation using an ad-hoc implemented Logistic Regression algorithm after an extensive process of feature extraction and selection TF-IDF based in order to reduce the number of extracted features. The Deep Learning module classi es data using Google Bert algorithm after a tuning phase on the vocabulary. It also performs a binary transformation and eventual text padding in order to better analyze the input data.
Multimedia Processing Module. This module is tailored for Fake Image Classi cation through Deep Learning algorithms, using ELA (Error Level Analysis) and CNN.
Due to space limitation, we discuss in the following only the details of the deep learning module and the obtained results. 2.1
The Deep Learning Module computes a binary classi cation on a text datasets of news that will be labelled as 0 if a news is marked as Real, and as 1 if it is marked as Fake. The Deep Learning Module classi es news content using a new language model called B.E.R.T. (Bidirectional Encoder Representations from Transformers) developed and released by Google. Prior to describing the algorithm features in detail, we brie y describe the auxiliary tools being used, while in Section 3 we describe the experimental evaluation that lead to our choice on BERT.
Colaboratory. Colab is intended for machine learning education and research, it requires no setup and runs entirely on the cloud. By using Colab it's possible to write and execute code, save and share analytics and it provides https://research.google.com/colaboratory access to expensive and powerful computing resources for free by a web interface. More in detail, Colab's hardware is powered by: Intel(R) Xeon(R) CPU @ 2.00GHz, nVidia T4 16 GB GDDR6 @ 300 GB/sec, 15GB RAM and 350GB storage.
Tensor Flow. It is devoted to train and run neural networks for image recognition, word embeddings, recurrent neural networks, and natural language processing. It is a cross-platform tool and runs on CPUs, GPUs, even on mobile and embedded platforms. TensorFlow uses data ow graphs to represent the computation ow, i.e., these structures describe the data ow through the processing nodes. Each node in the graph represents a mathematical operation, and each connection between nodes is a multidimensional data array called tensor. The TensorFlow Distributed Execution Engine abstracts from the supported devices and provides a high performance-core implemented in C++ for the TensorFlow platform. On top there are Python and C++ frontends. The Layers API provides a simple interface for most of the layers used in deep learning models. Finally, higher-level APIs, including Keras, makes training and evaluating distributed models easier.
Keras. It is a high-level neural network API, implemented in Python and capable of running on top of TensorFlow. It allows for easy and fast prototyping through: 1) User Friendliness as it o ers consistent and simple APIs that minimizes the number of user actions required for common use cases; 2) Modularity as neural layers, cost functions, optimizers, initialization schemes, activation functions and regularization schemes are all standalone modules that can be combined to create new models; 3) Extensibility as new modules are simple to add as new classes and functions.
Google BERT. This tool has been developed in order to allow an easier implementation of two crucial tasks for Natural Language Processing (NLP): Transfer Learning through unsupervised pre-training and Transformer architecture. The idea behind Transfer Learning is to train a model in a given domain on a large text corpus, and then leverage the gathered knowledge to improve the model's performance in a di erent domain. In this respect, BERT has been pre-trained on Wikipedia and BooksCorpus. On the opposite side, the Transformer architecture processes all elements simultaneously by linking individual elements through a process known as attention. This mechanism allows a deep parallelization and guarantee higher accuracy across a wide range of tasks. BERT outperforms previous proposed approaches as it is the rst unsupervised, fully bidirectional system for NLP pre-training. Pre-trained representations can be either context-free or context based dependig on user needs. Due to space limitations we do not describe in detail the BERT's architecture and the encoder mechanism. 3
In this section we will describe the fake news detection process for the deep learing module and the datasets we used as a benchmark for our algorithms. 3.1
Liar Dataset. This dataset includes 12.8K human labelled short statements
from fact-checking website Politifact.com. Each statement is evaluated by a
Politifact.com editor for its truthfulness. The dataset has six ne-grained labels:
pants- re, false, barely-true, half-true, mostly-true, and true. The distribution
of labels is relatively well- balanced. [
The processed dataset has been uploaded in Google Drive and, then, loaded in Colab's Jupyter as a Pandas Dataframe. It has been added a new column with the number of words for each row article. Using the command df:describe() on this column it is possible to print the following statistical information: count 15389.000000, mean 17.962311, std 8.569879, min 1.000000, 25% 12.000000, 50% 17.000000, 75% 22.000000, max 66.000000. These statistics show that there are articles with only one word in the dataset, so it has been decided to remove all rows with less than 10 words as they are considered poorly informative. The resulting dataset contains 1657 less rows than the original one. The updated statistics are reported in what follows: count 13732.000000, mean 19.228663, std 8.192268, min 10.000000, 25% 14.000000, 50% 18.000000, 75% 23.000000, max 66.000000. Finally, the average number of words per article is 19.
FakeNewsNet. This dataset has been built by gathering information from
two fact-checking websites to obtain news contents for fake news and real news
such as PolitiFact and GossipCop. In PolitiFact, journalists and domain experts
review the political news and provide fact-checking evaluation results to claim
news articles as fake or real. Instead, in GossipCop, entertainment stories, from
various media outlets, are evaluated by a rating score on the scale of 0 to 10 as
the degree from fake to real. The dataset contains about 900 political news and
20k gossip news and has only two labels: true and false. [
This dataset is publicly available by the functions provided by the FakeNewsNet team and the Twitter API. As mentioned above, FakeNewsNet can be split in two subsets: GossipCop and Politifact.com. We decided to analyse only political news as they produce worse consequences in real world than gossip ones. The dataset is well balanced and contains 434 real news and 367 fake news. Most of the news regards the US as it has already been noticed in LIAR. Fake news topics concern Obama, police, Clinton and Trump while real news topics refer to Trump, Republicans and Obama. Such as the LIAR dataset, it has been added a new column and used the command df:describe() to print out the following statistical information: count 801, mean 1459.217228, std 3141.157565, min 3, 25% 114, 50% 351, 75% 893, max 17377. The average number of words per articles in Politifact dataset is 1459, which is far longer than the average sentence length in Liar Dataset that is 19 words per articles. Such a statistics suggested us to compare the model performances on datasets with such di erent features. 3.2
The above mentioned datasets are available in CSV format and are composed of two columns: text and label. The news text need to be pre-processed for our analysis. In this respect, an ad-hoc Python function has been developed for unnecessary IP and URL addresses removal, HTML tags checking and words spell-check. Due to neural features, we decide to maintain some stop words in order to allow a proper context analysis. Thus, to ameliorate the noise problem, we created a custom list of stop words. We leverage Keras Tokenizer for preparing text documents for subsequent deep learning steps. More in detail, we create a vocabulary index based on word frequency, e.g., given the sentence The cat sat on the mat we create the following dictionary word index[the] = 1, word index[cat] = 2 so every word gets a unique integer value; the 0 value is reserved for padding. Lower integer means more frequent word. After this encoding step, we obtain for each text a sequence of integers. As BERT needs a more elaborated input than other neural networks wed need to produce a tsv le, with four columns, and no header. The columns to be added to dataset are: 1) guid, i.e., a row ID; 2)label, i.e., the label for the row (it should be an int); 3) alpha, a dummy column containing the same letter for all rows, it is not used for classi cation but it is needed for proper running of the algorithm and 4) text, i.e., the news content. The data needs to be converted in InputFeature object to be compatible with Transformer Architecture. The conversion process includes tokenization and converting all sentences to a given sequence length (truncating longer sequences, and padding shorter sequences). Tokenization is performed using WordPiece tokenization, where the vocabulary is initialized with all the individual characters in the language, and then the most frequent/likely combinations of the existing words in the vocabulary are iteratively added. Words that does not occur in the vocabulary are broken down into sub-words in order to search for possible matches in the collection. 4
In order to show that the Google BERT model we implemented outperforms
the results of the current performance state of art both on Liar dataset and
Polifact dataset, we report in Figure 2 and 3 the best results obtained for the
other approaches commonly used in literature for those datasets. In particular,
neural betworks such as CNN, BI-LSTM and C-HAN were initialized with
300dimensional pre-trained embeddings from GloVe [
We compared the performances on well-established evaluation measure like:
Accuracy, Precision, Recall, F1 measure, Area Under Curve (AUC) [
We hypothesize that our results are quite better due to a ne hyper parameter tuning we performed, a better pre-processing step and the proper transformation.
Confusion Matrix for LIAR dataset
Confusion Matrix for Polifact dataset
For the sake of completeness, we report in Figure 4a and Figure 4b the detailed confusion matrices obtained for LIAR and Polifact datasets. 5
In this paper, we investigated the problem of fake news detection by deep learning
algorithms. We developed a framework the leverage Google BERT for analyzing
real-life datasets and the results we obtained are quite encouraging. As for future
work, we would like to extend our analysis by considering also user pro les'
features, some kind of dynamic analysis of news di usion mechanism[