Introduction

Due to technological advances, more and more people have access to digital platforms. Users now have a much easier time interacting and communicating; because they can share their criteria regarding any news with friends or other users, and it is also generally cheaper to produce and consume news from digital platforms compared to traditional media, such as newspapers or television news channels.

These advantages of digital platforms allow the spread of fake news very quickly among thousands of users, thus causing disinformation among them. An example of the proliferation of false news on social networks was evidenced during the beginning of the pandemic, in which much false news regarding the origin, treatment, and transmission of SARS-Cov-2 was spread on social networks [1].

A solution to avoid the proliferation of misleading or false news on the networks, which have a great impact on society, would be to rely on professionals, such as journalists, to verify the veracity of the news based on published facts in newspapers or trusted sites. This solution is not very viable because it tends to be very slow and expensive as a result of the amount of information circulating on the networks.

As a result of this problem in the area of natural language processing, multiple investigations have been started aimed at the automatic detection of false news, because through the use of artificial intelligence, we can reduce the time and effort necessary for humans to invest in the classification of the news, and in this way stop the spread of the same on digital platforms.

In this paper, we have tackled the Multi-class fake news detection of news articles in English at CLEF2022-CheckThat!. This task consists of a multi-class classification of articles to determine if the claim made in the article is true, false, partially false, or other due to lack of evidence. The paper discusses results obtained using architectures based on machine learning and deep learning combined with stylometric features.

This paper is structured as follows: Section 2 presents a global overview of the state of the art in the area of fake news detection. In particular, the different perspectives addressed for the solution of this task are presented. Section 3 describes the dataset used for the task. Section 4 presents the two approaches used to solve the task, the first based on machine learning and the second on deep learning, both combined with the use of stylometric features. Section 5 presents the experiments carried out with both architectures and the results obtained. The paper ends by presenting the conclusions and acknowledgments.

Related Work

In the literature, there are multiple investigations related to detecting fake news. This task has been tackled from four perspectives: knowledge-based methods, origin-based methods, news propagation-based methods, and style-based methods [2].

The knowledge-based methods focus on verifying the news's content against known facts about it. The origin-based methods ascertain the source's credibility, i.e., where the news was published. These methods also consider the dissemination of the news on social media. On the other hand, the propagation-based methods carry out the fake news detection by evaluating the scope of the information on the Internet and analyzing how users disseminate this news. Finally, the style-based methods study the content of the news to assess the author's intention, whether or not they show the intent to deceive the reader [2].

The use of supervised classifiers to detect fake news based on style is prevalent, in particular, Support Vector Machines (SVM), Random Forest(RF), Naive Bayes, Logistic Regression(LR), and XGBoost. These algorithms receive the content of the news represented by syntactic, lexical, and semantic characteristics extracted from the news texts.

For example, in [3] the authors propose a method for detecting fake news based on machine learning. They also present ways to apply this method on Facebook. The author's proposed method uses the Naive Bayes classification model to predict whether a Facebook post will be labeled as real or fake.

In particular, [4] uses a supervised classifier combined with a feature selection-based method to assess the credibility of a corpus of tweets. In this work, the authors identify four types of features; these are features based on the messages (size of the messages, re-tweet, number of words of positive or negative sentiment contained in the message, and occurrence of hashtags or not), features based on users (registration age, number of followers, and number of tweets the user has written in their account), features based on topics (proportion of tweets containing urls, the ratio of tweets containing hashtags), and finally, features based on the propagation of the tweets (depth of the graph built based on the re-tweets, and the number of initial tweets of the topic).

Among the most recent proposals is the one presented by [5] in which, through the use of stylometric or linguistic characteristics and machine learning models, the authors improved the existing results in state of the art for the detection of false news, specifically in the dataset FakeNewsNet [6]. In the system proposed by the authors, they used three sets of stylometrics features that are most prominent in the news texts of the data set.

Many research works in the literature use deep learning architectures to detect fake news. In these architectures, the news content is first embedded at the word level, and then this embedding is processed by a neural network, for example, convolutional neural networks (CNN), recurrent neural networks (RNN) such as Long short term memory (LSTM), Bidirectional long short term memory (BI-LSTM), or a transformer architecture such as BERT [7]. The main advantage of using deep learning models over existing classical feature-based approaches is that these models can identify the best set of features describing texts on their own.

In [8], a hybrid model was proposed. The model is based on convolutional neural networks and outperform other traditional machine learning models. The author also compared the performance of SVM, LR, Bi-LSTM, and CNN models on his proposed dataset called "LIAR". On the other hand, in [9] an analysis of the linguistic features of an unreliable text was carried so that the authors were able to develop and present an LSTM model that obtained good results.

Currently, pre-trained language models such as BERT and ELMo are receiving great attention in different natural language processing tasks related to text classification. For example, [10] and [11] compare BERT to traditional machine learning methods. In [12] the author proposes the FakeBERT model, which is a combination of BERT and three parallel blocks of 1d-CNN that has different convolutional layers of different kernel sizes with filters for better learning.

Data description

The dataset used for this task is provided by CLEF2022 -CheckThat! Lab Fighting the COVID-19 Infodemic and Fake News Detection for the multi-class fake news detection of news articles in English [13]. The dataset has the format Public Id, Title, Text and Our Rating. The corpus consists of 1264 news collected from different fact-checking sites written in English. The news pieces are classified into four classes: true, false, partially false, and others due to lack of evidence. Table 1 shows the distribution of classes according to the four labels present in the dataset, it can be seen that there is an imbalance with respect to the labels. Table 2 shows a sample of the content of the dataset.

When reviewing the dataset, we found that there are some inconsistencies. For example, some instances have news content in both, Text and Title columns. Below, we show some of the inconsistencies we identify in the dataset:

• There are about 61 news titles that contain more than 40 words (these are complete news articles). • There are about 62 news texts with no more than 20 words • There are 178 repeated news.

• There are about 21 repeated news articles with different titles or labels (Table 3).

Methods

We propose two approaches for detecting fake news. The first approach is based on traditional machine learning using word, character, and POS tag n-grams. The second approach is based on deep learning combining pre-trained BERT embeddings with convolutional neural networks. In both methods, we introduced stylometric features to improve the performance of the classification models.

Sylometric Features

Stylometrics is a branch of computational linguistics that studies the statistical analysis of linguistic features in texts [14]. Stylometrics feature-based methods are used in multiple natural language processing tasks, including authorship attribution, authorship verification, author profiling, style change detection, and written text classification [15].

Stylometric features can be classified into lexical-based, syntax-based, structural, and text content-specific features. After thoroughly exploring the training data, we found some noticeable linguistic patterns that we used as additional stylometric features.

• Misspelled words.

• Use of tags (with @ or #) inside the text. • Text written in first person singular or plural.

• The writer addresses the reader by the pronoun "you".

• Repetition of sentences or paragraphs in the text.

• Overstatement of sentences with capital letters or interrogation and exclamation signs.

Machine Learning model

This section shows the proposed method using traditional machine learning classification algorithms. The method is implemented in python using the scikit-learn [16].

For training the fake news classification model, we added last year's training and test data of the competition to the corpus. Then we removed repeated news, promotional phrases, and contractions (e.g., we changed wouldn't to would not). It is worth noticing that the promotional phrases were difficult to find, and we likely left some in the texts. We then divided the corpus into train and test with a stratified 5-fold. After that, only with the text of the news, we extracted the following features:

• n-gram ranges of words with TF-IDF, leaving stopwords, capital letters, and numbers.

• n-gram ranges of characters with TF-IDF, leaving stopwords, capital letters, and numbers. • Sum of #, ?, ! and @. • Number of uppercase.

• Tagger of n-gram ranges of POS tags using NLTK.

• Number of repeated sentences.

• Number of misspelled words.

Depending on which attributes we wanted to use in the model, we joined them into a matrix (one for train and one for test) as new columns and then normalized them. In the case of n-grams, we used different ranges, including n from 2 to 4 (2,4), n from 3 to 5 (3,5), etc. To keep a simpler notation, we refer to these ranges only as "n-grams".

We used the training matrix to train a classification algorithm and then predict the label of the test and train data (of that fold). We used different classification algorithms like Logistic Regression (LR), Support Vector Machine (SVM) with a polynomial kernel of degree 3, Gradient Boosting classifier, and Multi-Layer Perceptron (MLP) classifier. We used the default parameters of the algorithms as implemented in the scikit-learn library.

Finally, we computed the mean of the f1-macro scores from the 5-folds. We tried different sets of features with different classification algorithms to find the highest score.

Deep Learning Model

Our second approach uses a BERT embedding layer connected to a convolutional neural layer. The output of this process is combined with stylometrics features extracted from the news.

We compose the architecture with several modules and multiple layers. The pre-processing module cleans and tokenizes the news texts. The text features module is responsible for generating the embeddings. Another module extracts and normalizes the stylometric features from the news texts. Finally, the architecture contains a combination module where we introduce the stylometric features to the final representation. The layers that perform the classification step are a Linear layer, a Dropout layer, and a Softmax layer. In figure 1 we show the diagram of our proposed architecture.

The first phase of the architecture consists of the pre-processing news module. In particular, during the experimentation phase, tests were carried out by removing stopwords, removing punctuation marks, converting characters to lowercase, removing promotional phrases, and unpacking contractions for better context (i.e., "won't" is changed into "will not").

The second phase of the architecture consists of obtaining the word embeddings. In this phase, experiments were carried out using BERT embeddings, word2vec and glove. The second phase of the architecture consists of obtaining the word embeddings and generating the linguistic features. In this phase, for the generation of word embeddings, experiments were carried out using bert embeddings, word2vec and glove. On the other hand, for the generation of linguistic features, once the texts of the news were pre-processed, we computed the number of characters in uppercase for each of the news items, the number of words in capital letters, the number of repeated sentences, the number of symbols (?, ¡,¡#, @) present in the news and the number of words with misspellings. After computing these features, we scaled the new features using the Min-Max normalization technique, thus normalizing the values to [0, 1].

The third phase consists of the concatenation of the resulting vector after being applied to a deep learning layer based on CNN or LSTM models, then applied to a linear layer, and finally applied to a dropout layer. This resulting vector is concatenated with the standard values referring to the computed linguistic characteristics of news texts. These linguistic features consist of the number of uppercase characters, number of uppercase words, number of symbols (?, !,#, @), and number of misspelled words.

Finally, a linear layer is applied to obtain the classification, followed by a softmax layer to the results obtained from the combination.

During the experimentation phase, we used the dataset resulting from the union of the training dataset presented in the CLEF-2021 CheckThat! lab task 3 on fake news detection [17], the testing dataset released in the CLEF-2021 CheckThat! lab task 3 on fake news detection and the dataset presented this year in the Multi-class fake news detection of news articles in English at CLEF2022-CheckThat. We eliminated the repeated instances and the instances that presented inconsistencies with the classification. Then we partitioned the data into 80% data for training and 20% data for validation.

Experiments and Results

In this section, we present the results of the experiment we performed during the experimentation phase.

Machine Learning Approach

We experimented with the traditional machine learning approach using different feature sets and classification algorithms. Overall, the Logistic Regression and MLP classifiers achieved better classification performance. Also, the best character n-gram set was (2,4), and the POS tags n-grams did not lead to better results.

Table 4 shows the combination of features and algorithms that yielded better results, where "X" indicates that we did not include the feature set, "O" that we include the feature set, and (_, _) the n-gram range used. We used either all the stylometric features or none of them in all experiments. The stylometric features column indicates the presence or absence of these features.

Table 5 shows the best results on the test set: the combination of stylometric features, word ngram, and character n-gram with the MLP algorithm. This combination allowed an improvement of over 2% points compared to the rest of the combinations.

Deep Learning Approach

We experimented with several deep learning architectures. The best results obtained, based on the 𝐹 1 macro, is composed of a convolutional neural network with bert base model uncased for the words embeddings generation1 , a batch size equal to 12, a dropout layer of 0.1, a number of kernels equal to 16, a number of epochs equal to 10, and cross-entropy loss function as a combination of parameters. Table 6 shows the different stylometric combinations that allowed obtaining the bests results with the deep learning architecture. The combinations are composed of the number of uppercase characters, number of repeated sentences, number of symbols, and number of spelling errors. In table 6, "X" indicates the exclusion of the feature in the experiment, and "O" indicates the inclusion of the feature.

Table 7 shows the results obtained by the deep learning approach. The stylometric features that allowed the best result were the number of uppercase characters and the number of repeated sentences present in the news texts. This combination allowed an improvement of over 2% points.

Conclusion

In this paper, we analyzed two approaches for the Multi-class fake news detection of news articles in English at CLEF2022-CheckThat!. In both approaches, we introduced stylometric features to improve the performance of the classification models. Our results show that including stylometric features can improve both approaches. Our best result was 0.2951 for the 𝐹 1 -macro score using as stylometric characteristics the number of uppercase characters, number of repeated sentences, number of symbols, and number of spelling errors.