Social media has become an inevitable part of our life as we are primarily dependent on them to get most of the news around us. However, the amount of false information propagated through it is much higher than the genuine ones, thus becoming a peril to society. In this paper, we have proposed a model for Fake News Classification as a part of CLEF2021 Checkthat! Lab 1 shared task, which had Multi-class Fake News Detection and Topical Domain Classification of News Articles. We have used an ensemble model consisting of pre-trained transformer-based models that helped us achieve 4ℎ and 1 positions on the leaderboard of the two tasks. We achieved an F1-score of 0.4483 against a top score of 0.8376 in one task and a score of 0.8813 in another.
Nowadays, social media is the primary platform for people to get the latest news and updates
happening around them, either political or entertainment or even health-related. People are
more dependent on reliable information during this pandemic situation, which is propagated
through social media. But here, the situation is quite diferent as many information are fake,
which spreads faster than authentic ones [
CLEF-2021 CheckThat! [
In this paper, we have used transformers, which is efective for text classification because
of their self-attention mechanism and a better understanding of word features. We have
used the transformer-based model [
Fake information analysis and detection have gained attention because of the ease of availability of the data from social media. This is because even social media needs to curb the spreading of misinformation through their platform. We have many traditional machine learning methods to classify fake and real information. However, the performance of such systems is not that accurate because of the inability to understand the data. Deep Learning is now becoming an integral part of these fake information detection systems because of the computation capability. We have analyzed some recent works which have used deep learning models and some of the newer datasets used for fake news classification, which are explained below.
Umer et al.[
There were two tasks given on the competition website2 under Fake News Detection by CLEF2021-CheckThat! Lab3, namely Multi-class Fake News Detection of News Articles (Task 3a) and Topical Domain Classification of News Articles (Task 3b). The training dataset for Task3a is shown in table 1, which had 900 articles with respective labels and the testing data had 364 articles without labels. The training dataset for Task3b is shown in table 1, which had 318 articles with respective labels and the testing data had 137 articles. The label distributions of both the datasets are shown in figure 1.
Task 3a 3b
Data Training 900 318
Testing
364
137
We had a classification task under both Task3a and Task3b where we had to classify the articles
into labels and domains as in figure 1. The proposed model had Text Preprocessing, Tokenization,
Model Architecture, and Ensemble Modeling [
preprocessed sentences shown are corresponding to the subtask data. A detailed description is given in the following subsections.
The data provided by the organizers were mainly retrieved from various news websites and articles which need to be preprocessed; for this, we used the clean-text4 library from python, which helped in removing contents like URLs, ASCII conversions, etc.
We were dealing with articles that had sentences. To process the sentence data, we need to convert them to some tokens and pass them on to the model. We have used the tokenization5 approach corresponding to the pre-trained model being used, which expects the tokens to be in an explicit structure depending upon the model. Each model will tokenize based on its structure during training on the data. We have used Longformer [17], and RoBERTa [18] models and a combination of them which are modified versions of BERT [19].
We have used pre-trained models6 as the base model for this classification task. The model has been individually trained for data using the pre-trained weights, which gives the probabilities for the diferent labels. As transfer learning is being used, the model has its own vocabulary and pre-trained embeddings, which is fine-tuned to get the predictions using the training data. The same tokenizer and model are used to get the predictions for test data.
In this method, we have used an ensemble approach to predict the labels for the testing data. We have applied RoBERTa and Longformer as the base models for our classification. Longformer was chosen as it can handle up to a maximum length of 4096, whereas RoBERTa was chosen because it outperformed BERT in most of the downstream tasks and benchmarks. We have used the model prediction vectors from these models and combined them to get the final result for our classification model. The final prediction is the sum of the individual predictions by the models and taking the maximum probability among the class labels. This helps in predicting the labels with more probability towards a particular category. 4.4.1. Data Analysis and Modeling The longformer model was mainly chosen because as per the token distribution in the figure 3, we can see that most of the articles have tokens which are less than 3000, but we have some 20 odd number of articles having tokens of size 4096 (which is the maximum model can handle) even after truncation. Even though we lose some information, for training the longformer model, 3000 was chosen as the maximum token length for both the task as they had a similar distribution of tokens. When we tried to use the full token size, there wasn’t much diference in the predictions.
The training strategy for the RoBERTa model was diferent as it can handle only tokens up to 512 length. To train the data with this model, we split the single article into small chunks keeping the same label for the split parts. We had split one article into 450 tokens (around 50 is left because the model splits words into subwords) each so that the model can perform well. For example, if an article had 2000 tokens, we split it into four articles having 450 tokens and one having 200 tokens with the same labels as the original one. The one drawback of this method is that while we are testing the data, articles are truncated to 450 tokens which can afect the results.
Ensemble model output is obtained using the predicted probabilities for each of the models. Instead of taking the final prediction from the models, we took the probabilities for each of the labels (Task 3a) and domains (Task 3b). These probabilities were summed and rounded up to get the final predictions. Result analysis for both the tasks will be explained in the following section.
We have fine-tuned the pre-trained models for the training data using AdamW [ 20] and learning rate 2e-5 (as recommended in the original model). We had used cross-entropy loss as the loss function. The experiments were performed on a nvidia-dgx machine with CentOS, Tesla V100 32GB GPU. The learning rate was kept the same for all the tasks, and the number of epochs varied from 10-15 with callbacks on validation loss. The same parameters were used for both the tasks as they had similar data; only classification labels were diferent.
In this section, we will discuss the individual model results for both tasks. We have used a training and validation split of 0.20 for the training data, and the results of validation data are shown in the table2. The longformer model with maximum token length 3000 gave an F1-score (weighted average was taken as it will consider the proportion for each label in the dataset) of 0.60 for task 3a. Even though longformer had more tokens, it couldn’t get better than that of RoBERTa. Hence we came up with an ensemble model as our proposed model whose results are explained in the following section.
While coming to task 3b, we have tried a diferent method because longformer could not perform well for the token length of size 3000. Here we have used the RoBERTa model (with maximum token length of 450) directly on the entire article without splitting for the long sequences whose results are shown in the table 2. Then we split the data into chunks of 450 as explained in the last section, whose results are shown as RoBERTa_SplitText in the table 2. These two models were used as an ensemble model for the task 3b classification because the model which was trained on chunks of text could have a better understanding.
The final model results for test data are as shown in table 3. As discussed in the methodology section, we have used an ensemble of the fine-tuned Longformer and RoBERTa as the final prediction for the test data. We can see from the results that for task 3b ensemble model outperformed, but the Longformer performed well for task 3a. The final results of our proposed model for task 3b achieved 1 position in the leaderboard of the competition (position on the leaderboard is given in brackets corresponding to the task). Eventhough the ensemble model could perform well for task 3b, we believe some mislabelled articles were causing the ensemble model to underperform in task 3a. Also, we have given F1- weighted score in the previous section because of the unbalanced class labels. Here, we have provided the F1-macro score because it was used as the final score for the leaderboard by the organizers.
In this modern social media age, we have to be vigilant in the rapid spreading of fake information, which can have immense ramifications on our day-to-day lives. In this paper, we have focussed on building a model to classify the news articles from social media comprising politics, entertainment, COVID-19, etc., as fake or not. We have fine-tuned transformer-based models Longformer and RoBERTa to predict the news articles. Moreover, our results got improved when we implemented the ensemble combination of these models. In the future, this method can be extended to learn more features with diferent models used in combination, and also, we would evaluate our model on generic Fake News datasets. Eficiently Block the Influence of Misinformation in Social Networks, IEEE Transactions on Computational Social Systems (2021) 1–12. doi:10.1109/TCSS.2021.3059430. [17] I. Beltagy, M. E. Peters, A. Cohan, Longformer: The Long-Document Transformer, 2020.
URL: https://github.com/allenai/longformer. arXiv:2004.05150. [18] Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, V. Stoyanov, RoBERTa: A robustly optimized BERT pretraining approach, 2019. URL: https://github.com/pytorch/fairseq. arXiv:1907.11692. [19] M.-w. C. Kenton, L. Kristina, J. Devlin, BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (2018). arXiv:arXiv:1810.04805v2. [20] I. Loshchilov, F. Hutter, Decoupled weight decay regularization, 2019. arXiv:1711.05101.