Fake news detection is recognized as a key natural language processing task. Recognizing the importance of the task, several attempts have been made for fake news detection in Western languages. However, fake news detection in Urdu has received little attention of researchers. A key reason to that is the scarcity of fake news datasets for Urdu. In order to promote research and development in the area, a track is dedicated to the second fake news detection in the Urdu language, UrduFake'21. This study has proposed to use ensembling machine learning models for Urdu fake news detection. The proposed approach employs a voting-based approach of the three most efective techniques to decide that the given news article is fake or real. For the evaluation of the proposed approach, experiments are performed using several classical machine learning techniques, three types of features, unigram, bigram and trigram and the released dataset. The results of the experiments revealed that the proposed approach is more efective than the individual techniques. According to the results released by the organizers our proposed approach achieved a macro average F1 and accuracy scores of 0.621 and 0.713, respectively, and it is ranked 4ℎ among the 19 submissions.
Over the years, news and media have become an integral part of our life. According to Statista,
the worldwide news, entertainment and media market has a market value of 2.1 trillion US
dollars [
Recognizing the importance of fake news, several studies have been conducted on fake news
detection [
To the best of our knowledge, [
The rest of the paper is organized as follows. Section 2 provides an overview of the related work. Section 3 presents the specifications of the dataset used for the experimentation. The Section 4 presents our proposed ensembling approach and the machine learning techniques used for the experimentation. The results of the experiments are presented in Section 5. Finally, section 6 concludes the paper.
It is observed that majority of the studies have been conducted on fake news detection in rich resource languages which includes, English, Chinese and Italian, whereas, little work has been done on low resource languages, such as Urdu [9]. Therefore, this section provides an overview of the notable existing studies conducted on Urdu fake news detection.
Table 1 provides a summary of the techniques proposed in literature for Urdu fake news detection. It can be observed from the table that the existing studies have used diverse techniques for fake news detection. For instance, a study from Urdufake’20 has used diferent variants of XGBoost and RoBERTa [10]. The results of the experiments showed that XGBoost outperformed the other techniques for Urdu fake new detection. Similarly, another study [11] used XLNet model with the AR pre-training method, whereas [12] used ensemble of machine learning techniques along with Multi-layer Dense Neural Network which achieved a very high F1 score.
The UrduFake’21 track at the Forum for Information Retrieval Evaluation 2021 (FIRE2021) has provided a corpus of 1600 news articles. The key strength of the corpus is that it includes real and fake news from diverse domains. That is, it includes news articles from business, health, showbiz, sports and technology. Another notable observation is that the corpus is balanced
Paper Techniques [9] LR, RF, SVM, AdaBoost, CharCNN-Roberta, XLNet pre-trained model, Dense Neural Network, Bi-directional GRU model, ULMFiT model, [10] XGBoost with multiple features, RoBERTa [11] XLNet with AR pre-training [12] Ensemble approach, Multi-layer Dense Neural Network [13] RF, Bi-directional Gated Recurrent Unit, Multi-head self-attention based transformer [14] SVN, RR, BERT, MLP, AdaBoost Gradient boosting, Extra trees [15] HTC, TL model, Ensemble techniques as it includes equal number of news articles from all the five domains, hence, providing equal opportunity for learning about fake news from all domains.
A summary specifications of the Urdu fake news corpus is presented in Table 2. It can be observed from the table that the dataset is composed of 950 real and 650 fake news. The presence of imbalance in the dataset may impede the efectiveness of supervised learning techniques. It can also be observed from the table that the training dataset is composed of 1300 news, whereas the testing dataset is composed of 300 new articles. Furthermore, the testing dataset is also imbalanced in favor of real news with a ratio of 2:1.
The focus of this study is to use of classical machine learning techniques for Urdu fake news detection. We have employed a systematic approach for ensembling classical machine learning techniques using a voting scheme. Furthermore, it involved the use of the training dataset and the preliminary testing dataset released before the submission of the notebook to UrduFake’21 track. However, the results presented in Section 5 are generated using a unseen testing dataset released by the organizers of UrduFake’21 for the competition.
As a starting point of the approach, a comprehensive set of classical machine learning techniques were identified. The set of the techniques identified for this study are presented in Table 3. It can be observed from the table that we identified nine techniques for the initial experimentation. Subsequently, experiments were performed using the initially released dataset and precision, recall and F1 score were computed for each technique. Accordingly, the top three most efective techniques, AdaBoost, LightGBM and XGBoost, were identified for further processing. To develop a deeper understanding of the three top performing techniques, the confusion matrices for each technique were generated. The confusion matrices of the three techniques are presented in Figure 1. Decision Tree, Logistic Regression, Random Forest, Naive Bayes, Support Vector Machine, K-Nearest Neighbor, AdaBoost, Light
GBM, XGBoost
The proposed ensemble technique employs a voting-based approach for predicting the final label of each news. An overview of the proposed approach is presented in Figure 2, whereas the details of the proposed approach are as follows: • Convert the complete annotations into numeric form by replacing real news (R) with 0 and fake news (F) with 1. That is, convert the benchmark values, as well as the predictions, into binary numbers. • Calculate the sum of all predicted values and store them separately. That is, for one news article, the sum of the results is 0 if all of the three top performing techniques predicted the news as a real, and 3 if all of the three techniques predicted the news as fake. Similarly, the value varies between 0 and 3 depending upon the predictions of the three techniques. • Produce the prediction labels by using the sum of the results that were stored in an earlier step. That is, if any of the 2 techniques predicted that the news is fake declare the news as fake (1), otherwise declare the news as real (0). • As a final step, determine the final label of each news by using the voting scheme. That is, if a label is declared as 1 (fake), whereas the prediction of XGBoost is real (0) and at the same time sum of results is 0, declare the final label as 1 (fake). Similarly, if the label is 0 (real), the prediction of XGBoost is 1 and the sum of separately stored results is 2, declare the final label as 0 (real). Whereas, if both conditions are false, the prediction of the XGBoost should be considered as the final label.
Experiments are performed using ten techniques, nine classical machine learning techniques and our proposed technique. For these techniques, unigram, bigram and trigram features are used. For the experiments the training and the testing dataset discussed in Section 3 is used. That is, 1300 news articles are used for training and 300 news articles are used for testing. The code used for the experiments can be downloaded from GitHub1. Subsequently, Precision, Recall and F1 scores are computed for the two classes. Also, macro average F1 and accuracy score are computed for all the techniques. The details of the results of unigram features are presented in Table 4 as the efectiveness of unigram features achieved a higher efectiveness score than bigram and trigram features.
It can be observed from the table that three techniques achieved a higher accuracy score of greater 0.70. These are Random Forest, AdaBoost and our ensembling approach. It can also be observed from the table that the macro average F1 score of two techniques is greater than 0.60, whereas, the macro average F1 score of the third technique is less than 0.60. The two techniques that achieved a macro average F1 score greater than 0.60 are AdaBoost and our ensembling approach.
As the macro average score of these two techniques are exactly equal to 0.62, therefore a further comparison of these two techniques is performed in-terms of the F1 scores of each type 1https://github.com/socialmedialisteninglab/FakeNewsDetectionUrdu2021 of news. It can be observed from the table that the F1 score of fake news class for the ensembling approach is slightly higher than that of AdaBoost. Furthermore, the recall score achieved by the proposed approach for the fake news class is higher than that of AdaBoost. This represents that the proposed technique is slightly more efective than AdaBoost for identifying fake news.
The importance of fake news detection is well-established and a number of studies have been conducted on fake news detection for Western and Asian languages. However, Urdu fake news detection has received less attention despite the fact that the risk posed by fake news in comparable with the Western world. To that end, UrduFake’21 track at the Forum for Informational Retrieval Evaluation 2021 (FIRE2021) has taken a significant leap towards promoting fake news detection research in the Urdu language. In this study, an ensemble of classical machine learning models is proposed for fake news detection. The proposed approach relies on using a voting scheme between the three most efective techniques for the detection of Urdu fake news. Experiments are performed with the using the unseen testing dataset released by UrduFake’21, nine classical machine learning technique and the proposed approach. The results of the 19 teams released by the organizers ranked our proposed approach 4ℎ among the 19 submissions. on fake news identification in Urdu, in: Proceedings of the Forum for Information Retrieval Evaluation, volume 2826, CEUR-WS, 2020, pp. 37–40. [9] M. Amjad, G. Sidorov, A. Zhila, A. F. Gelbukh, P. Rosso, Overview of the shared task on fake news detection in Urdu at FIRE 2020., in: Proceedings of the Forum for Information Retrieval Evaluation, volume 2826, CEUR-WS, 2020, pp. 434–446. [10] N. Lina, S. Fua, S. Jianga, Fake news detection in the Urdu language using CharCNNRoBERTa, in: Proceedings of the Forum for Information Retrieval Evaluation, volume 2826, CEUR-WS, 2020, pp. 447–451. [11] A. F. U. R. Khiljia, S. R. Laskara, P. Pakraya, S. Bandyopadhyaya, Urdu fake news detection using generalized autoregressors, in: Proceedings of the Forum for Information Retrieval Evaluation, volume 2826, CEUR-WS, 2020, pp. 452–457. [12] A. Kumar, S. Saumya, J. P. Singh, NITP-AI-NLP@UrduFake-FIRE2020: Multi-layer dense neural network for fake news detection in Urdu news articles., in: Proceedings of the Forum for Information Retrieval Evaluation, volume 2826, CEUR-WS, 2020, pp. 458–463. [13] S. M. Reddy, C. Suman, S. Saha, P. Bhattacharyya, A gru-based fake news prediction system: Working notes for Urdufake-FIRE 2020., in: Proceedings of the Forum for Information Retrieval Evaluation, volume 2826, CEUR-WS, 2020, pp. 464–468. [14] N. N. A. Balaji, B. Bharathi, SSNCSE_NLP@Fake news detection in the Urdu language (UrduFake) 2020, in: Proceedings of the Forum for Information Retrieval Evaluation, volume 2826, CEUR-WS, 2020, pp. 469–473. [15] F. Balouchzahi, H. Shashirekha, Learning models for Urdu fake news detection., in: Proceedings of the Forum for Information Retrieval Evaluation, volume 2826, CEUR-WS, 2020, pp. 474–479.