Fake news has the potential to mislead the public, damage social order, undermine government legitimacy, and pose a major danger to societal stability. As a result, early identification of fake news via Internet platforms is critical. The majority of previous research has focused on detecting false news in resourcerich languages like English, Hindi, and Spanish. The current study makes use of an Urdu language dataset to detect fake news. Three diferent models have been proposed in the paper. The first one is a dense neural network (DNN)-based model, the second one is a Majority voting-based ensemble model, and the third one is the Probability averaging-based ensemble model. The proposed dense neural network-based model performed better with character n-gram TF-IDF features and achieved a macro 1-score of 0.59 and an accuracy of 0.72. The code for the proposed models is available at https://github.com/Abhinavkmr/Urdu_Fake_News_Detection.git Forum for Information Retrieval Evaluation, December 13-17, 2021, India
People are more inclined to pick an online platform for generating or consuming news because
of the ease of access and freedom to distribute Internet content [
Verónica et al. [
Despite having over 100 million speakers globally, Urdu has a limited number of labeled
datasets, making it a resource-poor language in NLP. As a result, only a few eforts for detecting
false news in Urdu have been reported. Amjad et al. [16, 17] created a benchmark dataset for
Urdu fake news. Kumar et al. [
The rest of the paper is organized as follows: The following is how the rest of the article is structured: The details of the proposed model, as well as the dataset description and feature extraction, are explained in Section 2. Section 3 details a variety of experiments and their outcomes. Finally, Section 4 brings the article to a close-by presenting the most important ifnding.
The overall flow diagram of the proposed models can be seen in Figure 1. Three diferent models were proposed for the fake news identification from Urdu news: (i) Dense Neural Network (DNN)-based model, (ii) Majority voting-based ensemble model, and (iii) Probability averagingbased ensemble model. The overall data statistic used to validate the proposed system can be seen in Table 1.
The suggested dense neural network (DNN) architecture is made up of four layers, each with 1,024, 512, 128, and 2-neurons. The top 15,000, uni-gram, bi-gram, and tri-gram character-level
g n i t o V y t i r o j a
M ty g i l n i i b g rabo reav P A
TF-IDF features are utilized as input to the DNN model. We conducted extensive experiments to ifnd the best-suited hyper-parameters because the performance of deep learning-based models is sensitive to the hyper-parameters chosen. The best results were obtained using a dropout rate of 0.3, a learning rate of 0.001, a batch size of 16, binary cross-entropy as a loss function, and Adam as the optimizer with 100 epoch training.
In the case of the Majority voting-based ensemble model, predictions of Logistic Regression, Decision Tree, and Adaboost classifiers are used to find the final class value. The final class value is decided based on the majority voting. The overall diagram of the model can be seen in Figure 1. To provide input to the classifiers, top 30,000 uni-gram, bi-gram, and tri-gram character-level TF-IDF features were used.
In the case of the Probability averaging-based ensemble model, Logistic Regression and AdaBoost classifiers are used to get the class probability value for fake and real classes. Then the classwise probability averaging was performed to get the final probability and based on the final probability final class level is determined. The overall flow diagram of the model can be seen in Figure 1. To provide input to the classifier, top 30,000 uni-gram, bi-gram, and tri-gram character-level TF-IDF features were used. To implement all the classifiers, Sklearn Python library1 is used with default parameters.
The performance of the proposed models is measured in terms of precision, recall, 1-score, and accuracy. Along with this, the confusion matrix is also plotted to visualize the performance. 0.23 0.97 0.60 0.13 0.99 0.56 0.05 1.00 0.53 175 150 125 100 75 50 25
The results for diferent models are listed in Table 2. The accuracy of the suggested DNN model was 0.72 and the macro 1-score was 0.59. The suggested method achieves a recall of 0.23 for the fake class. Figure 2 depicts the DNN model’s confusion matrix. The suggested Majority voting-based ensemble model has a macro 1-score of 0.52 and an accuracy of 0.70. It had a recall of 0.13 for the false article class. Figure 3 shows the confusion matrix for the ensemble model based on majority voting. The proposed Probability averaging-based ensemble model is able to achieve a macro 1-score of 0.45 and an accuracy of 0.68. For the fake article class, it achieved a recall of 0.05. The confusion matrix for the Probability averaging-based ensemble model can be seen in Figure 4.
The widespread dissemination of erroneous information has afected both individuals and society. In this paper, we suggest three distinct methods for detecting false news in Urdu news articles. With a macro 1 -score of 0.59 and an accuracy of 0.72, the suggested dense neural network-based model fared better. In the future, a more robust ensemble-based model for obtaining classification accuracy might be created. For the identification of fake news from Urdu news articles, a Transformer-based approach can also be investigated. news in a new corpus for the spanish language, Journal of Intelligent & Fuzzy Systems 36 (2019) 4869–4876. [12] A. Giachanou, P. Rosso, F. Crestani, Leveraging emotional signals for credibility detection, in: Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, 2019, pp. 877–880. [13] B. Ghanem, P. Rosso, F. Rangel, An emotional analysis of false information in social media and news articles, ACM Trans. Internet Technol. 20 (2020). URL: https://doi.org/10.1145/ 3381750. doi:1 0 . 1 1 4 5 / 3 3 8 1 7 5 0 . [14] A. Priya, A. Kumar, Deep ensemble approach for COVID-19 fake news detection from social media, in: 2021 8th International Conference on Signal Processing and Integrated Networks (SPIN), IEEE, 2021, pp. 396–401. [15] P. K. Roy, S. Chahar, Fake profile detection on social networking websites: A comprehensive review, IEEE Transactions on Artificial Intelligence 1 (2020) 271–285. doi: 1 0 . 1 1 0 9 / T A I . 2 0 2 1 . 3 0 6 4 9 0 1 . [16] M. Amjad, G. Sidorov, A. Zhila, H. Gómez-Adorno, I. Voronkov, A. Gelbukh, Bend the Truth: A benchmark dataset for fake news detection in Urdu and its evaluation, Journal of Intelligent & Fuzzy Systems 39 (2020) 2457–2469. doi:1 0 . 3 2 3 3 / J I F S - 1 7 9 9 0 5 . [17] M. Amjad, G. Sidorov, A. Zhila, Data augmentation using machine translation for fake news detection in the Urdu language, in: Proceedings of The 12th Language Resources and Evaluation Conference, 2020, pp. 2537–2542. [18] A. F. U. R. Khiljia, S. R. Laskara, P. Pakraya, S. Bandyopadhyaya, Urdu fake news detection using generalized autoregressors, in: FIRE (Working Notes), 2020. [19] S. M. Reddy, C. Suman, S. Saha, P. Bhattacharyya, A GRU-based fake news prediction system: Working notes for UrduFake-FIRE 2020., in: FIRE (Working Notes), 2020, pp. 464–468. [20] A. Maaz, S. Butt, H. I. Amjad, A. Zhila, G. Sidorov, A. Gelbukh, Overview of the shared task on fake news detection in Urdu at FIRE 2021., in: In CEUR Workshop Proceedings, 2021. [21] A. Maaz, S. Butt, H. I. Amjad, A. Zhila, G. Sidorov, A. Gelbukh, UrduFake@ FIRE2021: Shared track on fake news identification in Urdu., in: In Forum for Information Retrieval Evaluation, 2021.