Fake news can mislead public opinion, weaken social order, limit the legitimacy of government, and lead to a serious threat to social stability. Therefore, the early detection of fake news from the online platform is extremely important. Most of the previous literature has focused on finding fake news from resource-rich languages like English, Hindi, and Spanish. The current work utilizes the dataset of Urdu language for fake news detection. Two diferent models have been proposed in the paper. The first one is an ensemble-based technique and the second one is a multi-layer dense neural network. The multilayer dense neural network-based approach performed better with character n-gram TF-IDF features to achieve a macro 1-score of 0.8101.
Social media and blogs have become a major source of news across the world [
The research community has been very active in the detection of systems for the automated fake identity of news, reviews, person, and so on [9, 10] in recent past. In the area of natural language processing, fake news detection has emerged as one of the most researched topics. The identified problem would have been simpler if the news on the web could have appeared only in a single language. But, More than 5000 plus languages are used worldwide. Developing a generalized fake news detection system that works for all languages is nearly impossible. Substantial works have been done for resource-rich languages like English, Hindi, Spanish, and so on. For example, [11] presented a system for fake news detection in English news by extracting lexical, syntactic, and semantic features. [12] proposed a model to detect fake news in Spanish using several lexical features like bag-of-words, parts of speech, and n-grams. [13, 14] proposed deep long short term memory model for fake news detection.
Despite having more than 100 million speakers worldwide, Urdu is still a resource-poor language in NLP due to the scarcity of the labelled datasets. Therefore, very few works have been reported for Urdu fake news identification. [15, 16] proposed a benchmark dataset for fake news detection in Urdu. The current paper utilizes the dataset proposed in the shared task UrduFake-FIRE2020 [17] for fake news detection in the Urdu language. The ensemble-based model and a multi-layer dense neural network-based model are proposed. The results show that with a 0.8101 macro F1-score, the character n-gram features with a multi-layer deep neural network performed best.
The rest of the article is organized as follows: Section 2 explains the details of the proposed model along with the dataset description and feature extraction. Section 3 reports various experiments and their results. Finally, Section 4 concludes the paper by highlighting the main ifnding.
The overall model diagram for the proposed multi-layer dense neural network can be seen in Figure 1. Along with it, an ensemble approach for identifying fake news is also proposed. The data and model description used in the current work is explained in the following subsections.
The current work utilizes the dataset shared for the UrduFake-FIRE20201 task for detecting fake news in Urdu language[17, 18]. The dataset contains news related to six diferent domains such as technology, education, business, sports, politics, and entertainment. The overall data statistic can be seen in Table 1.
The experimentation was performed with two diferent models: (i) Ensemble approach, and (ii) Multi-layer dense neural network. In both cases, one to three-gram (1-3 gram) character TF-IDF (Term-Frequency-Inverse-Document-Frequency) features were used for model training. In the case of the ensemble approach, the combination of Random forest, Decision tree, and AdaBoost classifiers were used. The predictions of each classifier were combined and performed the majority voting among them to assign a final class tag to each of the test data samples.
Like the ensemble approach, in the case of the multi-layer dense neural network, 1-3 gram character TF-IDF features were given input to the dense layer of the network, as it is shown in Figure 1. The network has four dense layers containing 2048, 512, 128, and 2-neurons at the first, second, third, and fourth layers, respectively. A regularizer Dropout with a rate of 0.4 is used in between every pair of dense layers to minimize the over-fitting. In each of the dense layers, except the output layer, ReLu activation function is used whereas at the output layer Softmax activation function is used to get the probability of being real and fake news. As the performance of deep learning-based models is very sensitive to the hyper-parameters, we performed extensive experiments with varying batch size, learning rate, dropout rate, epochs, and the number of neurons in every hidden layer. The best-suited hyper-parameters for the proposed system are listed in Table 2.
Three diferent predictions for the testing dataset were submitted for the evaluation at the workshop. We submitted two diferent runs from the ensemble approach and one run from the multi-layer dense neural network models. The results for each of the runs are listed in Table 3. For the ensemble approach, the best results were achieved for the second run (Run-2) with the precision of 0.9387, recall of 0.3066 for the fake class whereas the precision of 0.7037, recall of 0.9880 for the real class. In terms of macro 1-score and accuracy, the ensemble approach Dropout layer (Dropout rate = 0.4) , sn xa o m reun ftoS 2 Dropout layer (Dropout rate = 0.4) Dropout layer (Dropout rate = 0.4) , s n ro u u L e e 8n R 2 1 , s n ro u u L e e 2n R 1 5 , s n reuo eLu 1-3 gram character TF-IDF features (10,000 top
features) achieved a macro 1-score of 0.6870 and an accuracy of 0.7325, as can be seen from Table 3.
The proposed multi-layer dense neural network achieved the best performance among all the three submitted predictions. The multi-layer dense neural network achieved the precision of 0.8900, recall of 0.5933 for fake class, and precision of 0.7966 and recall of 0.9560 for real class. In terms of macro 1-score and accuracy, the multi-layer dense neural network achieved a macro 1-score of 0.8101 and an accuracy of 0.8200, as can be seen from Table 3.
The widespread dissemination of misleading information has significantly afected individuals and society. In this work, we have proposed an ensemble approach and a multi-layer dense neural network for the detection of fake news written in the Urdu language. The proposed multi-layer dense neural network performed best with the character n-gram TF-IDF features and achieved a macro 1-score of 0.8101. [8] J. P. Singh, N. P. Rana, Y. K. Dwivedi, Rumour veracity estimation with deep learning for twitter, in: International Working Conference on Transfer and Difusion of IT, Springer, 2019, pp. 351–363. [9] S. Saumya, J. P. Singh, Detection of spam reviews: A sentiment analysis approach, Csi
Transactions on ICT 6 (2018) 137–148. [10] S. Saumya, J. P. Singh, Spam review detection using lstm autoencoder: an unsupervised approach, Electronic Commerce Research (2020) 1–21. [11] V. Pérez-Rosas, B. Kleinberg, A. Lefevre, R. Mihalcea, Automatic detection of fake news, arXiv preprint arXiv:1708.07104 (2017). [12] J.-P. Posadas-Durán, H. Gómez-Adorno, G. Sidorov, J. J. M. Escobar, Detection of fake news in a new corpus for the spanish language, Journal of Intelligent & Fuzzy Systems 36 (2019) 4869–4876. [13] A. Giachanou, P. Rosso, F. Crestani, Leveraging emotional signals for credibility detection, in: Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, 2019, pp. 877–880. [14] B. Ghanem, P. Rosso, F. Rangel, An emotional analysis of false information in social media and news articles, ACM Transactions on Internet Technology (TOIT) 20 (2020) 1–18. [15] M. Amjad, G. Sidorov, A. Zhila, H. Gómez-Adorno, I. Voronkov, A. Gelbukh, “bend the truth”: Benchmark dataset for fake news detection in urdu language and its evaluation, Journal of Intelligent & Fuzzy Systems (2020) 1–13. [16] M. Amjad, G. Sidorov, A. Zhila, Data augmentation using machine translation for fake news detection in the urdu language, in: Proceedings of The 12th Language Resources and Evaluation Conference, 2020, pp. 2537–2542. [17] M. Amjad, G. Sidorov, A. Zhila, A. Gelbukh, P. Rosso, Urdufake@fire2020: Shared track on fake news detection in urdu (2020). Proceedings of the 12th Forum for Information Retrieval Evaluation (FIRE 2020), Hyderabad, India. [18] M. Amjad, G. Sidorov, A. Zhila, A. Gelbukh, P. Rosso, Overview of the shared task on fake news detection in urdu at fire 2020, CEUR Workshop Proceedings (2020). Working Notes of the Forum for Information Retrieval Evaluation (FIRE 2020), Hyderabad, India.