Automated fake news detection has become vital in today's digital age. Diferentiating legit news from fake ones has become an important classification challenge in natural language processing (NLP). Various transformer-based deep learning approaches have taken widespread adoption from the research community due to its outstanding performance. We have participated in the 2020 Fake News Detection Challenge in the Urdu Language organized by Center for Computing Research (CIC), Instituto Politécnico Nacional (IPN), Mexico and have stood second. In this work, we have implemented a generalized autoregressor based model to classify news into fake or real. We have achieved an overall accuracy of 0.8400 and F1 macro score of 0.8370.
The age of the internet has led to a massive free flow of information. This information flow
has an added advantage of being easily accessible through widespread virtualization, leading
to cost benefits to humankind. This increased accessibility has led to widespread usage and
adaptability of the services it has to ofer, to the point that one’s principal source of information
becomes the internet. Due to its ubiquitous and global nature, it is quite dificult to have a
central authority to have control over it, without defeating the true purpose it has to ofer. As
we are moving more and more of our life to the virtual world, the importance of detecting
fake news has increased manifold as it has proven to have a negative impact on our society
[
Our work mainly focuses on the fake news detection task of the Urdu language. In this work,
we have considered the task of fake news detection as a binary classification problem. We
have used XLNet [
Rest of the paper is organized as follows. In Secti2owne discuss the relevant works in fake news detection. Dataset used in our work is described in Sect3i.oWne describe our system in detail in Section4. Experimental setup is discussed in brief in Sectio5.nWe discuss the result of our model and its analysis in Sectio6.nFinally, we conclude along with future works in Section 7.
Work related to automated fake news detection can be broadly classified into three distinct
categories’ viz. based on content, propagation, and social cont8e]x.tM[ajorly, content-based
approaches have been used in the past that leverages lexical and syntactical features to detect
writing styles and words to capture the deceptive nature of the news. One such wor9k]b,yus[es
satirical cues based on a support vector machine (SV1M0)] [algorithm to detect fake news. In
the past researchers have also worked upon to compare the linguistic features of fake news with
real news 1[
Very few works have been done on fake news detection on the Urdu language in the past,
due to very less availability of the requisite dataset for this task. For the given Urdu fake news
detection task 1[
For our work, the organizers of 2020 Fake News Detection in the Urdu Language Task provided
us with the dataset for fake news detection14[]. The authors of 1[
The data provided by the organizers was already preprocessed as discussed by the authors of [14]. For the monolingual corpus, we have used UrduH1atcokpreprocess the collected corpus. The preprocessing step included the removal of URLs, email IDs, and mobile numbers. The text was also normalized, space was added after punctuation marks and extra spaces were removed. Diacritics were also removed along with normalizing Urdu characters to obtain the text in the proper Urdu range.
For pretraining as well as fine-tuning we have followed the architectur5e]o.fT[he pretraining
and fine-tuning procedure are described in Sectio4n.1 and Section4.2 respectively. We have
used the XLNet model instead of bidirectional encoder representations from transformers
(BERT) [19] as, the later model has some drawbacks to ofer. The BERT based architecture tends
to corrupt the input with masks as it does not consider the dependency between the masked
positions. Also, the BERT model has a discrepancy between fine-tuning and pretraining the
model [
Leveraging unlabeled corpora has been very beneficial in various NLP tasks. Moreover, data for unsupervised tasks are easily available in large quantities compared to their supervised counterparts. Principally, there are two distinct pre-training objective’s viz. autoencoding (AE) and autoregressive (AR) language modelin5g].[Hence, following the work of5][, we have pre-trained our model based on the XLNet architecture, which employs a generalized autoregressive method that takes the advantage of both pre-training objectives AE and the AR language modeling. Apart from the architectural improvements in BERT, this XLNet model based on [20], integrates the recurrence mechanism and relative encoding scheme that improves performance in tasks involving longer sentences, like the given task of fake news classification.
For fine-tuning the pre-trained model for the main binary classification task i.e. to detect
whether the news is legit or fake we have followed the fine-tuning procedure of BER19T] [
as given by [
We have trained the XLNet model that uses the AR pre-training method and employs the use of language modeling objectives based on permutation. Since, each individual Urdu document/news provided by the organizers was to be predicted as fake or real, follo5w]iwnge [used a longer sequence length of 512. Due to a smaller classification dataset and limited computation power, we trained a much smaller model as originally trained by the autho5r].s Wofe[have used 6 layers with 4 attention heads, and an embedding dimension of 256. Inherently, larger parameters were scaled down by four. Here, the XLNet based model is pre-trained on the same domain of news dataset. Feature weighting scheme’s are not used. The embedding features from the pre-training step (obtained after Section4.1) is used in the fine-tuning step (as discussed in Section4.2) for classification. The XLNet classification task can be termed as the classification algorithm used.
The results for the fake news detection task was declared by the organizers of 2020 Fake News Detection in the Urdu Language. The authors of this work participated by the team name CNLP-NITS and as shown by the overall results on the web2sioteur team stood second out of the 39 teams participated. The participated systems were evaluated based on precision, recall, and F1 score. Our system reported an overall accuracy of 0.8400 and an overall F1 macro score of 0.8370. Precision, Recall and F1 score for the fake class and the real class are as shown in Table1.
2https://www.urdufake2020.cicling.org/results-and-rankings
Our automated fake news detection system adopts the generalized autoregressors technique for the binary classification task. Even though our XLNet based model shows good results, there are avenues for improvements. The monolingual corpus required for the pre-training step can be increased further. Several architectural improvements could also be made to incorporate the challenges faced in the Urdu language.
Authors would like to thank the organizers of the 2020 Fake News Detection in the Urdu Language Task from Center for Computing Research (CIC), Instituto Politécnico Nacional (IPN), Mexico for organizing this competition. The authors are also grateful to the Center for Natural Language Processing (CNLP) and Department of Computer Science and Engineering at National Institute of Technology, Silchar for providing the requisite support and infrastructure to execute this work. potentially misleading news, in: Proceedings of the Second Workshop on Computational Approaches to Deception Detection, 2016, pp. 7–17. [10] W. S. Noble, What is a support vector machine?, in: Nature Biotechnology, 2006, pp.
1565–1567. [11] H. Rashkin, E. Choi, J. Y. Jang, S. Volkova, Y. Choi, Truth of varying shades: Analyzing language in fake news and political fact-checking, in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, 2017, pp. 2931–2937. [12] V. Pérez-Rosas, B. Kleinberg, A. Lefevre, R. Mihalcea, Automatic detection of fake news, in: Proceedings of the 27th International Conference on Computational Linguistics, COLING 2018, Santa Fe, New Mexico, USA, August 20-26, 2018, 2018, pp. 3391–3401. [13] M. Potthast, J. Kiesel, K. Reinartz, J. Bevendorf, B. Stein, A stylometric inquiry into hyperpartisan and fake news, in: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 1: Long Papers, 2018, pp. 231–240. [14] M. Amjad, G. Sidorov, A. Zhila, H. Gómez-Adorno, I. Voronkov, A. F. Gelbukh, ”bend the truth”: Benchmark dataset for fake news detection in urdu language and its evaluation, in: J. Intell. Fuzzy Syst., 2020, pp. 2457–2469. [15] M. Amjad, G. Sidorov, A. Zhila, Data augmentation using machine translation for fake news detection in the urdu language, in: Proceedings of The 12th Language Resources and Evaluation Conference, LREC 2020, Marseille, France, May 11-16, 2020, 2020, pp. 2537–2542. [16] J. P. Posadas-Durán, H. Gómez-Adorno, G. Sidorov, J. J. M. Escobar, Detection of fake news in a new corpus for the spanish language, in: J. Intell. Fuzzy Syst., 2019, pp. 4869–4876. [17] A. Giachanou, P. Rosso, F. Crestani, Leveraging emotional signals for credibility detection, in: Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2019, Paris, France, July 21-25, 2019, 2019, pp. 877–880. [18] B. Ghanem, P. Rosso, F. M. R. Pardo, An emotional analysis of false information in social media and news articles, in: ACM Trans. Internet Techn., 2020, pp. 19:1–19:18. [19] J. Devlin, M. Chang, K. Lee, K. Toutanova, BERT: pre-training of deep bidirectional transformers for language understanding, in: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2019, pp. 4171–4186. [20] Z. Dai, Z. Yang, Y. Yang, J. G. Carbonell, Q. V. Le, R. Salakhutdinov, Transformer-xl: Attentive language models beyond a fixed-length context, in: Proceedings of the 57th Conference of the Association for Computational Linguistics, 2019, pp. 2978–2988.