Automatic detection of fake news is a highly important task in the contemporary world. This study reports the 2nd shared task called UrduFake@FIRE2021 on identifying fake news detection in Urdu language. The goal of the shared task is to motivate the community to come up with eficient methods for solving this vital problem, particularly for the Urdu language. The task is posed as a binary classification problem to label a given news article as a real or a fake news article. The organizers provide a dataset comprising news in five domains: (i) Health, (ii) Sports, (iii) Showbiz, (iv) Technology, and (v) Business, split into training and testing sets. The training set contains 1300 annotated news articles -750 real news, 550 fake news, while the testing set contains 300 news articles -200 real, 100 fake news. 34 teams from 7 diferent countries (China, Egypt, Israel, India, Mexico, Pakistan, and UAE) registered for participation in the UrduFake@FIRE2021 shared task. Out of those, 18 teams submitted their experimental results and 11 of those submitted their technical reports, which is substantially higher compared to the UrduFake shared task in 2020 when only 6 teams submitted their technical reports. The technical reports submitted by the participants demonstrated diferent data representation techniques ranging from count-based BoW features to word vector embeddings as well as the use of numerous machine learning algorithms ranging from traditional SVM to various neural network architectures including Transformers such as BERT and RoBERTa. In this year's competition, the best performing system obtained an F1-macro score of 0.679, which is lower than the past year's best result of 0.907 F1-macro. Admittedly, while training sets from the past and the current years overlap to a large extent, the testing set provided this year is completely diferent.
The proliferation of social media brought in various forms of cybercrime that urgently need
automatic solution for the safety of people online and beyond [
Natural language processing (NLP) researchers formulated the problem into subcategories of
fake news such as satire [
This paper describes the UrduFake@FIRE2021 shared task and its results. The task invited
the participants to tackle the problem of automatic fake news detection in Urdu in Nastalíq
script . The problem is shaped into a binary classification problem in which news articles from
various sources including such news outlets as BBC Urdu News, CNN Urdu, Express-News,
Jung News, Naway Waqat, and others, are ofered for classification as fake or real. During the
active competition phase the ground truth annotations for the testing set were hidden from
the participants, while the training set was provided with the corresponding ground truth
annotations. After the end of the competition, the both parts of the dataset were made publicly
available along with the corresponding ground truth annotations at the CICLing 2021 UrduFake
track at FIRE 2021 shared task homesite 1. This year’s track is the continuation of CICLing 2020
UrduFake track at FIRE 2020 [
The paper is structured as follows. An overview of previous relevant research can be found
in Section 3. We provide the task description in Section 4 and explain in detail the data
collection and annotation procedure in Section 5. Training and testing set splits and statistics are
outlined in Section 5.2. Sections 6 and 7 describe the choice of evaluation metrics and baselines
correspondingly. A high level overview and comparison of the solutions and approaches
submitted by the participants is provided in Section 8 along with the final results summarized
in Section Sections 9. A brief summary of the UrduFake@FIRE2021 track can be found in a
separate publication [
Urdu is the national language of Pakistan and has more than 230 million 2 speakers worldwide.
Many of these speakers carry out their written communication in the Nastalíq script. Urdu is
commonly written in the Nastalíq script, while the Devanagari script is commonly used for
Hindi. However, due to cultural and geographical proximity, Devanagari may be also used for
writing in Urdu. This creates a situation of digraphia for the Urdu language when two scripts
are used for writing in a language. Apart from this commonality, Urdu has other structural
similarities with Hindi and other South Asian languages [
South Asia has been sufering from numerous instances of fake news afecting its political, social, and economic situation. For example, Dr. Shahid Masood 3 who works as a TV anchor in Pakistan, was exiled and tortured for spreading false information about a child rape case. Another case of fake news in India was reported in the Washington Post 4, where many innocent people died because of a child traficking report.
These severe consequences of fake news reporting surge the urge for high quality automation of fake news detection in Urdu. Given that despite the numerous speakers Urdu is still a low/medium resourced language, we strive for providing larger annotated datasets and incentivize the community to develop state-of-the-art solutions for early detection of fake news.
Contemporary fake news is not solely produced by humans, but can also be generated through
bots [
Earlier solutions [
The dataset created for fake news identification mostly rely on social media platforms and
news outlets. The majority of the existing datasets are available in English [
Some of the online challenges to improve automatic fake news systems include Fake News
Challenge 5, multiple fake news detection competitions on Kaggle 6 as well as shared task tracks
organized by the academic community: PAN 2020 [
This task is aimed to motivate the community to come up with methods and systems for automatic fake news detection in Urdu language by providing an annotated dataset with a train/test split and competitive settings. The challenge is posed as a binary classification task where participants are to train their classifiers on the provided training part of the dataset and to submit the labels, either fake or real, for each news article from the testing set, the ground truth annotations for the latter being hidden from the participants. Organizers compute the evaluation metrics for each submission by comparing the submitted labels to the ground truth annotations.
The motivations of this shared task is to investigate whether and to which extent the textual content alone can be grounds for fake news detection and examine the eficiency of machine learning algorithms in identifying fake news articles written in Urdu in the Nastalíq script.
Here, a fake news article and fake news detection are defined as follows:
• F a k e N e w s : A news article that contains factually incorrect information with the intention
to deceive a reader and to make the reader believe that it is factually correct.
• F a k e N e w s D e t e c t i o n : Suppose that is a news article (without annotation) and ∈ ,
where is the total number of news articles. A fake news detection is a process in which
an algorithm calculates the likelihood of whether a given news article is a fake news
article by assigning a value between 0 and 1. In mathematical terms, this can be described
as () ∈ [
More elaborated definition of fake news is provided in our previous work [ 54].
This section gives an outline of the dataset created for the UrduFake shared task at FIRE 2021. Our previous research [54] reported the first version of this dataset, called “Bend The Truth” that contained 500 real news and 400 corresponding fake news. A new training dataset and test dataset data was acquired using the dataset collection and annotation guidelines presented in our previous research [54]. The dataset presented in this shared task is publicly available and can be used for research objectives 7.
The training dataset was released on April 30, 2021 8. It is important to mention that the training dataset used in 2021 UrduFake task comprised 1300 news article. This dataset was made up by combining the training dataset, which we presented in our previous research [54] “Bend The Truth” and testing dataset collected for UrduFake 2020 shared task. The training dataset contained 750 real news articles and 550 fake news articles. we presented a new test dataset that contained 200 real news and 100 fake news articles collected from January 2021 to August 2021 to test the proposed systems.
A crowdsourcing technique was used to collect the fake news articles. In other words, the fake news were composed by hiring professional journalists who deliberately wrote fake news of the corresponding real news. The journalists were provided a set of instructions to follow while writing fake news articles. This dataset contains five domains of the news: (i) Business, (ii) Health, (iii) Sports, (iv) Showbiz (entertainment), and (v) Technology.
All the news articles were labelled into two two types of news: (i) real news article, and (ii) fake news article. Diferent techniques were used to annotate and assemble real and fake news. This dataset can be used for future research using supervised machine learning and deep learning techniques. Figure 1 shows the list of news organizations used to crawl news articles.
To assemble real news articles, various traditional news media mainstream were used to crawl news manually. Manual procedures were followed to annotate a news article using the underline guidelines, the news would label as real news. The news organizations used to gather news items for annotation are presented in Figure 1 and all the news were manually crawled. The following guidelines were used to annotate a news item as a real news: 7https://github.com/MaazAmjad/Urdu-Fake-news-detection-FIRE2021 8https://www.urdufake2021.cicling.org/home 1. The news article was labeled as real news if the news meets the following criteria: • That news article is published by a credible newspaper or a prominent news media agency. • The integrity of that news article can be verified by other credible newspaper agencies. This was an important point to do fact-checking. For example, manual source verification was performed to check place of the event, image, date of the news and whether the provided information in the news article matched with the same news article but published by other newspaper or news agency as well. • Incongruity between news titles and its content was also confirmed to ensure that a news article has a correlation between the news headline and the body text. We read the complete news articles to check the incongruity between news titles and the body text.
It is important to highlight that a news article was removed If it did not fulfil one of the aforementioned criteria. Diferent news articles contained diferent words length. For example, CNN publish news articles that contains between 200-300 words. On the other hand, a news article published by BBC Urdu news typically contains on average 1500 words. Therefore, the real news articles contains heterogeneous length of words. This is how all the real news articles were collected and annotated.
To obtain fake news, the services of professional journalist were used who work in diferent news organizations in Pakistan. We hired professional columnist because they are expertise in writing news articles, and use diferent journalists techniques to make the news interesting to hook and and their written fake news can easily trick the reader. The real news articles were provided to the journalists and they were asked to write fake news corresponding to the real news. In other words, if a real news contains story about football, the correspond fake news article should also contain similar story but with fabricated information.
We used professional “crowdsourcing” for collecting fake news and the reasons are described as follows: 1. The news articles analysis with manual procedures for verification through web scraping approach was unfeasible. This is due to the facet that it is extremely challenging task to ifnd the corresponding fake news of a real news article. 2. No online service in Urdu language is available for news fact-checking. Unlike English, the news fact-checking is manually performed in Urdu.
This dataset contains news of five domains: (i) business, (ii) education, (iii) sports, (iv) showbiz (entertainment), and (v) technology. The journalists expertise was taken into account to ensure that the fake news corresponding to the real news is written by the domain expert. The journalists were asked to keep the same length of the news (fake news article should have the same words length as real news). In addition, we also instructed journalists to mitigate defined patterns so that the undesirable clues should not be induced to classify news articles. Therefore, journalists’ expertise were used to collected all the fake news articles.
The training set contained 1300 news articles, in which 750 news articles were annotated as real, and 550 news articles were annotated as fake news article. The training set and the testing set contained five types of news: (i) Business, (ii) Health, (iii) Showbiz (entertainment), (iv) Sports, and (v) Technology. Participants were allowed to use of the training set for validation, development, and parameter tuning. The training dataset made up by combining the training dataset, which we presented in our previous research [54] “Bend The Truth” and the testing dataset collected for UrduFake 2020 shared task.
The new test set was introduced that contained 200 real news and 100 fake news articles collected from January 2021 to August 2021. The test set was presented without the ground truth labels so that all the participants could evaluate and test the performance of their proposed systems. To evaluate and compare the performance of the classifiers submitted by the participants, the organizers used the truth labels of the test set. It is worth mentioning that the participants were unaware of the distributions of real and fake news in the test set.
In this shared task, we divided the dataset into two parts: (i) training set, and (ii) testing set. Initially, the training set was released so that the participants can train their classification models. Then, the test set was released so that the participants can predict the labels of whether a given news is real or fake. Table 1 describes the corpus distribution of the news articles by topics for the training and testing sets.
This is a binary classification task in which the task is to classify a news article as fake or real. All the participating teams were allowed to submit up to 3 diferent runs, i.e., labels for the testing set generated by their proposed classifiers. The ground truth annotations were used to compare the labels predicted by the participants’ classifiers. We used the evaluation metrics commonly used to measure the performance of binary classification on imbalanced datasets: two sets of Precision (P), Recall (R), and F1 score, one for the “real” class treated as a target class and the other for the “fake” class; the inter-class metrics Accuracy and F1-macro. The macro-averaged F1-macro, which is the average of F1real and F1fake, was also calculated to accommodate the dataset skew towards the real class. As detection of both classes (real and fake) is equally important, this is why we evaluated performance against both classes.
To introduce a baseline, we used the bag of words (BoW) approach. We used a combination of character, word, and function word bi-grams with TF-IDF weighting scheme for text representation. Function words are similar to stopwords, for more elaborated definition and list we suggest to refer to [58]. Decision Tree was selected as a classifier, which achieved surprisingly good results compared to other traditional ML classifiers on our trial runs. d In the trial runs, ifve weighting schemes (tf-idf, log-ent, norm, binary, relative frequency) [ 54] were used for the experiments along with diferent machine learning classifiers such Decision Tree, Random Forest, Logistic Regression, AdaBoost, SVM, and Naive Bayes. We tried diferent -grams, = {1, ..., 7} . We noticed that the classifiers started to obtain insignificant results when = 5 or higher. Finally, the Decision Tree algorithm outperformed other classifiers in identifying fake news. The baseline code is publicaly available 9.
9https://github.com/MaazAmjad/Urdu-Fake-news-detection-FIRE2021
This section briefly overviews the methods applied in the competition by the teams. In total 34 teams registered for the competition, and 18 teams submitted experimental results on a test dataset. We report the findings of 11 teams who submitted their methodologies in the form of technical report papers. The registered participants were from the countries where Urdu language has presence or cause interest: Pakistan, India, United Arab Emirates, Israel, and Egypt. Table 2 shows the approaches used by the teams and table 3 tells the best run scores achieved through those methods. and TF-IDF of words as well as character n-grams as features. Similar to team MUCIC, an ensemble of ML classifiers (RF, MLP, AdaBoost, and GraidentBoost) were used with soft voting to achieve the highest F1 macro. 10. Iqra Ameer: This is another study that used BERT-base model. The best results were reported using both the training and validation set for training of the model. 11. Sakshi Kalra: The best team runs used an ensemble of various transformer methods (RoBERTa, XLM-RoBERTa and Multilingual BERT) as well as a single specialized transformer RoBERTa-urdu-small. The text input was normalized. Interestingly, the best performing method on the test set turned out to be RoBERTa-urdu-small which exceeded the three-transformer ensemble method (XLM-RoBERTa+Multilingual BERT+RoBERta).
Each team submitted three runs (proposed three diferent systems), and only the best run was considered for comparison. We calculated the results of all the submitted runs by each teams individually and only reported the results obtained by the best run. Table 3 shows the the results of the best run (among up to three submitted runs) submitted by the participating teams. We used F1-macro score to rank the participants systems. The aggregated statistics about the performance is presented in Table 4.
It can be observed that only two systems outperformed the baseline, and all the other systems did not beat the F1-macro score of the baseline. The team Nayel obtained the the best results in terms of F1-macro, Accuracy, Pfake (precision) scores. The team Abdullah-Khurem obtained the the second best results in terms of F1-macro, Accuracy, Rfake (recall), Preal (precision), and F1fake scores. Moreover, the baseline approach with the combination of char-word-function words bi-gram with tf-idf weighting scheme using Decision Tree classifier the third position in the shared task with the diference of 2.8% from Nayel system and 1.2% from Abdullah-Khurem system in F1-macro score.
Table 3 presents the best results of the submitted systems.
Table 4 presents aggregated statistics of the submitted systems. 10. Conclusion Automatic fake news detection is an important task, especially in low resource languages. This research presents the second shared task (the first task was organized in 2020) in identifying fake news in Urdu namely the UrduFake 2021 track at FIRE 2021. A training and testing dataset was presented so that the participants could train and test their proposed systems. The dataset contained news in five domains (business, health, sports, showbiz, and technology). All the real news were crawled from credible sources and manually annotated while the fake news were written by the professional journalists.
In this shared task, thirty four teams from seven diferent countries registered and eighteen teams submitted their proposed systems (runs). The participants used diferent techniques ranging from the traditional feature-crafting and application of traditional ML algorithms to word representation through pre-trained embeddings to contextual representation and end-toend neural network based methods. The approaches used included ensemble methods, CNN, and non-Urdu specialized Transformers (BERT, RoBERTa) as well as Urdu-specialized (MuRIL, RoBERTa-urdu-small) .
Team Nayel outperformed all the proposed systems by using the linear SVM optimized with Stochastic Gradient Descent and obtained F1-macro score of 0.67. This result reveals that classical feature-based models perform better compared to the contextual representation and large neural network algorithms. The characteristics of the dataset require further investigation to better explain this observation.
This shared task aims to attract and encourage researchers working in diferent NLP domains to address the automatic fake news detection task and help to mitigate the proliferation of fake content on the web. Moreover, this also ofers a unique opportunity to explore the suficiency of textual content modality alone and efectiveness of fusion methods. In addition, an annotated news dataset in Urdu is also provided to encourage more research to address the automatic fake news detection in Urdu language.
This competition was organized with the support from the Mexican Government through the grant A1-S- 47854 of the CONACYT, Mexico and grants 20211784, 20211884, and 20211178 of the Secretaría de Investigación y Posgrado of the Instituto Politécnico Nacional, Mexico. [54] M. Amjad, G. Sidorov, A. Zhila, H. Gomez-Adorno, I. Voronkov, A. Gelbukh, Bend the Truth: A benchmark dataset for fake news detection in Urdu and its evaluation, Journal of Intelligent & Fuzzy Systems 39 (2020) 2457–2469. doi:1 0 . 3 2 3 3 / J I F S - 1 7 9 9 0 5 . [55] A. Kumar, S. Singh, G. Kaur, Fake news detection of indian and united states election data using machine learning algorithm (2019). [56] L. Derczynski, K. Bontcheva, M. Liakata, R. Procter, G. Wong Sak Hoi, A. Zubiaga, SemEval2017 task 8: RumourEval: Determining rumour veracity and support for rumours, in: Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), Association for Computational Linguistics, Vancouver, Canada, 2017, pp. 69–76. URL: https://www.aclweb.org/anthology/S17-2006. doi:1 0 . 1 8 6 5 3 / v 1 / S 1 7 - 2 0 0 6 . [57] G. Gorrell, E. Kochkina, M. Liakata, A. Aker, A. Zubiaga, K. Bontcheva, L. Derczynski, SemEval-2019 task 7: RumourEval, determining rumour veracity and support for rumours, in: Proceedings of the 13th International Workshop on Semantic Evaluation, Association for Computational Linguistics, Minneapolis, Minnesota, USA, 2019, pp. 845–854. URL: https://www.aclweb.org/anthology/S19-2147. doi:1 0 . 1 8 6 5 3 / v 1 / S 1 9 - 2 1 4 7 . [58] M. Amjad, G. Sidorov, A. Zhila, H. Gomez-Adorno, I. Voronkov, A. Gelbukh, Bend the Truth: A benchmark dataset for fake news detection in Urdu and its evaluation, Journal of Intelligent & Fuzzy Systems 39 (2020) 2457–2469. doi:1 0 . 3 2 3 3 / J I F S - 1 7 9 9 0 5 .