This overview paper describes the first shared task on fake news detection in Urdu language. The task was posed as a binary classification task, in which the goal is to diferentiate between real and fake news. We provided a dataset divided into 900 annotated news articles for training and 400 news articles for testing. The dataset contained news in five domains: (i) Health, (ii) Sports, (iii) Showbiz, (iv) Technology, and (v) Business. 42 teams from 6 diferent countries (India, China, Egypt, Germany, Pakistan, and the UK) registered for the task. 9 teams submitted their experimental results. The participants used various machine learning methods ranging from feature-based traditional machine learning to neural networks techniques. The best performing system achieved an F-score value of 0.90, showing that the BERT-based approach outperforms other machine learning techniques.
https://nlp.cic.ipn.mx/maazamjad/ (M. Amjad) © 2020 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). CEUR Workshop Proceedings
Automatic fake news identification is dificult, because we deal with very high level semantic
phenomenon and at the first glance fake news look like real news. There are several types of
news that are considered fake news. The researchers [
The term “f a k e n e w s ” is not a simple concept. Publishers have been spreading false and
misleading information even before the availability of the Internet. Diferent studies proposed
various definitions of fake news [
Automatic detection of fake news is crucial to prevent the devastating and havoc impact that the fake news phenomenon causes worldwide. Throughout history, humans have always been prejudiced and intolerant to diferent views. For example, in 1620, Francis Bacon 5 emphasized the consequences of inaccurate language in his book Novum Organum, “The ill and unfit choice of words wonderfully obstructs the understanding.” To this point, fake news detection is a means to eliminate the vast and disastrous efects produced by misinformation. Further, the ever increasing pace and scale of fake news propagation can be mitigated only by automating the solutions and increasing their efectiveness.
Fortunately, fake news detection attracted many researchers, in particular after the US 2016 presidential election. The importance of the task visibly increased for the Natural language processing (NLP) community as well as the demand for the solutions from the industry. Given the topicality of fake news detection and the urgency of coming up with an efective solution, this competition aims to gain the attention of a larger research community and to incentivize development of diferent solutions to combat the propagation of fake news on the Web.
A large number of existing studies in the literature examined automatic fake news detection in multiple languages, such as English, Spanish, German, Chinese, and Arabic. However, a very limited work was done in Urdu language to automatically identify fake news content on the web. Urdu is a widely spoken language having more than 100 million native speakers worldwide6. 3https://www.theguardian.com/books/2019/nov/22/factitious-taradiddle-dictionary-real-history-fake-news 4https://www.theguardian.com/books/2019/jun/19/dominic-raab-disinformation 5https://www.theguardian.com/technology/2016/nov/29/fake-news-echo-chamber-ethics-infosphere-internetdigital
6http://www.bbc.co.uk/voices/multilingual/urdu.shtml However, according to the best of our knowledge, there are no automatic web sources to verify the authenticity of news articles in Urdu language. This situation requires the attention of researchers working in NLP to develop tools and solutions in verifying the authenticity of news articles written in Urdu language. Note that Urdu is a low resource language, i.e., it does not have many NLP tools and corpora data.
Urdu is spoken mainly in Pakistan. The fake news phenomenon had bad efects in Pakistan’s social, politics and economical situation. For example, a Pakistani TV anchor Dr. Shahid Masood7, was sent to jail and barred from hosting the TV show due to deceptive claims and spreading fake news on the rape case of a teenage girl during a television show. Similarly, according to the Washington Post8, fake news about child traficking led to many deaths of innocent people in India.
In addition to this, BBC reported that some Indian sites claimed a civil war9 had broken out in one of the cities of Pakistan. The report mentioned that some Indian websites described the situation in Pakistan as dangerous and the civil war resulted in the deaths of many city police oficers. Moreover, the websites claimed that tanks had been seen on the streets, which eventually proved to be fake news. Therefore, this urge to conduct studies to combat the dissemination of fake content in Urdu language.
A number of approaches for automatic fake news detection were proposed. These approaches
are based on statistical text analysis to tackle fake news detection. Previous studies used various
datasets, which comprised mainstream media news articles and news published on social media.
Notably, the majority of work focused on English [
Correspondingly, challenges10 and shared tasks on automatic fake news detection were
proposed, such as SemEval 2017 task 8 RumourEval for English [
Several studies [
The task of fake news detection is to solve a binary classification problem, in which the input is
a news article and the output is the assigned label (real or fake). Built around the idea reported
in recent studies that the textual content can be helpful to identify fake news [
The task was made publically available11 on June 30, 2020, and the same day the training dataset was released as well. The training dataset was splitted into two subfolders, (i) real news subfolder, and (ii) fake news subfolder. The real news subfolder contained 500 real news articles, and the fake news subfolder contained 400 fake news articles. We released the test dataset on August 31, 2020. Like for the training dataset, the testing dataset was also splitted into two subfolders, (i) real news, and (ii) fake news. The real news subfolder contained 250 real news articles, and the fake news subfolder contained 150 fake news articles. The participating teams submitted their system until September 10, 2020. Each team could submit up to 3 diferent runs.
This section provides an overview of the dataset developed for the shared task. A smaller
version of the dataset, named “Bend The Truth”, along with the detailed information about the
collection and annotation description was presented in the recent study [
The fake news articles were intentionally written by hired professional journalists under
specific instructions. The domains of the news present in our dataset are: (i) Business, (ii)
Health, (iii) Showbiz (entertainment), (iv) Sports, and (v) Technology. They are similar to the
dataset [
For the training and development purposes, we ofered 900 news articles from the previously published “Bend The Truth” dataset. The 400 news articles from the previously unseen and unpublished newly collected part were held out as a test dataset.
For the dataset annotation, rigorous guidelines and annotation procedures were defined. The news articles were annotated into two types of news: (i) real news article, and (ii) fake news 11https://www.urdufake2020.cicling.org/home 12https://github.com/UrduFake/urdufake2020eval.git article. The dataset can be used as a corpus for supervised machine learning. It is possible to use the knowledge of the dataset annotation procedure for applying to the underlying characteristics of fake news in addition to linguistics features, but we do not recommend it, because in real life there is no such information. We followed diferent strategies for real and fake news annotation.
For the annotation of real news articles, initially numerous news articles from diferent mainstream were collected. Table 1 shows news agencies used to collect news articles for annotation. To annotate a news article as a real news, the major points in the real news data collection and annotation were: 1. The data collection and annotation procedure were performed manually. 2. The news article was labeled as real news if the news meets the following criteria: • A reliable newspaper or a prominent news agency published that news article. • Other authentic and credible newspaper agencies published the same news article and the veracity of the news article can be easily verified using information such as place of the event, image, date, etc. We also performed manual source verification from where the news are originated. We further compared and cross checked diferent sources (mainstream news agencies) to verify the information present in the news article. • We also confirmed that a news article has a correlation between its title and its content. We read the complete news articles to find out the correlation between the title and the content.
If a news article does not follow one of these criteria, we simply discard that news article.
Note that the length of all the news articles is heterogeneous. The reason is that each news agency has a diferent style of news articles. For example, BBC Urdu contained on average more than 1,500 words in a news article. Thus, we selected real news articles carefully following the described procedure.
To obtain fake news in this dataset, we used professional journalist services from various news agencies in Pakistan: Express news, Dawn news, etc., who were asked to write fake news stories that correspond to the original real news articles. This is a peculiar attribute of this dataset, because it ensured that the fake news articles realistically imitated real life approach to fake news creation. In real life, it is journalists who are responsible for writing fake news articles. Obviously, not all journalists do it. Still, people of this profession have a better understanding of how to write an article (real or fake) and make it interesting to hook and, in case of fake news, trick the reader.
The reasons to use professional “crowdsourcing” to collect fake news are the following: 1. Finding and verifying the falsehood of fake news in the same domain as the available real news articles is a challenging task that requires a huge amount of time and resources unavailable to a small group of organizers. Thus, manual analysis of hundreds of thousands of news articles for verification through web scraping approach was unfeasible. 2. Unlike the case of the English language, most of the news verification in Urdu language is done manually due to the absence of web services that ofer news validation. We should mention that the news articles style and language characteristics vary depending on the news domain. Our dataset contains news in five major domains: sports, business, education, technology, and showbiz. Thus, we assigned news articles according to the journalists expertise in the corresponding domain. Moreover, all the journalists were given instructions to minimize the possibilities of introducing defined patterns that can provide undesirable clues in the classification task. Also, some technical guidelines, such as the requirement that the lengths of fake news should be in the range of those of the original news, were provided. Finally, all fake news articles were prepared using journalists’ expertise.
The training set was made available to the participants to develop their approaches to identify fake news. It contained 900 news articles, annotated in a binary manner as real or fake. 500 news articles were annotated as real, and 400 articles were annotated as fake. The real news part of the dataset was retrieved from January 2018 to December 2018, which is diferent from the test set.
All five topic domains, i.e., (i) Business, (ii) Health, (iii) Showbiz (entertainment), (iv) Sports, and (v) Technology were present in the training dataset. That is, we did not hold out any domain from the training set to make it “unseen” for the participants.
The use of the training set for validation, development, and parameter tuning purposes was at the participants’ discretion.
The test dataset was used to evaluate the performance of the submitted classifiers. It was provided to all the participants without the ground truth labels. The truth labels were only used by the organizers to evaluate and compare the performance of participants’ approaches.
To create the testing dataset, news articles were retrieved from January 2019 to June 2020. It also has all five types of news as the training set. The test dataset is composed of 400 news articles. The ground truth distribution among these 400 news articles was 250 real news articles and 150 fake news articles. We emphasize again that this information, along with labels, was not made available to the participants.
To prepare the data for the experiments, the corpus was split into train and test sets. In the first stage of the shared task, the training dataset was released which contained 900 news samples (500 real and 400 fake). In the second stage, we released the test dataset which contained 400 news articles (250 real and 150 fake). Table 2 describes the corpus distribution of the news articles by topics for the training and testing sets.
The task consists in classifying a news article as fake or real news. First, the training dataset was released for the participants to develop and train their systems, and subsequently, the test dataset was released. Each participant team had an option to submit only 3 diferent runs. The participants’ submissions were evaluated by comparing the labels predicted by the participants’ classifiers and the ground truth labels. To quantify the classification performance, we employed the commonly used evaluation metrics: Precision (P), Recall (R), Accuracy, and two F1-scores, namely, F1real for the prediction of label “real” and F1fake for the prediction of label “fake” out of all news. Additionally, to accommodate the skew towards the real class, which dominates (it has more samples than the fake news class), we used the macro-averaged F1-macro which is the average of F1real and F1fake.
As this is a binary classification problem and the dataset contains two equally important classes, we measured both of them, evaluating the quality of predicting whether a news article is real, i.e., treating the “real” label as a positive, or target, class, and the quality of predicting whether the news article is fake, i.e., considering the label “fake” as a positive class. It is important to mention that since the dataset is not balanced, this is why these metrics are used.
The final ranking is based on the F1-macro score. It can be observed that F1-macro and accuracy are correlated, as it is expected.
We provided three baseline systems with the goal that their performance could serve as reference
points for qualitative evaluation of the submissions’ placement in the ranking. First, we provided
the Random Baseline as the most basic and trivial baseline, which is expected to be ranked at
the bottom with a more massive gap from the participating systems. Second, we provided the
most traditional baseline: bag of words (BoW) model. It uses words as features and then apply a
machine learning classifier. In this baseline, we used binary weighting scheme (i.e., a feature is
present or not) with Logistic Regression classifier. For the third baseline, we provided the results
of character bi-gram with tf-idf weighting scheme using Logistic Regression classifier, which
achieved surprisingly good results. Overall, we tried five weighting schemes (tf-idf, logent,
norm, binary, relative frequency) [
This section gives a brief overview of the systems submitted to this competition. 42 teams registered for participation, 9 teams submitted their runs. Registered participants were from 6 diferent countries (India, Pakistan, China, Egypt, Germany, and the UK). This wide range of the regions where the interested participants were located confirms the importance of this task. The team members came from various types of organizations: universities, research centers, and industry.
As the initial step of the experimental setup, the majority of participating teams performed data cleaning and preprocessing such as stop words elimination. In particular, the system submitted by the team MUCS removed the stopwords, whereas the systems submitted by teams BERT 4EVER, Chanchal_Suman, CNLP-NITS, SSNCSE_NLP and NITP_AI_NLP decided to leave them.
Further, all the news articles were represented with diferent text representation techniques. The team MUCS used traditional bags of words representation. Similarly, the teams MUCS, NITP_AI_NLP, and SSNCSE_NLP used the n-gram (words or characters) representation weighted with tf-idf. The teams BERT 4EVER, CNLP-NITS, and Chanchal_Suman represented news article texts using word embeddings. In particular, only one team, SSNCSE_NLP, represented texts using Word2Vec embeddings, while the team SSNCSE_NLP employed FastText embeddings. Furthermore, the team Chanchal_Suman used Urdu word embedding and only one team, BERT 4EVER, used the contextual representation using BERT [21], which is one of the most recent and advanced manners of text representation.
To implement their classifiers, some participating teams used the traditional, i.e., non-neural machine learning algorithms, while some teams submissions were based on various neural network architectures. The team MUCS used two classical machine learning classifiers such as Multinomial Naive Bayes and Logistic Regression with default parameters, and the same team also used LSTM in the experimental setup. Similarly, the team SSNCSE_NLP used machine learning classifiers such as Multi-Layer Perceptron (MLP), AdaBoost (AB), ExtraTrees (ET), Random Forest (RF), Support Vector Machine (SVM), and Gradient Boosting (GB). Another team, NITP_AI_NLP, used ensemble models by combining Random Forest, Decision Tree and AdaBoost classifiers. The team NITP_AI_NLP also used a multi-layer dense neural network. In contrast, one team, Chanchal_Suman, used Gated Recurrent Unit (GRU). All the participating teams, except one team (NITP_AI_NLP), used Transformers. Description of the approaches is presented in Table 3.
Among all the submitted runs, the results of the best run (among up to three submitted runs) are presented in Table 4. The systems are ranked by the F1-macro score. Table 5 provides the aggregated statistics about the performance of all non-trivial systems, including the baselines, that is, all of the systems apart from the random baseline.
We observe that except one system, all the other participating teams’ systems outperformed the random baseline in terms of F1-macro score. The BERT 4EVER system achieved the best F1-macro, Accuracy, as well as Rfake (recall), F1fake, and Preal (precision) scores. However, the baseline approach with character bi-grams and Logistic Regression achieved the second position in the shared task with just 1.1% diference in F1-macro from BERT 4EVER, which is quite an unexpected result. The explanation of this fact is a question for further research.
Table 4 presents the best results of the submitted systems.
Over 50% of the systems obtained F1-macro and Accuracy of more than 0.8 (Table 4), which is a reasonably high result. In Table 3, we observe that most of these high performing systems achieve better Recall for fake news, and better Precision for the real news detection. This tells us that these systems tend to “mistrust” the news articles and tag more news as fake than there are in reality. This can be considered as an overly secure approach.
At this moment, it is hard to judge whether any of these approaches is ready to be applied “in the wild”. While the results of F1real and F1fake over 0.9 shown by the winning BERT 4EVER system are impressively high, the modest size of the provided training and testing datasets cannot guarantee the same performance on an arbitrary text input. To ensure the scalability of the presented approaches, more multifaceted research at a larger scale is needed. We see that one of the paths is a community-driven efort towards the increase of available resources and datasets in the Urdu language.
Table 5 presents statistics of the submitted systems.
This paper describes the first competition on automatic fake news detection in Urdu, the UrduFake 2020 track at FIRE 2020. We provided the training and testing parts of the dataset that included news articles in five domains (business, health, sports, showbiz, and technology). The news articles in the dataset were manually annotated with labels “fake” and “real” with a slightly imbalanced ratio of approximately 60% real news and 40% fake news.
Forty two teams from six diferent countries registered for this task. Nine teams submitted their experimental results (runs). The approaches employed by the submitted systems varied from the traditional feature-crafting and application of traditional ML algorithms to word representation through pre-trained embeddings to contextual representation and end-to-end neural network based methods. In particular, ensemble methods were used in the traditional ML case. LSTM, and Transformers (BERT) were used in neural network based solutions.
Among all the submissions, only the best submitted model, BERT 4EVER, outperformed the character bi-grams with Logistic Regression baseline achieving F1-macro score of 0.90. This confirms that contextual representation and large neural network techniques outperform classical features-based models, which has been shown in many recent studies in all branches of natural language processing.
This competition aimed to encourage researchers working in diferent NLP domains to attempt to tackle the proliferation of fake content on the web. It also provided an opportunity to fully explore the suficiency of textual content modality and efectiveness of fusion methods. And last but not the least, this track provides a useful resource in the form of an annotated dataset for other researchers working in automatic fake news detection in Urdu.
This competition was organized with partial support of National Council for Science and Technology (CONACYT) A1-S-47854, SIP-IPN 20200797 and 20200859 and CICLing conference. The work of the last author was partially funded by MICINN under the research project MISMISFAKEnHATE on MISinformation and MIScommunication in social media: FAKE news and HATE speech (PGC2018-096212-B-C31). Association for Computational Linguistics, Vancouver, Canada, 2017, pp. 69–76. URL: https://www.aclweb.org/anthology/S17-2006. doi:1 0 . 1 8 6 5 3 / v 1 / S 1 7 - 2 0 0 6 . [18] G. Gorrell, E. Kochkina, M. Liakata, A. Aker, A. Zubiaga, K. Bontcheva, L. Derczynski, SemEval-2019 task 7: RumourEval, determining rumour veracity and support for rumours, in: Proceedings of the 13th International Workshop on Semantic Evaluation, Association for Computational Linguistics, Minneapolis, Minnesota, USA, 2019, pp. 845–854. URL: https://www.aclweb.org/anthology/S19-2147. doi:1 0 . 1 8 6 5 3 / v 1 / S 1 9 - 2 1 4 7 . [19] B. Ghanem, P. Rosso, F. Rangel, An emotional analysis of false information in social media and news articles, ACM Transactions on Internet Technology (TOIT) 20 (2020) 1–18. [20] A. Giachanou, P. Rosso, F. Crestani, Leveraging emotional signals for credibility detection, in: Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, 2019, pp. 877–880. [21] J. Devlin, M.-W. Chang, K. Lee, K. Toutanova, Bert: Pre-training of deep bidirectional transformers for language understanding, arXiv preprint arXiv:1810.04805 (2018).