Fake news is affecting our lives since the internet has become popular. Particularly, in this era of social media it is very easy to spread and be affected by fake news. In this work we have developed machine learning models which can classify a news claim into four classes. This work has been done under the competition of CheckThat!2021 task-3a. We have conducted our experiment on Check that lab's dataset. Our work has been done only on linguistic features. We have experimented both with traditional Machine Learning algorithms and Deep Learning algorithms. LSTM outperformed other traditional machine learning algorithms and with Adam optimizer LSTM gave a f1-macro score of 26.07%.
The internet has become one of the biggest part of our life. We are using it every day in our day to day life. The usage of the internet and number of users are being increased every day. According to datareportal.com [16] the number of internet user by April, 2021 is 4.72 billion (which is 60% of the world’s population). Every day we read news articles, blogs and news content in various forms (e.g., images and videos). So it is easy to get distracted and deceived by any sort false representation of an event or event which does not exist but depicted with a verified style. Fake news is affecting our life and bringing damage to so many people. In 2016 US presidential Election fake news played a vital role. Alexandre Bovet et. al. 2019 [22] showed 25% of the news in the time of US presidential election was biased. In [15], data shows that the number of blog posts appear only in WordPress is 70 million each month. These findings only show that the number of potential fake news are not so little. And we can easily be misled by those false news. During the COVID-19 pandemic we have seen so many fake news spread in different communities all around the world. In India several fake news spread during this pandemic which created confusion about COVID-19 in the community. In [17] summarized the COVID-19 related fake news in India. Another fake news spread in India which claim the people who are taking COVID-19 vaccine may die within two years [18]. In Bangladesh several fake news created a huge confusion among students during this pandemic. Several fake news claimed the higher secondary examination will be taken place soon in May-June, 2020. In 2019, A mother was killed in Bangladesh as a result of a fake news which claims children’s head are being used in the construction of Padma Bridge but later investigation did not find any clue of this claim and they also found that the mother was innocent [20]. In October, 2020 a fake news claimed that France footballer Paul Pogba left France National Football team [19]. Several studies have been done to classify and categorize fake news. B Bharali et. al. 2017 [21] addressed six categories of fake news: “1. Disinformation, 2. Propaganda 3.
2021 Copyright for this paper by its authors. Hoaxes 4. Satire/Parody 5. Inaccurate 6. Partisanship”.
To detect fake news automatically we need to use machine learning techniques because the amount of articles appearing on the internet every single day is impossible to classify with traditional approaches. The sub-domain of machine learning which deals with text classification is called Natural Language Processing. Using NLP techniques, we can detect whether a given news article is fake or real or some further classification. Researchers have done a lot of work in this field. Some have worked only with text and some others have worked with additional entities like news sources and user opinions. After analyzing those previous work, we have come up with two research questions to conduct our study.
RQ1: Can we identify the impact factor in linguistic analysis based fake news detection? RQ2: Between traditional machine learning and deep learning which one gives better performance? The rest of the paper is structured as follows: In the literature review section we have discussed how others have conducted their research and which approach they have found as best performer. Next, the research methodology section will describe step by step how we have conducted our experiment. And then in the result analysis section we will compare the result of different methodologies we have used. The rest three sections are conclusion, acknowledgement and reference.
Fake news detection got wide attention in the machine learning research community after the US
election in 2016. T S Reshmi et. al. 2021 [
There are various types of research methodologies in Natural Language Processing for fake news
detection. To detect fake news, researchers usually follow some particular methodologies. Some use
only a content and context based approach, some have taken into account the user opinion on social
media on the same news from various users and some other researchers considered the source of the
news [
As this work has been done under CheckThat!2021, they have provided the dataset. The dataset contains four columns: public_id, title, text and our rating. ‘our rating’ basically contains the classes we will classify. A snippet of the dataset has been given below. Our rating contains four classes: False, Partially false, True and Other. A distribution of these classes in the figure below: When it comes to news content based fake news detection, some researchers considered only main news content, not the title and some researchers took account of both title and main news content. In our research we have applied both approaches and shown the performance analysis in the result analysis section.
In the preprocessing step, first, we have removed number and punctuation from our dataset. Then we removed one and two-character length words. We used python regular expressions for this task. Machine Learning algorithms consider “Word” and “word” as separate entities and this is a problem in experiment because they both have the same meaning. To avoid this problem, we have converted the whole dataset into lowercase character. After doing these steps we have removed stop words. We have used the nltk library to remove stopwords. After then we did word level tokenization on both title and text using nltk.word_tokenize. For stemming we have used Portstemmer() and for lemmatizaiton we used WordNetLemmatizer(). After completing the preprocessing steps we have started feature extraction which we will discuss in the next section.
Feature extraction is mandatory for machine learning tasks. It helps to reduce the training time by reducing dimensionality. In the feature extraction phase we have used two feature extraction techniques which are common in natural language processing. We used TF-IDF and CountVectorizer in our work. Term Frequency-Inverse Document Frequency (TF-IDF) is a statistical measure commonly used in natural language processing which evaluates how relevant a word is to a document in a collection of documents. This is done by multiplying two metrics: how many times a word appears in a document, and the inverse document frequency of the word across a set of documents [23]. CountVectorizer is used to transform a given text into a vector on the basis of the frequency (count) of each word that occurs in the entire text.
In this research we have followed a research model which has been presented below. After getting the dataset the first thing we did is preprocessing it and then we have extracted features using some common feature extraction technique and then used some traditional machine learning algorithm and a deep learning algorithm (LSTM) to classify the news into one of the four classes given in the ‘our rating’.
In our work we have experimented with both traditional ML algorithms and deep learning algorithms. We have conducted our experiment in two different ways. In the first approach, we have conducted the experiment by concatenating both news title and main news content. And in the second approach we have experimented only with news content without taking the news title into account. For both approach we took 80% for training and 20% for testing. In the first approach, by using count vectorizer and tf-idf transformer in the pipeline we got a f1-macro score of 35% with logistic regression. And by limiting the maximum feature to 1000, we were able to increase the performance by 5%. Support Vector Machine with linear carnal gave 38% f1-macro score. In the first approach, among all traditional machine learning algorithms which we have experimented, Multinomial Naïve Bayes classifier gave the best f1-macro score (43%) on the training dataset.
In second approach, Random Forest Classifier and XGBClassifier performed better than the first approach and the rest four algorithm did not do well than the first approach. A performance comparison has been given below.
F1-macro score (One) We have got our best performing algorithm with deep learning. We applied our second approach with LSTM and it performed way better than traditional ML algorithms, with softmax activation function and adam optimizer we got a validation accuracy of 99%. But we have found our model was over fitted after the CheckThat!2021 result publication and the performance was very poor (26.07% f1-macro score). We are working on our LSTM based model to overcome this problem. For task-3a, the best score was 83.76% and the least score was 13.47%.
In this work we tried to build a machine learning model to classify fake news. We have experimented with both machine learning and deep learning based models. Our Machine Learning based model was outperformed by deep learning based model on training data. That’s why we submitted it to the completion but for the overfitting problem it did not perform well on test data. We are working on our best performing deep learning model to make it better and improve the performance.
This is unbelievable support we’ve got from some of the faculty members and seniors of DIU NLP and ML Research Lab to continue our whole research flow from the beginning. We acknowledge Dr. Firoj Alam for guiding us and informing the secretes of research workshops. Also, we’re thankful to Daffodil International University for the workplace support and the academic collaboration in some cases. Dr. Touhid Bhuiyan and Dr. Sheak Rashed Haider Noori also supported us with guidance, motivation, and advocating in institutional supports. Lastly, for sure we’re thankful to our God always for every fruitful work with our given knowledge.
Khan, Junaed Younus, et al. "A benchmark study on machine learning methods for fake news detection." arXiv preprint arXiv:1905.04749 (2019).
Roy, Arjun, et al. "A deep ensemble framework for fake news detection and classification." arXiv preprint arXiv:1811.04670 (2018).
Shahi, Gautam Kishore. "AMUSED: An Annotation Framework of Multi-modal Social Media Data." arXiv preprint arXiv:2010.00502 (2020).
Shahi, Gautam Kishore, Anne Dirkson, and Tim A. Majchrzak. "An exploratory study of covid-19 misinformation on twitter." Online Social Networks and Media 22 (2021): 100104.
https://hostingtribunal.com/blog/blog-posts-per-day/#gref https://datareportal.com/global-digital-overview https://www.dw.com/en/india-covid-misinformation/a-57414876 https://www.hindustantimes.com/india-news/will-you-die-within-2-yrs-after-takingvaccine-centre-busts-fake-news-post-101621961181280.html
https://www.thedailystar.net/sports/football/news/fake-unacceptable-news-pogbaslams-rumors-him-quitting-france-1984513
https://bdnews24.com/bangladesh/2019/07/24/no-one-listened-to-what-she-saidwitnesses-recount-lynching-of-a-mother-in-bangladesh
Bharali, Bharati, and Anupa Lahkar Goswami. "Fake news: Credibility, cultivation syndrome and the new age media." Media Watch 9.1 (2017): 118-130.
Bovet, Alexandre, and Hernán A. Makse. "Influence of fake news in Twitter during the 2016 US presidential election." Nature communications 10.1 (2019): 1-14. https://monkeylearn.com/blog/what-is-tf-idf/