In the present attempt, we have developed a framework to detect the fake news spreaders on twitter by utilizing their tweets. Here, we have employed the pre-trained sentence embedding of Google and fed this embedding to a Long Short Term Memory (LSTM) based deep learning framework. Finally, the embedding is passed through an attention layer and predicts whether an author is prone to spread fake news or not. We have built models for two languages - English and Spanish. We have achieved 72% accuracy in this fake news spreader detection task.
1 Introduction In the present work, we have developed techniques to detect profiles of fake news spreaders. Fake news detection is becoming one of the challenging tasks of recent years. However, in this present task, instead of detecting fake news, we are concentrating on detecting it at user level. Thus, in order to accomplish our goals, we handle tweets at chunk level instead of handling each tweet, because here we have to detect the author of fake news, not the news type.
We participated in the profiling fake news spreader shared task [
The rest of the paper is organized as follows. Related work on this particular topic is discussed in Section 2 whereas Section 3 briefly shows the insights of the datasets. Section 4 describes the method we used to detect the fake author and also describes our models and proposed architecture in depth. Section 5 is dedicated to experiments and results. Finally, in Section 6, we present the conclusions and briefly discuss about future work.
Fake news is currently one of the hottest topics of last four to five years and many researches are being conducted in this field. Some of the researchers suggest solving the fake news detection problem using the content of the news and some suggest detecting it based on the social context. Therefore, we can detect fake news in two ways – a) news content model, and b) social context model. In case of news content model, we can detect fake news in two ways – a) knowledge based detection, and b) style based detection. In knowledge based detection, we generally check the news content or extract the knowledge from the news or other repositories and compare it with the authentic news sites whereas in case of style based detection, we focus on the writing styles instead of the news content or knowledge. Here, we mainly focus on the linguistic features and readability features of the news. In social content model, we can detect fake news also in two ways – a) propagation based techniques, and b) credibility based techniques. In propagation based techniques, we find various news propagated on the social media and track the original news based on such propagated news. In credibility based technique, we have to find the relationship between news article and users, publishers, posts, comments etc.
George et. al. [1] analyse the influence of linguistic properties and contextual
features in detecting fake news by using different type of techniques like Naïve
Bayes, SVM, KNN etc. Perez-Rosas et. al. [2] here cover seven different types of
news domains and analyse linguistic differences in fake and real news and also
compare different domain characteristics. Bedi et. al. [4] use authorized news
database to verify fake news and real news. Dey et. al. [5] do feature extraction and
analyse the linguistic patterns and then apply KNN algorithm to classify news. Uppal
et. al. [
However, one problem with these above mentioned techniques is that they
detect the fake news after it spreads on social media. But, if we can detect it from the
source, this problem can be avoided. So, one of our aims should be to detect the
author of the fake news. Already, there are some works on fake news spreader
detection [
We participated in the profiling fake news spreader shared task of the PAN workshop at the CLEF (Conference and Labs of the Evaluation Forum) 2020 conference. In this task, the dataset was provided by the shared task organizers. We are given with hundred tweets of each author and a total of 300 authors tweets have been given to us for training. For each English and Spanish language, we are given 300 authors tweet (100 tweets for each author) to train the model. For evaluation, we have to submit our software to the TIRA infrastructure and after that we have to execute it on the test dataset. We have given snapshots of some tweets of both English and Spanish language from my training set.
Spanish tweets of an author
English tweets of an author 4
Pre-processing: Here, we first tried to conduct some text pre-processing techniques on the tweets by removing everything except letters from ‘a’- ‘z’ and ‘A’ – ‘Z’. We have also removed all the urls and html tags or elements present in the tweets. Finally, we removed all types of emoji or emoticons present in the tweets. We send this preprocessed data to a universal-sentence-encoder. But, we observed that the universalsentence-encoder performs better if we send those tweets in raw form rather than preprocessed form. Thus, we choose to use those tweets without pre-processing. LSTM Framework: Here, we have implemented embedding of our tweets and then passed it through the LSTM layer and then attention layer and predicted the output lastly through the sigmoid activation function. Here, we have used sentence embedding, instead of word embedding. We have employed universal-sentenceencoder-xling_en_es_1 (This is a cross lingual module and an extension of the normal universal sentence encoder). We also tried with universal-sentence-encodermultilingual-large-v3, but as this shared task is specifically for English and Spanish language, we choose universal-sentence-encoder-xling_en_es_1 (which is specifically trained for English and Spanish language) which can handle 16 languages including English and Spanish. We named it as Universal Sentence Encoder or U.S.E., interchangeably.
One feature of this universal sentence encoder is that it always gives output of a 512 dimension vector whatever be the input is. Here, we are given with the tweets of an author as the input to the U.S.E. and it produces 512 dimension vector. Thus, for each of the authors, we are getting an output of (100 x 512) dimensional vector output, as we have 100 tweets of each author. As we have 300 such authors in both English and Spanish languages for training our model, we developed a LSTM network with 128 units. The overall framework takes 512 features at a time and contains 100 such timestamps. In order to avoid over fitting, we have also used dropout of 0.3 and recurrent dropout of 0.2. We tried with different hyper-parameters but these two gives the best result. We also tried with different batch-sizes such as 16, 32 and 64 but we choose 64 among these.
In the present framework, we tried to train both the languages differently and then used to train both the language jointly. At first, we used any of these two sentence embedding – 1) U.S.E. multilingual large-v3 (version 3) and 2) U.S.E. xling_en_es-1. After that, we feed those outputs to the LSTM network. Because, we aimed to capture long term dependency through LSTM. However, in some cases, it is forgetful and does not know which input should be given more or less importance.
For example (from twitter) - “Breaking News: Tom Hanks and his wife Rita
Wilson announced Wednesday that they have tested positive for the coronavirus”.
Here, to predict the word “coronavirus”, we should give more emphasis to “tested”
and “positive” instead of “Tom Hanks” or “Rita Wilson”. So, here we have to give
relative importance to each of the words instead of giving them same importance.
Therefore, we have applied the attention model which is presented in the Bahdanu’s
paper [
On the other hand, if we do average of the above two validation accuracies, we achieved around 71% accuracy. Thus, finally, we choose universal sentence encoder xling-en_es-1 trained on both the languages with batch size of 64 over the other models as its validation accuracy is 81.50%. When we submit our final model which uses Universal Sentence Encoder xling-en_es-1 and trained on both the languages jointly with the batch size of 64 to TIRA for evaluation on hidden test set, we get an accuracy of 64% in English language and 80% accuracy on Spanish language, respectively.
We achieved 64% accuracy in English language fake author detection task, which says that there is a scope of lot to improve and there is some issue of overfitting too though we used dropout and recurrent dropout at the time of training the model. Thus, in future, we should take this note to improve this model. If we want to use this software in real life then how will it perform much depends on how the test dataset of shared task of PAN reflects the real world dataset. We achieve 80% accuracy in Spanish language author detection task, which is quite satisfactory, but again if we want to implement this software in real world, then its performance totally depends on how much test dataset of PAN shared task reflects the real world dataset. We can also use different types of embedding in future for this task like BERT, but main disadvantage of BERT is that it can take maximum 512 words at a time, so there is a constraint. However, we will plan some reliable frameworks to handle the issues.