Awakened at CheckThat! 2022: Fake News Detection using BiLSTM and Sentence Transformer

Abstract

In recent years, online social networks and online news venues have become some of the main news and event-related information spreading mediums. Although using these mediums has facilitated the speed of accessing information, it also created a new phenomenon used for propaganda and disinformation: fake news. As fake news has detrimental consequences to society, new technologies need to be developed in order to stop their harmful efects. In this paper, we propose two Bidirectional Long Short-Term Memory (BiLSTM) architectures with sentence transformers to solve two tasks: (1) a multi-class mono-lingual fake news detection task (i.e., mono-lingual task); and (2) a multi-class cross-lingual fake news detection task (i.e., cross-lingual task). For the mono-lingual task, we train and test a BiLSTM with BART sentence transformers model on an English dataset and obtain an accuracy of ∼ 0.53 and an F1-Score of ∼ 0.32. For the cross-lingual task, we train a BiLSTM with XLM sentence transformers model on an English dataset and test the model using transfer learning on a German dataset. For this task, we obtain an accuracy of ∼ 0.28 and an F1-Score of ∼ 0.19.

Keywords

• eol>Fake News Detection
• Neural Networks
• Sentence Transformers
• Transfer Learning

1. Introduction

With the digital age, new mass media paradigms for information distribution have been adopted by the general public. The current paradigms have shifted from the journalistic rigorous imposed by editors to personalized social media where anyone can spread event related news. This new approach aggravates the risk of fake news [ 1, 2 ], which has detrimental consequences to society by facilitating the spread of misinformation in the form of fake news, propaganda, conspiracy theories, political bias, etc. The practices of spreading misinformation online by malicious actors need to be tackled from diferent points of view, i.e., from a journalistic and fact-checking perspective to a more technological-based one.

In this paper, we address the problem of detecting fake news from a technological perspective by using the CheckThat! 2022: Fake News Detection Challenge datasets. To tackle the problem we propose two neural network with sentence transformer models for (1) multi-class monolingual fake news detection; and (2) multi-class cross-lingual fake news detection. We use transfer learning in order to train a model on an English dataset and test it on German text.

In this work, we aim to answer the following two research questions: (1) Does a simple neural network with sentence transformers ofer good results for multi-class mono-lingual fake news detection? (2) Can cross-lingual sentence transforms be used through transfer learning in multi-class cross-lingual fake news detection?

To answer question (1), we propose the use of a Bidirectional Long Short-Term Memory (BiLSTM) neural network with BART sentence transformers. While to answer question (2), we train a BiLSTM neural network with XLM sentence transformers on English textual data and use transfer learning to solve a cross-lingual fake news detection task by testing the model on German textual data.

This paper is structured as follows. In Section 2, we discuss some of the current literature on fake news detection. In Section 3, we present our approach for mono-lingual and cross-lingual fake news detection. In Section 4, we present the datasets, experimental setup, and results. Finally, in Section 5, we summarize our findings and hint at future work.

2. Related Work

The task of fake news detection has been examined from various perspectives and using various models from traditional Machine Learning to more elaborate yet more powerful Neural Network based models. Extensive work in the field of fake news detection has led to many solutions focusing on either model or data-driven approaches. Among the traditional Machine Learning models (e.g., Support Vector Machine, Logistic Regression, Decision Trees, AdaBoost, Naïve Bayes), the model that performs very good in many cases is Multinomial Naïve Bayes [ 3 ].

Several current solutions use complex Deep Neural Network architectures for this task. Many solutions for multi-class mono-lingual fake news detection show promising results when using Convolutional Neural Network (CNN) based architectures. FNDNet [ 4 ] is such an architecture that obtains good results in comparison even with recurrent networks, i.e., LSTM. OPCNNFAKE [ 5 ] is an optimized CNN based solution that uses a hyperopt optimization technique to adapt the values of parameters for each component layer in order to achieve high performance. Other Deep Learning solutions focus on recurrent networks, e.g., (Bi)GRU, (Bi)GRU, (Bi)LSTM, obtaining the best results when also using attention mechanisms [ 1 ]. BiLSTM based solutions (e.g., Samantaray and Kumar [ 6 ], Trueman et al. [ 7 ]) are very promising as this type of recurrent network is able to capture both past and future information.

In multi-class classification, the employed embedding model is very important. As shown in Ilie et al. [ 1 ], many Deep Learning models have an increase in accuracy when using custom trained word embeddings versus pre-trained ones. Other models use advanced pre-trained transformers instead of the more classical word embeddings. Diferent transformer models can be applied for fake news detection, e.g., BERT (Bidirectional Encoder Representations from Transformers) [ 8 ], RoBERTa (A Robustly Optimized BERT pre-training Approach) [ 9 ], BART (Bidirectional and Autoregressive Transformer) [ 10 ]. As such, MisRoBÆRTa [ 2 ] is a complex architecture that combines BART and RoBERTa for a multi-class classification task. Another solution to the multi-class classification problem is proposed by Liu et al. [ 11 ] that ofers a two-stage BERT-based model.

3. Methodology

In this section, we present the methodology used for fake news detection. For encoding the text, we used two sentence transformer [ 12 ] approaches. (1) For multi-class fake news detection of news articles in English, we use BART (Bidirectional and Auto-Regressive Transformers) [ 10 ] sentence transformers. (2) For cross-lingual news articles, we use XLM (Cross-Lingual Language Model) [ 13 ] sentence transformers. We employed a BiLSTM (Bidirectional Long Short-Terms Memory) as the classification model.

3.1. Sentence Transformers

Sentence transformers are a modification of the pre-trained BERT networks that use siamese and triplet network structures to derive semantically meaningful sentence embeddings that can be compared using cosine-similarity [ 12 ]. We construct BART [ 10 ] and XLM [ 13 ] sentence transformers for the mono-lingual and cross-lingual classification tasks, respectively.

XLM [ 13 ] is a Transformer architecture that uses two approaches during pre-training depending on the type of data. For mono-lingual data, it uses an unsupervised modeling technique such as Casual Language Modeling (CLM) or Masked Language Modeling (MLM). For cross-lingual data, XLM employs a supervised modeling technique that combines MLM with Translation Language Modeling (TLM).

BART [ 10 ] is a generalized BERT that uses a transformer-based neural machine translation architecture. The architecture uses a left-to-right decoder (as in GPT [14] architecture) and a standard Sequence-to-Sequence bidirectional encoder (as in BERT [ 8 ]).

3.2. Classification Models

For classification, we propose a deep neural network architecture that contains the following layers: (1) Input layer; (2) BiLSTM layer; and (3) Dense layer.

The input layer instantiates the neural network. It is used to produce a symbolic tensor-like object that has the size of the sentence transformer.

LSTM (Long Short-Term Memory)) [15] is a recurrent neural network that process past information using two state components: (1) a hidden layer for the short-term memory; and (2) an internal cell state for long-term memory. The BiLSTM layer encapsulates both past and future information through the use of two hidden states. The forward hidden state processes the past information using a forward LSTM, while the backwards hidden state process the future information provided by employing a backward LSTM. To encode both the past and future, the BiLSTM concatenate into on hidden state the forward and backward hidden state at every time-step. The number of units for this layer can be determined experimentally using ablation and hyperparameter testing (see [ 2 ] for more details). For the LSTM cell, we use the classic implementation presented in [15, 16]. For this cell, the recurrent activation function is sigmoid, the kernel weights are initialized using the Glorot uniform linear transformation [17], and the bias vector is initialized with zeros.

The Dense Layer is a fully connected Perceptron layer used for classification. The number of units in this layer is equal to the number of classes. The activation function for this layer is the sigmoid.

4. Experimental Results

In this section, we present the experimental results of the proposed models for the CheckThat! 2022 Fake News Detection task for both the mono-lingual and cross-lingual challenges.

4.1. Dataset

The CheckThat! 2022 Task 3 [18, 19] consists of two subtasks as follows: (1) multi-class monolingual fake news detection of news articles (English) [20, 21] (i.e., mono-lingual task); and (2) multi-class cross-lingual fake news detection task (German) (i.e., cross-lingual task). The steps used in the data collection are defined in Shahi [22].

For the mono-lingual task, the English training data is the same as from the CheckThat! 2021 version [23]. The number of classes for this task is four: false, partially false, other, and true. The number of labels has been defined after a thorough study of 83 classes was conducted by fact-checkers [24]. The dataset contains an English training, development, and testing set.

For the cross-lingual task, a new test dataset in German is introduced. The main focus of this task is to use transfer learning to detect fake news content in low resource languages. Thus, the training for this task is done on the English training and development datasets, and then it is tested on the German testing set. For this task, we use the same labels as for the mono-lingual task.

As we used the same training data for both the mono-lingual and cross-lingual tasks, we concatenated the English training and development sets to train the model. Table 1 shows the label distribution for the training dataset. We observe that the dataset is highly imbalanced. Table 2 presents the label distribution for the test dataset.

4.2. Experimental Setup

For the sentence transformer, we used the pre-trained BART (facebook/bart-large) and XLM (sentence-transformers/stsb-xlm-r-multilingual) from HuggingFace Transformer [25]. We train the sentence transformers using the SentenceTransformers Python 3 package [ 12 ].

For classification, the BiLSTM layer uses 100 LSTM units configured as in [ 15]. The dense layer contains 4 units (equal to the number of classes), and the sigmoid function as activation. We used the ADAM optimizer and a 64 batch size. The model is trained for 100 epochs. To prevent overfitting, we used an early stopping mechanism that monitors the Accuracy during training. We use Keras with TensorFlow as backend for implementing the neural model.

The implementation is available online on GitHub at the following url: https://github.com/ elena-apostol/AwakenedCheckThat2022.

4.3. Results

Table 3 presents the overall results for both the mono-lingual (i.e., both train and test sets use English texts) and cross-lingual (i.e., the train set is in English and the test set is in German). We observe that, by training the BiLSTM with BART sentence embeddings on English, we obtain an accuracy of ∼ 0.53 for the mono-lingual task and an accuracy of ∼ 0.28 for the cross-lingual task. With these results, the Awakened team obtained the 3 and the 5ℎ place in the competition for the mono-lingual task and cross-lingual task, respectively. As a general observation, the results are highly influenced by the dataset’s size and class imbalance.

For the mono-lingual task, the low performance of the model is directly impacted by two dataset related aspects: (1) the dataset size is small, being inadequate for a neural network approach; and (2) the dataset is highly imbalanced, miss-classification being a real challenge. When analyzing the evaluation metrics per class (Table 4), these two aspects are even more emphasized by the results obtained per class. For the false label, the model obtains ∼ 0.67 precision and ∼ 0.83 recall. Thus, the interpretation of these results shows that the models manage to correctly determine fake news. For the true label, the model obtains ∼ 0.76 precision and ∼ 0.21 recall. This shows that the model also manages to discriminate well between true news and the other 3 types of texts. Also, these results show that the contextual, semantic, and syntactic information encoded by the sentence transformer for the true and false labels are very specific to this classes. Thus, the textual dissimilarities between fake news and real news are more prominent. The precision and recall for the partially true (∼ 0.13 precision and ∼ 0.32 recall) and other (∼ 0.04 precision and ∼ 0.03 recall) classes are very small. These results indicate that these textual data are more similar to the other two classes. Thus, the model does not manage to discriminate correctly between these two labels and the true and false ones.

For the cross-lingual task, we observe that the model manages to obtain an accuracy of ∼ 0.28 and an F1-Score of ∼ 0.19 (Table 3). These results are also impacted by the transfer learning algorithm besides the dataset’s size and the imbalanced labels. Based on these observations, we can conclude that the multi-lingual sentence transformers do not manage to correctly ifnd similarities between the English and German texts that are labeled with the same class. When analyzing the per class results, we observe that the true labeled German documents are predicted with a high precision (∼ 0.59), but the recall for these labeled documents is ∼ 0.05. The interpretation of these values for the true label is that the model manages to determine the true positives more accurately than false negatives. In other words, for the true labeled documents, the model manages to return more relevant results to this label than irrelevant ones but does not manages to return most of the relevant results for this label. For the false labeled German documents, the interpretation of the results is, as expected, in reverse as for the true labeled documents. Thus, with a precision of ∼ 0.35 and a recall of 0.65, the model manages to return most of the relevant results for this label but does not manage to return all relevant results to this label. The model does not manage to classify any of the German documents in the test set labeled with other and it has a very low F1-Score for the prediction of documents labeled with partially true.

5. Conclusions

In this paper, we trained two BiLSTM neural networks with sentence transformers for data encoding models to detect the veracity of fake news. The first model is trained and tested on an English news article dataset for multi-class mono-lingual fake news detection. This model encodes the textual data using BART sentence transformer. The second model is trained on the same English dataset and tested on a German dataset for multi-class cross-lingual fake news detection. The model encodes the textual data using XLM sentence transformer and takes advantage of transfer learning to solve the task of cross-lingual fake news detection. We use the first model to answer our first research question (1). With an accuracy of ∼ 0.53 and a F1-Score of ∼ 0.32, we conclude that it is worth investigating more the use of simple neural networks with sentence transformers for mono-lingual fake news detection task. The second model is used to answer our second research question (2). With an accuracy of ∼ 0.28 and a F1-Score of ∼ 0.19, we conclude that the BiLSTM XML sentence transform model does not manage to correctly find similarities between the English and German texts. Although, the use of cross-lingual transformers and transfer learning for multi-class classification in theory could prove useful, for the multi-class cross-lingual fake news detection task at head, they perform poorly. For both mono-lingual and cross-lingual tasks, we observed that: (1) the dataset size needs to be large to be adequate for a neural network approach; and (2) the dataset needs a balanced label distribution to mitigate against miss-classification.

In future work, we aim to use transformer embeddings instead of sentence transformers. We also plan to test other cross-lingual transformers for transfer learning in a larger study to determine if the conclusions obtained on this small dataset generalize or are obtained by this data-driven approach.

Acknowledgments

The research presented in this paper was supported in part by the German Academic Exchange Service (DAAD) through the projects "AWAKEN: content-Aware and netWork-Aware faKE News mitigation" (grant no. 91809005) "Deep-Learning Anomaly Detection for Human and Automated Users Behavior" (grant no. 91809358), in part by the German Federal Ministry of Education and Research (BMBF) project "PANQURA - a technology platform for more information transparency in times of crisis" under Grant 03COV03F, in part by the European Union project "FAST-LISA - Fighting hAte Speech Through a Legal, ICT and Sociolinguistic approach" under Grant 101049342, and in part by the EU CEF project "NORDIS - NORdic observatory for digital media and information DISorder" under Grant number2394203). [14] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever, et al., Language models are unsupervised multitask learners, OpenAI blog 1 (2019) 9. [15] S. Hochreiter, J. Schmidhuber, Long Short-Term Memory, Neural Computation 9 (1997) 1735–1780. doi:10.1162/neco.1997.9.8.1735. [16] F. A. Gers, J. Schmidhuber, F. Cummins, Learning to forget: Continual prediction with

LSTM, Neural Computation 12 (2000) 2451–2471. doi:10.1162/089976600300015015. [17] X. Glorot, Y. Bengio, Understanding the dificulty of training deep feedforward neural networks, in: Y. W. Teh, M. Titterington (Eds.), Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, volume 9 of Proceedings of Machine Learning Research, PMLR, Chia Laguna Resort, Sardinia, Italy, 2010, pp. 249–256. URL: https://proceedings.mlr.press/v9/glorot10a.html. [18] P. Nakov, A. Barrón-Cedeño, G. D. S. Martino, F. Alam, J. M. Struß, T. Mandl, R. Míguez, T. Caselli, M. Kutlu, W. Zaghouani, C. Li, S. Shaar, G. K. Shahi, H. Mubarak, A. Nikolov, N. Babulkov, Y. S. Kartal, J. Beltrán, The CLEF-2022 CheckThat! Lab on Fighting the COVID19 Infodemic and Fake News Detection, in: Lecture Notes in Computer Science, Springer International Publishing, 2022, pp. 416–428. doi:10.1007/978-3-030-99739-7_52. [19] P. Nakov, A. Barrón-Cedeño, G. Da San Martino, F. Alam, J. M. Struß, T. Mandl, R. Míguez, T. Caselli, M. Kutlu, W. Zaghouani, C. Li, S. Shaar, G. K. Shahi, H. Mubarak, A. Nikolov, N. Babulkov, Y. S. Kartal, J. Beltrán, M. Wiegand, M. Siegel, J. Köhler, Overview of the CLEF-2022 CheckThat! Lab on Fighting the COVID-19 Infodemic and Fake News Detection, in: Proceedings of the 13th International Conference of the CLEF Association: Information Access Evaluation meets Multilinguality, Multimodality, and Visualization, CLEF ’2022, Bologna, Italy, 2022. [20] G. K. Shahi, D. Nandini, FakeCovid – A Multilingual Cross-domain Fact Check News Dataset for COVID-19, in: Workshop Proceedings of the 14th International AAAI Conference on Web and Social Media, ICWSM, 2020, pp. 1–9. URL: http://workshop-proceedings. icwsm.org/pdf/2020_14.pdf. doi:10.36190/2020.14. [21] J. Köhler, G. K. Shahi, J. M. Struß, M. Wiegand, M. Siegel, T. Mandl, Overview of the CLEF-2022 CheckThat! Lab Task 3 on Fake News Detection, in: Working Notes of CLEF 2022—Conference and Labs of the Evaluation Forum, CLEF ’2022, Bologna, Italy, 2022. [22] G. K. Shahi, AMUSED: An Annotation Framework of Multi-modal Social Media Data, arXiv preprint arXiv:2010.00502 (2020). [23] G. K. Shahi, J. M. Struß, T. Mandl, Overview of the CLEF-2021 CheckThat! lab task 3 on fake news detection, Working Notes of CLEF (2021). [24] G. K. Shahi, A. Dirkson, T. A. Majchrzak, An exploratory study of COVID-19 misinformation on Twitter, Online Social Networks and Media 22 (2021) 100104. [25] T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz, J. Davison, S. Shleifer, P. von Platen, C. Ma, Y. Jernite, J. Plu, C. Xu, T. L. Scao, S. Gugger, M. Drame, Q. Lhoest, A. Rush, Huggingface’s transformers: State-of-the-art natural language processing, in: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, Association for Computational Linguistics, 2020, pp. 38–45. doi:10.18653/v1/2020.emnlp-demos.6.

References

[1] V.-I. Ilie , C. - O. Truică , E.-S. Apostol , A. Paschke , Context-Aware Misinformation Detection: A Benchmark of Deep Learning Architectures Using Word Embeddings, IEEE Access 9 ( 2021 ) 162122 - 162146 . doi: 10 .1109/ACCESS. 2021 . 3132502 .

[2] C.-O. Truică , E.-S. Apostol, MisRoBÆRTa: Transformers versus Misinformation, Mathematics 10 ( 2022 ) 1 - 25 ( 569 ). doi: 10 .3390/math10040569.

[3] M. Singh , M. W. Bhatt , H. S. Bedi , U. Mishra, Performance of Bernoulli's Naïve Bayes classifier in the detection of fake news , Materials Today: Proceedings ( 2020 ).

[4] R. K. Kaliyar , A. Goswami , P. Narang , S. Sinha, FNDNet - A deep convolutional neural network for fake news detection , Cognitive Systems Research 61 ( 2020 ) 32 - 44 . doi: 10 . 1016/j.cogsys. 2019 . 12 .005.

[5] H. Saleh , A. Alharbi , S. H. Alsamhi , OPCNN-FAKE: Optimized convolutional neural network for fake news detection , IEEE Access 9 ( 2021 ) 129471 - 129489 .

[6] S. Samantaray , A. Kumar , Bi-directional Long Short-Term Memory Network for Fake News Detection from Social Media , in: Intelligent and Cloud Computing , Springer, 2022 , pp. 463 - 470 .

[7] T. E. Trueman , A. Kumar , P. Narayanasamy , J. Vidya , Attention-based C-BiLSTM for fake news detection , Applied Soft Computing 110 ( 2021 ) 107600 . doi: 10 .1016/j.asoc. 2021 . 107600 .

[8] J. Devlin , M.- W. Chang , K. Lee , K. Toutanova , BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, in: Conference of the North American Chapter of the Association for Computational Linguistics , ACL , 2019 , pp. 4171 - 4186 . doi: 10 .18653/v1/ N19 -1423.

[9] Y. Liu , M. Ott , N. Goyal , J. Du , M. Joshi , D. Chen , O. Levy , M. Lewis , L. Zettlemoyer , V. Stoyanov, RoBERTa: A Robustly Optimized BERT Pretraining Approach , 2019 . arXiv: 1907 .11692.

[10] M. Lewis , Y. Liu , N. Goyal , M. Ghazvininejad , A. Mohamed , O. Levy , V. Stoyanov , L. Zettlemoyer, BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension, in: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , Association for Computational Linguistics, 2020 , pp. 7871 - 7880 . doi: 10 .18653/v1/ 2020 .acl-main. 703 .

[11] C. Liu , X. Wu , M. Yu , G. Li , J. Jiang , W. Huang , X. Lu , A Two-Stage Model Based on BERT for Short Fake News Detection , in: International Conference on Knowledge Science, Engineering and Management, Springer, 2019 , pp. 172 - 183 . doi: 10 .1007/978-3- 030 -29563-9\ _ 17 .

[12] N. Reimers , I. Gurevych , Sentence-BERT: Sentence Embeddings using Siamese BERTNetworks , in: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) , Association for Computational Linguistics , 2019 , pp. 3982 - 3992 . doi: 10 .18653/v1/d19- 1410 .

[13] A. Conneau , G. Lample, Cross-lingual Language Model Pretraining , in: Advances in Neural Information Processing Systems , volume 32 , 2019 , pp. 1 - 11 . URL: https://proceedings. neurips.cc/paper/2019/file/c04c19c2c2474dbf5f7ac4372c5b9af1-Paper.pdf.