Diverse information sources are becoming more accessible, and news can spread quickly, including fake and highly biased news. It is, therefore, essential to transparently convey to a user what news is potentially fake. Attributing statements inside those fake news to known sources is a potential way to check their validity. However, such attributions are seldom usable due to missing data sets. Therefore, this work aims to define a framework for attributing statements to source documents that do not necessarily contain the exact statements. We are applying our approach to classifying fake news in a search setting and include visual depictions of attributions to explain why something is considered fake or reliable. This method can support users in deciding which news to read and aid in grouping information by credibility.
Our investigation only assumes the existence of textual data. Hence, we do not consider methods
requiring other data like social networks (e.g. Shu, et al. [
Subset Extraction
Training and
Test Split Test Data
(Cosine Similarity) Filtering
Concatenated Feature Vector
Our approach uses deep learning-based semantic embeddings and similarities between sentences. The goal is to support a user’s decision on the reliability of an article while maintaining accuracy. The overview of the approach is shown in Fig. 1 and consists of three parts: 1) Embedding: We are using a self-trained Doc2Vec (|| = 20) and a pre-trained DeBERTa called Base-MNLI 1. 2) Comparisons: We calculate the nearest neighbor for sentences using cosine similarity between their embeddings and filter them by thresholding on the average similarity found. 3) Feature Creation: From the nearest neighbors we calculate a feature vector of length 34 containing: (i) the absolute and relative counts of labeled attributions, (ii) the pair-wise counts of two adjacent sentences and (iii) the start and end label of each text. These features are extracted on our two data sets, KOREA and COMPETITION, which are subsets of the Fake News Corpus2, filtered by a keyword match where KOREA used the keywords korea and nuclear and COMPETITION fun and competition, leaving 4046 fake and 7917 reliable instances, and 5577 fake and 14996 reliable samples, respectively. We did try known data sets like the Fake News dataset3, made available by Kaggle and the ISOT Fake News Dataset4 from the University of Victoria. However, both data sets were not promising since they were easily classifiable using simple stylistic features with over 94% accuracy. We employed ElasticSearch5 to index the KOREA data set and built a simple UI on top, adding a highlighter to show how sentences were attributed in the text. A demo of the UI is shown in a video at https://youtu.be/ZbqgIBQ4cI0.
In pre-experiments, we determined that a Random Forest (RF) consistently gave us the best results, which is why we discuss it here. Table 1 shows the results for the test data on the KOREA data set. The results for the COMPETITION are almost the same. Both data sets indicate a high precision and F1 measure with simple attributions. Only the recall for fake news is not excellent yet. If the threshold of attributions is changed, the recall increases for a high precision cost. However, we concluded that these types of attributions form a robust classifier. 1https://github.com/microsoft/DeBERTa 2https://github.com/several27/FakeNewsCorpus 3https://www.kaggle.com/datasets/jruvika/fake-news-detection 4https://www.uvic.ca/ecs/ece/isot/datasets/fake-news/index.php 5https://www.elastic.co/elasticsearch Additionally, many attributions provide a Table 1: Precision, Recall and F1-Score using a RF on good insight into why a text is considered the KOREA test data. fake. For example, a sourced article describes precision recall f1-score support talks with North Korea about its nuclear profake 0.95 0.89 0.92 1360 gram, whereas the target sentence reports a reliable 0.94 0.98 0.96 2588 readiness to dismantle it altogether. However, a weakness of the approach is in its nature mmiaccrrooaavvgg 00..9945 00..9943 00..9944 33994488 since some attributions are not sensible from a user’s perspective. I.e., the general content of the text is similar, but the sentences have no relationship. The misleading attributions are a challenge that has to be overcome in future work. Many unreliable attributions remain since we optimized the thresholding mostly on classification accuracy. These less sensible attributions can help in classification since they partially stem from a similarly worded article, often from the same source domain. A stricter source selection strategy might help sort out such attributions and decrease run time with only a minor reduction in accuracy.
In this paper, we presented a simple method of source-based attribution for fake news classification. This method works reasonably well on a small data set but needs to be tested on a larger domain. Based on initial experiments, the cross-domain accuracy drops significantly due to missing attributions. Hence, more work towards domain-independence and better source selection is required.