Factchecking has always been a part of the journalistic process. However with newsroom budgets shrinking [Pew16] it is coming under increasing pressure just as the amount of false information circulating is on the rise [MAGM18]. We therefore propose a method to increase the e ciency of the factchecking process, using the latest developments in Natural Language Processing (NLP). This method allows us to compare incoming claims to an existing corpus and return similar, factchecked, claims in a live system|allowing factcheckers to work simultaneously without duplicating their work.
In recent years, the spread of misinformation has become a growing concern for researchers and the public at large [MAGM18]. Researchers at MIT found that social media users are more likely to share false information than true information [VRA18]. Due to renewed focus on nding ways to foster healthy political conversation, the pro le of factcheckers has been raised.
Factcheckers positively in uence public debate by publishing good quality information and asking politicians and journalists to retract misleading or false statements. By calling out lies and the blurring of the truth, they make those in positions of power accountable. This is a result of labour intensive work that involves monitoring the news for spurious claims and carrying out rigorous research to judge credibility. So far, it has only been possible to scale their output upwards by hiring more personnel. This is problematic because newsrooms need signi cant resources to employ factcheckers. Publication budgets have been decreasing, resulting in a steady decline in the size of their workforce [Pew16]. Factchecking is not a directly pro table activity, which negatively a ects the allocation of resources towards it in for-pro t organisations. It is often taken on by charities and philanthropists instead.
To compensate for this shortfall, our strategy is to harness the latest developments in NLP to make factchecking more e cient and therefore less costly. To this end, the new eld of automated factchecking has captured the imagination of both non-pro ts and start-ups [Gra18, BM16, TV18]. It aims to speed up certain aspects of the factchecking process rather than create AI that can replace factchecking personnel. This includes monitoring claims that are made in the news, aiding decisions about which statements are the most important to check and automatically retrieving existing factchecks that are relevant to a new claim.
The claim detection and claim clustering methods that we set out in this paper can be applied to each of these. We sought to devise a system that would automatically detect claims in articles and compare them to previously submitted claims. Storing the results to allow a factchecker's work on one of these claims to be easily transferred to others in the same cluster. 2.1 It is important to decide what sentences are claims before attempting to cluster them. The rst such claim detection system to have been created is ClaimBuster [HNS+17], which scores sentences with an SVM to determine how likely they are to be politically pertinent statements. Similarly, ClaimRank [JGBC+18] uses real claims checked by factchecking institutions as training data in order to surface sentences that are worthy of factchecking.
These methods deal with the question of what is a politically interesting claim. In order to classify the objective qualities of what set apart di erent types of claims, the ClaimBuster team created PolitiTax [Car18], a taxonomy of claims, and factchecking organisation Full Fact [KPBZ18] developed their preferred annotation schema for statements in consultation with their own factcheckers. This research provides a more solid framework within which to construct claim detection classi ers.
The above considers whether or not a sentence is a claim, but often claims are subsections of sentences and multiple claims might be found in one sentence. In order to accommodate this, [LGS+17] proposes extracting phrases called Context Dependent Claims (CDC) that are relevant to a certain `Topic'. Along these lines, [AJC+19] proposes new de nitions for frames to be incorporated into FrameNet [BFL98] that are speci c to facts, in particular those found in a political context. 2.2
It is much easier to build a dataset and reliably evaluate a model if the starting de nitions are clear and objective. Questions around what is an interesting or pertinent claim are inherently subjective. For example, it is obvious that a politician will judge their opponents' claims to be more important to factcheck than their own.
Therefore, we built on the methodologies that dealt with the objective qualities of claims, which were the PolitiTax and Full Fact taxonomies. We annotated sentences from our own database of news articles based on a combination of these. We also used the Full Fact de nition of a claim as a statement about the world that can be checked. Some examples of claims according to this de nition are shown in Table 1. We decided the rst statement was a claim since it declares the occurrence of an event, while the second was considered not to be a claim as it is an expression of feeling.
Full Fact's approach centred around using sentence embeddings as a feature engineering step, followed by a simple classi er such as logistic regression, which is what we used. They used Facebook's sentence embeddings, InferSent [CKS+17], which was a recent breakthrough at the time. Such is the speed of new development in the eld that since then, several papers describing textual embeddings have been published. Due to the fact that we had already evaluated embeddings for clustering, and therefore knew our system would rely on Google USE Large [CYK+18], we decided to use this instead. We compared this to TFIDF and Full Fact's results as baselines. The results are displayed in Table 2.
However, ClaimBuster and Full Fact focused on live factchecking of TV debates. Logically is a news aggregator and we analyse the bodies of published news stories. We found that in our corpus, the majority of sentences are claims and therefore our model needed to be as selective as possible. In practice, we choose to lter out sentences that are predictions since generally the substance of the claim cannot be fully checked until after the event has occurred. Likewise, we try to remove claims based on personal experience or anecdotal evidence as they are di cult to verify. Traditional text clustering methods, using TFIDF and some clustering algorithm, are poorly suited to the problem of clustering and comparing short texts, as they can be semantically very similar but use different words. This is a manifestation of the the data sparsity problem with Bag-of-Words (BoW) models. [SR15]. Dimensionality reduction methods such as Latent Dirichlet Allocation (LDA) can help solve this problem by giving a dense approximation of this sparse representation [BNJ03]. More recently, e orts in this area have used text embedding-based systems in order to capture dense representation of the texts [WXX+15]. Much of this recent work has relied on the increase of focus in word and text embeddings. Text embeddings have been an increasingly popular tool in NLP since the introduction of Word2Vec [MCCD13], and since then the number of di erent embeddings has exploded. While many focus on giving a vector representation of a word, an increasing number now exist that will give a vector representation of a entire sentence or text. Following on from this work, we seek to devise a system that can run online, performing text clustering on the embeddings of texts one at a time 3.1.1
Some considerations to bear in mind when deciding on an embedding scheme to use are: the size of the nal vector, the complexity of the model itself and, if using a pretrained implementation, the data the model has been trained on and whether it is trained in a supervised or unsupervised manner.
The size of the embedding can have numerous results downstream. In our example we will be doing distance calculations on the resultant vectors and therefore any increase in length will increase the complexity of those distance calculations. We would therefore like as short a vector as possible, but we still wish to capture all salient information about the claim; longer vectors have more capacity to store information, both salient and non-salient.
A similar e ect is seen for the complexity of the model. A more complicated model, with more trainable parameters, may be able to capture ner details about the text, but it will require a larger corpus to achieve this, and will require more computational time to calculate the embeddings. We should therefore attempt to nd the simplest embedding system that can accurately solve our problem.
When attempting to use pretrained models to help in other areas, it is always important to ensure that the models you are using are trained on similar material, to increase the chance that their ndings will generalise to the new problem. Many unsupervised text embeddings are trained on the CommonCrawl 1 dataset of approx. 840 billion tokens. This gives a huge amount of data across many domains, but requires a similarly huge amount of computing power to train on the entire dataset. Supervised datasets are 1CommonCrawl found at http://commoncrawl.org/ unlikely ever to approach such scale as they require human annotations which can be expensive to assemble. The SNLI entailment dataset is an example of a large open source dataset [BAPM15]. It features pairs of sentences along with labels specifying whether or not one entails the other. Google's Universal Sentence Encoder (USE) [CYK+18] is a sentence embedding created with a hybrid supervised/unsupervised method, leveraging both the vast amounts of unsupervised training data and the extra detail that can be derived from a supervised method. The SNLI dataset and the related MultiNLI dataset are often used for this because textual entailment is seen as a good basis for general Natural Language Understanding (NLU) [WNB18]. 3.2
In order to choose an embedding, we sought a dataset to represent our problem. Although no perfect matches exist, we decided upon the Quora duplicate question dataset [SIC17] as the best match. To study the embeddings, we computed the euclidean distance between the two questions using various embeddings, to study the distance between semantically similar and dissimilar questions.
The graphs in gure 1 show the distances between duplicate and non-duplicate questions using di erent embedding systems. The X axis shows the euclidean distance between vectors and the Y axis frequency. A perfect result would be a blue peak to the left and an entirely disconnected orange spike to the right, showing that all non-duplicate questions have a greater euclidean distance than the least similar duplicate pair of questions. As can be clearly seen in the gure above, Elmo [PNI+18] and Infersent [CKS+17] show almost no separation and therefore cannot be considered good models for this problem. A much greater disparity is shown by the Google USE models [CYK+18], and even more for the Google USE Large model. In fact the Google USE Large achieved a F1 score of 0.71 for this task without any speci c training, simply by choosing a threshold below which all sentence pairs are considered duplicates.
In order to test whether these results generalised to our domain, we devised a test that would make use of what little data we had to evaluate. We had no original data on whether sentences were semantically similar, but we did have a corpus of articles clustered into stories. Working on the assumption that similar claims would be more likely to be in the same story, we developed an equation to judge how well our corpus of sentences was clustered, rewarding clustering which matches the article clustering and the total number of claims clustered. The precise formula is given below, where Pos is the proportion of claims in clusters from one story cluster, Pcc is the proportion of claims in the correct claim cluster, where they are from the most common story cluster, and Nc is the number of claims placed in clusters. A,B and C are parameters to tune.
A
Pos + B
Pcc (C
Nc)
This method is limited in how well it can represent the problem, but it can give indications as to a good or bad clustering method or embedding, and can act as a check that the ndings we obtained from the Quora dataset will generalise to our domain. We ran code which vectorized 2,000 sentences and then used the DBScan clustering method [EKSX96] to cluster using a grid search to nd the best value, maximizing this formula. We used DBScan as it mirrored the clustering method used to derive the original article clusters. The results for this experiment can be found in Table 3. We included TFIDF in the experiment as a baseline to judge other results. It is not suitable for our eventual purposes, but it the basis of the original keyword-based model used to build the clusters 2. That being said, TFIDF performs very well, with only Google USE Large and Infersent coming close in terms of `accuracy'. In the case of Infersent, this comes with the penalty of a much smaller number of claims included in the clusters. Google USE Large, however, clusters a greater number and for this reason we chose to use Google's USE Large. 3
Since Google USE Large was the best-performing embedding in both the tests we devised, this was our chosen embedding to use for clustering. However as can be seen from the results shown above, this is not a perfect solution and the inaccuracy here will introduce inaccuracy further down the clustering pipeline. 3.3
We decided to follow a methodology upon the DBScan method of clustering [EKSX96]. DBScan considers all distances between pairs of points. If they are under then those two are linked. Once the number of connected points exceeds a minimum size threshold, they are considered a cluster and all other points are considered to be unclustered. This method is advantageous for our purposes because unlike other methods, such as K-Means, it does not require the number of clusters to be speci ed. To create a system that can build clusters dynamically, adding one point at a time, we set 2Described in the newslens paper [LH17] 3Google USE Large is the Transformer based model, found at https://tfhub.dev/google/universal-sentence-encoderlarge/3, whereas Google USE uses a DAN architecture the minimum cluster size to one, meaning that every point is a member of a cluster.
A potential disadvantage of this method is that because points require only one connection to a cluster to join it, they may only be related to one point in the cluster, but be considered in the same cluster as all of them. In small examples this is not a problem as all points in the cluster should be very similar. However as the number of points being considered grows, this behaviour raises the prospect of one or several borderline clustering decisions leading to massive clusters made from tenuous connections between genuine clusters. To mitigate this problem we used a method described in the Newslens paper [LH17] to solve a similar problem when clustering entire articles. We stored all of our claims in a graph with the connections between them added when the distance between them was determined to be less than . To determine the nal clusters we run a Louvain Community Detection [BGLL08] over this graph to split it into de ned communities. This improved the compactness of a cluster. When clustering claims one by one, this algorithm can be performed on the connected subgraph featuring the new claim, to reduce the computation required.
As this method involves distance calculations between the claim being added and every existing claim, the time taken to add one claim will increase roughly linearly with respect to the number of previous claims. Through much optimization we have brought the computational time down to approximately 300ms per claim, which stays fairly static with respect to the number of previous claims. 4
Next Steps The clustering described above is heavily dependent on the embedding used. The rate of advances in this eld has been rapid in recent years, but an embedding will always be an imperfect representation of an claim and therefore always an area of improvement. A domain speci c-embedding will likely o er a more accurate representation but creates problems with clustering claims from di erent domains. They also require a huge amount of data to give a good model and that is not possible in all domains.
Thanks to Anil Bandhakavi, Tom Dakin and Felicity Handley for their time, advice and proofreading. [AJC+19]
[BFL98] [BGLL08] [BM16] [BNJ03] [Car18] [CKS+17]
Schwenk, Loic Barrault, and Antoine Bordes. Supervised learning of universal sentence representations from natural language inference data, 2017. [CYK+18] Daniel Cer, Yinfei Yang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St.
John, Noah Constant, Mario GuajardoCespedes, Steve Yuan, Chris Tar, YunHsuan Sung, Brian Strope, and Ray Kurzweil. Universal sentence encoder.
2018. [EKSX96] [Gra18] [HNS+17]
Sander, and Xiaowei Xu. A density-based algorithm for discovering clusters in large spatial databases with noise. pages 226{ 231. AAAI Press, 1996.
[Jon72] [LGS+17] [LH17]
tation of term speci city and its application in retrieval. Journal of Documentation, 28:11{21, 1972.
Mevan Babakar, and Arkaitz Zubiaga. Towards automated factchecking: Developing an annotation schema and benchmark for consistent automated claim detection. 2018.
jder, Shay Hummel, Ranit Aharonov, and Noam Slonim. Unsupervised corpus{wide claim detection. In Proceedings of the 4th Workshop on Argument Mining. Association for Computational Linguistics, 2017.
Philippe Laban and Marti Hearst. newslens: building and visualizing longranging news stories. In Proceedings of the Events and Stories in the News Workshop, pages 1{9, Vancouver, Canada, 2017. [MAGM18] Bertin Martens, Luis Aguiar, Estrella Gomez-Herrera, and Frank MuellerLanger. The digital transformation of news media and the rise of disinformation and fakenews. 2018. [MCCD13] Tomas Mikolov, Kai Chen, Greg Corrado, and Je rey Dean. E cient estimation of word representations in vector space.
2013. [PNI+18]
media, 2016. [SIC17] [SR15] [TV18] [VRA18] [WNB18]
NAACL-HLT 2018, 2018. [WXX+15] Peng Wang, Jiaming Xu, Bo Xu, ChengLin Liu, Heng ZhangFangyuan Wang, and Hongwei Hao. Semantic clustering and convolutional neural networkfor short text categorization. Number 6, pages 352{ 357, 2015.