Internet is the all-singing, all-dancing, multi-pronged tool of modern man, but, for its many bene ts it also presents us with daunting challenges. The user is bombarded with tons of information and it is becoming increasingly di cult to tell fact from ction. Consequently, over the past few years research work has intensi ed in this regard to better equip the user against false information and indoctrination. In this paper, we propose a system for automatic retrieval of supporting claims given an input claim. Our system uses elasticsearch along with a transformer model to generate a similarity score in two steps. We have submitted our system to CLEF-2020 Check That! Lab's Task-2: Claim Retrieval and our primary evaluations gave promising results which we will present, analyse and improve henceforth in the paper.
The internet has come a really long way in the past couple of decades in terms
of acceptability and reach. And with easy accessibility to low-cost internet, and
wide-appeal of social media services like Twitter, there is an overabundance of
unveri ed information online. Now, this glaring lack of factual validation
combined with the lucrative carte blanche that the internet allows, has given way to
the taxing problem of gross misinformation and disregard for factual correctness.
Needless to say, this has been a green pasture for feckless rumour-mongers and
ill-intentioned propagandists. So, now more than ever, there is an exigent demand
for automatic fact checking to safeguard internet users from misinformation and
indoctrination. To that end, multiple investigative-journalistic fact-checking
organisations like Snopes, IFCN, Full Fact etc. have emerged. But, manual fact
checking is a very demanding and slow process that can take one full day to
research and write about a claim [
Looking at these problems faced in manual fact checking, automated fact
checking can be very advantageous to fully automate or assist the existing
pipelines. There has been a signi cant amount of work in the eld of automated
fact checking in recent years. Various tasks have been formulated related to or
that come under automated fact checking [
In this paper, we focus on one of the key components of a fact-checking
pipeline, that is, claim retrieval. The principles governing our approach are
simplicity, directness and scalability. Our system uses elastic search along with
transformer model similarity between the tweet and claim to generate a score in two
steps : exact matching of words using BM25 and then, an NLP-based model to
check for semantic similarities [
We propose an unsupervised system for the task. Our submitted system uses elastic search BM25 score along with cosine similarity between two text pieces' BERT encodings. Figure 1 shows the working of our submitted system. 1https://github.com/us241098/checkthat2020_submission Firstly, all veri ed claims are indexed in elasticsearch. Ideally some preprocessing like removing URLs, hashtags and @ sign should be performed before indexing, but we skip this step as veri ed claims in the corpus are mostly free of these. Creating this index ensures quick retrieval of related articles at the time of query. At the time of query, we retrieve top 1000 matching claims for the query/tweet along with their BM25 scores. BM scores are dependent on the exact match between the words of claim and tweet and are assigned to every claim-tweet pair. However, our initial experiments have shown that using just these scores for ranking claims give underwhelming results, hence, we use transformer-based similarity in our next step and update ranks after adding the scores assigned by it. Since BM25 similarity relies on exact word match between the text pieces, it fails to capture the semantic similarity between claims and tweets. To compensate for this, we have used a BERT ne-tuned on NLI data to generate encodings for our text piece, we get the similarity between two encodings by computing cosine distance between them. Based on the similarity we update the scores from BM25 and subsequently update the ranks. In our system's implementation we have used the sentence transformer python package and bert base nli mean tokens model. 3
In this section, results of our submission to Task 2 will be discussed and compared to the best performing system. In the subsequent subsection, we will discuss the cases where our system performs poorly and attempt to improve our submitted system. Table 1 shows our system's result, metrics on which systems were evaluated were MAP@1, 3, 5, all, Precision@1, 3, 5 and Reciprocal Rank@1, 3, 5. Systems were ranked on the basis of performance on MAP@5. Our submission was ranked 6th among 9 participants. @1 @3 @5 - @1 @3 @5 @1 @3 @5 Buster.AI(1st) 0.897 0.926 0.929 0.929 0.895 0.32 0.195 0.895 0.923 0.927 trueman(6th) 0.743 0.768 0.773 0.782 0.74 0.267 0.164 0.74 0.766 0.771 trueman (unsubmitted) 0.757 0.797 0.80 0.808 0.759 0.283 0.173 0.759 0.798 0.802 3.1
We manually analysed our results and found that our system made errors in the following cases:
Proper Noun Overlap: We observed that our system sometimes fails and returns claims that do not have the same proper nouns as the query and sometimes lets some proper nouns slip by. Table 2 shows some examples of the faulty results vis-a-vis proper noun overlap. Good to see the Nish- Video shows an amaz- The comedian and actor ioka shot being replayed ing behind-the-back re- Tim Allen wrote a lengthy again and again. . . that's turn by Japanese tennis Facebook post that atwhat we should be talking player Yoshihito Nishioka. tacked liberals and Demoabout. . . @MikeCTennis (10082) cratic politicians and was (1051) shared widely in August 2019. (9254)
Hyperlinks: We found that in many cases, the textual information in a tweet is very vague and crucial information regarding the tweet is contained in a hyperlink. So, to solve this problem we sought to extract article titles from the hyperlinks. Some erroneous results dealing with hyperlinks are shown in Table 3.
Hashtags (#) and At (@) sign: Upon analysis, we found out that our system was having major trouble if crucial proper nouns were contained inside a hashtag or at (@) sign. Some sample tweets showcasing this problem are tabularised below in Table 4. 3.2
Learning from the above analysis we have added a new module to our submitted system that takes removal of special symbols and noun overlap into account and accordingly updates the similarity scores. Since more often than not people put crucial (for retrieval purposes) information inside hashtags and at signs, A spill by the Environ- A new study reporting mental Protection Agency on the 2015 death of a rendered the normally Colorado infant claims the pristine blue Animas River event was the world's rst a terrifying mustard yel- documented pot overdose. low. (2558) (1838) There are small islands of re ants oating in the
oodwaters from Tropical Storm Harvey. (716)
Actor Kurt Russell said that he has never seen a man as dedicated and determined as President
Trump. (3125)
in our improved system we separated the words contained in these signs
using regular expressions and prioritised them when checking for overlaps. Ideally,
overlap of all proper nouns should be prioritised and to that end we added a
module to reward proper noun overlaps. We calculate the overlap using
Levenshtein distance [
In this paper we proposed a system for retrieval of veri ed claims given a query and submitted our results presented our results at CLEF-2020 Check That Lab!'s Claim Retrieval task. We strove to resolve some of the issues that our system faced initially by annotating and analysing faulty results, and managed to improve our performance. Yet, we strongly believe that there is a margin for future work to resolve some existing problems and to broaden the model's knowledge source in order to achieve practical application. The problem that the model was facing in regard to hyperlinks still needs some work in order to achieve full resolution. Since most of the links shared were from news or media websites, we think that title and byline retrieval from the hyperlinks will improve the system's performance considerably. Also, inculcating Image modality to analyse tweets with pictures merits future work in this regard. For the sake of this research, we considered fact-checking websites as a knowledge source in order to validate claims. We aspire to do the same and regard fact-checking and investigative journalism websites as a knowledge source but in an Indian context. This approach will improve our model's contextual sensibility and improve chances of identifying factual incorrectness and misinformation signi cantly. We believe that these approaches are good enough to merit signi cant future works and further research.