Introduction

The speed and scale at which information can be created and propagated has increased markedly over the last few decades, due to the rise of internet-based content creation and social media. Along with the many benefits of this increase, it has also enabled malicious actors to initiate disinformation campaigns [1] with real world implications, shaping public opinion in forms of politics, pandemics, and more. Human fact checking efforts, while usually high-quality (i.e. high precision) and impactful, fail to match false information's reach, speed and scale [2]. To remedy this, researchers around the world have begun to utilize Machine Learning in order to automate portions of the fact checking process, with the goal of scaling and speeding up these efforts to match the strong need. However, many challenges exist when fact checking itself becomes automated or determination of veracity is left in the hands of opaque machines. Therefore, one of the leading approaches relies on a specific subproblem, commonly referred to as "Claim Matching", whereby machine learning is used to map novel content, such as social media posts, with a database of fact-checked claims, i.e., claims that have already been verified to be true or false [3,4]. Claim matching approaches can work well either as standalone systems, or else integrated into human in-the-loop systems, where the automated portion can be used to triage and prioritize incoming posts/claims, and/or to point users to existing fact checks, e.g. via fact checking tiplines [5].

Current literature on claim matching focuses on encoding-based methods [6,7]. Most of these approaches are built with deep pre-trained language models. These models have become ubiquitous in natural language processing since the emergence of BERT [7] in 2018. In addition to encoding-based approaches, generative models have recently received renewed interest due to their ability to provide further insight and understanding of the model's inferences [8]. To the best of our knowledge, we have not seen deep generative models used specifically within the subdomain of claim matching.

Data

For evaluation we use CLEF's CheckThat! task 2a-English claim-matching competition data from 2021 and 2022 [9,10]. The data-sets comprise of tweets, a verified natural-language claims database, and their corresponding connections, with each tweet mapped to precisely one claim. The claims database has approximately 14000 claims. Each year, CLEF's CheckThat! team provides 1000 training points, 200 development points, and around 200 for test. Note that at the time of writing, we have access only to the test labels from 2021's competitions, but not for 2022; our results are quoted from the official, finalized leaderboard.

Candidate Selection

Our claim matching pipeline is broken down into two stages following the traditional neural IR methodology of first using an efficient model for candidate selection, and then following up with a more expensive model to reevaluate those candidates. More specifically, we will consider methods that run in 𝒪(𝐶) to be efficient, where 𝐶 is the size of the claim database. For candidate selection, we test 2 approaches. The first is a lightweight bag-of-words approach which only utilizes unigram distributions, and the second uses a sentence transformer, which is still linear, but models context via deep neural networks. We will discuss each in turn, before introducing the re-ranking approach in the following section.

Bag of Words

We use BM25 for the bag of words baseline, implemented by the rank-bm25 python package. We preprocess the text in 3 steps: (1) we concatenate the claim's title, header, and body (see, e.g., [11]); (2) we convert all text to lowercase and remove stop words, using NLTK's English list [12]; and finally (3) and we apply Porter stemming.

Sentence Transformer

Sentence transformers [13] utilize the widely successful self-attention based architectures [14], but are trained to produce embeddings projected to a unit sphere in Euclidean space. This means that computing the angle (or Euclidean distance) between two inputs can have contextual meaning, enabling search with 𝒪(𝑇 + 𝐶) inferences; the alternative of computing each tweetclaim pair directly requires 𝒪(𝑇 * 𝐶) runs, where 𝑇 is the number of tweets. Specifically, we use Sentence-T5 [15], which to the best of our knowledge is the strongest sentence transformer when normalized by model size.

To best incorporate batched training, we use a Multiple Negative Rankings (MNR) loss function. MNR's batched loss is described by

ℒ(ℬ) = ∑︁ (𝑥,𝑦)∈ℬ − log 𝑆(𝑥, 𝑦) ∑︀ (_,𝑦 * )∈ℬ 𝑆(𝑥, 𝑦 * )(1)

Where, given a network 𝑓 , 𝑆(𝑥, 𝑦) is the function 𝑒𝑥𝑝(𝑓 (𝑥) 𝑇 𝑓 (𝑦)). This loss function is appealing for efficiency reasons: it can be computed with a single training step using inter-batch normalization. In our case, we also wanted to add "harder" negatives because otherwise the task becomes too easy for the model. In other words, we have ℬ = ℬ + ∪ ℬ − , and the loss becomes

ℒ(ℬ) = ∑︁ (𝑥,𝑦)∈ℬ + − log 𝑆(𝑥, 𝑦) ∑︀ (_,𝑦 * )∈ℬ 𝑆(𝑥, 𝑦 * )(2)

To determine the hard negatives, we use our BM25 Model to create ranked lists, and take the top ranked negative tweet.

Generative Re-ranking

The re-ranking task is posed as: given a tweet 𝑡 and a set of claim candidates 𝐶 𝑐𝑠 , return a ranked list of the claims. As discussed above, the generative approach models 𝑝(𝑡|𝑐) and uses that probability to generate the ranked list.

We decided to use a full-decoder setup, given the recent success of models like GPT-3 and Jurassic [16,17], who have shown numerous state-of-the-art results in low-data settings. For the backbone architecture, we use GPT-Neo's 1.3 Billion parameter model [18], which is a large auto-regressive transformer trained on the Pile [19]. We leave most of the architecture as-is, only modifying the model's usage of positional embeddings. In our case, we reset the ids for the claim and tweet separately.

To use the full decoder setting, we need to convert our inputs into a prompt. We use a straightforward setup which takes as input a claim 𝑐 and a tweet 𝑡, and turn it into the prompt < 𝑏𝑜𝑠 > 𝐶𝑙𝑎𝑖𝑚 : {𝑐} < 𝑒𝑜𝑠 >< 𝑏𝑜𝑠 > 𝑇 𝑤𝑒𝑒𝑡 : {𝑡} < 𝑒𝑜𝑠 >, where < 𝑏𝑜𝑠 > and < 𝑒𝑜𝑠 > represent the model tokenizer's native beginning-of-sentence and end-of-sentence tokens respectively.

Training Objective

While most approaches that leverage generative models for re-ranking rely on negative examples [20], we introduce a new method that does not. We do so by taking advantage of a full decoder's setup to also model the priors of the tweet, 𝑝(𝑡). This allows us to create a retriever-agnostic model, which requires no assumption of negatives. While this capability may be useful in contexts where retrieving negatives are difficult, we show that a fused objective when negatives are available performs the strongest.

Mutual Information is defined as

𝐼(𝑋; 𝑌 ) = ∑︁ 𝑥∈𝑋 ∑︁ 𝑦∈𝑌 𝑃 (𝑥, 𝑦) log 𝑃 (𝑥, 𝑦) 𝑃 (𝑥)𝑃 (𝑦) (3) = E 𝑋,𝑌 log 𝑃 (𝑥|𝑦) 𝑃 (𝑥)(4)

The expectation in the last line can be approximated stochastically in the optimization scheme but the fractional piece would require each batch to be made up of an additional element (prompt(t), prompt(t,c)) ∈ ℬ. Given network 𝑞 𝜃 , the optimization problem becomes

𝜃 * = 𝑎𝑟𝑔 min 𝜃 E 𝐶,𝑇 [− log 𝑞 𝜃 (𝑡|𝑐)) + log 𝑞 𝜃 (𝑡)](5)

To handle the objective not being taken over by the model weakening certain priors, common practice is to develop a hinged form of the objective. So we create the Hinged Mutual Information (HMI) loss as

ℒ 𝐻𝑀 𝐼 = E 𝐶,𝑇 [𝑚𝑎𝑥(0, 𝜆 − log 𝑞 𝜃 (𝑡|𝑐)) + log 𝑞 𝜃 (𝑡))](6)

This will remove elements with values above threshold 𝜆, which in training regimes that only pass the data at most once, is equivalent to removing them from the training set. To take advantage of the data, we decide to only mask the prior in the setting of surpassing the threshold and revert the loss back to Maximum Likelihood Encoding (MLE). We will denote this new loss as Hinged Prior Mutual Information Loss (HPMI)

ℒ 𝐻𝑃 𝑀 𝐼 = E 𝐶,𝑇 {︃ − log 𝑞 𝜃 (𝑡|𝑐) + log 𝑞 𝜃 (𝑡), if − log 𝑞 𝜃 (𝑡|𝑐) 𝑞 𝜃 (𝑡) < 𝜆 − log 𝑞 𝜃 (𝑡|𝑐), else(7)

While there is the memory-based drawback that each training element requires 2 runs of the model, one for computing 𝑞 𝜃 (𝑡|𝑐) and one for 𝑞 𝜃 (𝑡), we can take advantage of it by extracting additional regularization using the models Posterior. The only adjustment is that we input 𝑞 𝜃 (𝑐|𝑡) instead of 𝑞 𝜃 (𝑡), requiring a flipped ordered prompt: tweet → claim. Given our first run we also inherently get a claim prior 𝑞 𝜃 (𝑐) thanks to the decoder setup, which in tandem with the posterior lets us model mutual information another way by trying to model log 𝑝(𝑐|𝑡) 𝑝(𝑐) . Since we care less about claim priors, we regularize our loss by just using its hinged form, which we will denote Posterior based Hinged Mutual Information Loss ℒ 𝑃 𝑜𝐻𝑀 𝐼 which is identical to equation 6 just by replacing likelihood and tweet prior with the posterior and claim prior. Note that we do not care about modeling the claim via MLE in the hinged condition as we do in equation 7, so we just zero it out in those cases. Additionally, this doesn't model the exact same Mutual Information metric as the other losses as it adjusts the random variables positioning, which was another reason we reset the position-ids to help mitigate that discrepancy. Now that we have explained our retriever free training objective to best utilize vanilla data, since we do have a retriever we also incorporate negatives via negative log likelihood loss (nl3u) [20] getting a final training objective of Results on Test set of CLEF CheckThat! 2021 task 2a Claim Matching dataset. sent-T5 refers to sentence-T5 large model, neo-reranked is GPT-Neo re-ranker, and aschern refers to the 2021 winning submission [11].

ℒ 𝑚𝑖𝑥 = ℒ 𝐻𝑃 𝑀 𝐼 + ℒ 𝑃 𝑜𝐻𝑀 𝐼 + ℒ 𝑁 𝐿3𝑈(8)

Results

Model

Team/User MAP MAP@1 MAP@5 Motlogelwan .8775 .8325 .8731 Simba [21] .9075 .8756 .9075 Viktor [22] .9223 .9043 .9222 BigIR [23] .9225 .8995 .9211 RIET Lab (ours)

.957 .9426 .9555

Table 2

Officially released final Leaderboard for CLEF CheckThat! 2022 task 2a competition.

Model MAP@1 MAP@5 NL3U [20] . For evaluation, we follow the CLEF competitions leader-board evaluation and use Mean Average Precision (MAP). While they were finally scored on MAP@5, we show results for both MAP@1 and MAP@5. In this setting we evaluate with only 5 candidates.

To compute Average Precision @k (AP@k) for some ranked list 𝑅 with singular gold label 𝑔, it follows this formula

𝐴𝑃 @𝑘{𝑅, 𝑔} = 𝑘 ∑︁ 𝑖=1 1[𝑅[𝑖 − 1] == 𝑔] 1 𝑖(9)

where 1[•] is just an indicator function. MAP@k is just the mean of equation 9 over all ranked lists. Which with set of ranked list and corresponding label pairs denoted as Ω, becomes

𝑀 𝐴𝑃 @𝑘{Ω} = 1 |Ω| ∑︁ 𝑅,𝑔∈Ω 𝐴𝑃 @𝑘{𝑅, 𝑔}(10)

The results of baselines, encoding methods, and re-ranked versions are in Table 1 where we evaluated on 2021's competition data where the test set is accessible. As seen, re-ranking heavily improved upon it's candidate selection counterpart. Compared to the previous years competition winner that utilized sentence models and probabilistic re-ranking [11], the best model is 6 percentage points stronger. In Table 2 you see the results of 2022's competition's top 5 submissions where we submitted our generative re-ranking pipeline utilizing 25 candidates per tweet. We placed first beating out second place by around 3.4 percentage points.

We also completed a small ablation study on the rewards discussed in section 4, where the results are shown in Table 3. While a negative required loss still beat the ones we proposed where it isn't, it is only by a slight margin. Combining the approaches yielded the best results, and is what we used for our submission within the competition.

Conclusion and Future Directions

Claim matching holds significant promise as a component in mitigating mis-and disinformation. In this report, we have introduced a new objective for full-decoder generative re-ranker architectures that do not rely on negative samples as well as a mixed objective. Our approach is competitive; it improves upon the state-of-the-art results in the CLEF-2021 check-that task 2a, and was utilized to place first within the 2022 competition.

In this report, we focus only on using this pipeline for claim matching. Our work points to several directions for future work, such as applying the pipeline to other challenges; and comparing different objective functions, as well as their corresponding data assumptions. Though beyond the scope of this report, we further hypothesize that utilizing this pipeline in a human-in-the-loop system would be valuable, given the access to confidence statistics from the generative approach, which could not be achieved in encoder systems; we leave this study for future work.

Instructions to Reproduce

For finetuning the Sentence-T5 model we use the AdamW optimizer with a constant learning rate of 5e-6 with batch-size 6 for a single epoch (1 negative per positive in each batch). Training was performed on a single A6000 GPU. For finetuning the re-rankers we also use the AdamW optimizer with a constant learning rate of 2e-5 with batch-size 4 and reward hyperparameter 𝜆 of 2 for a single epoch. We performed this portion of training across 4 A6000 GPUs.

Our code can be found at https://github.com/RIET-lab/GenerativeClaimMatchingPipeline.