The growing spread of misinformation, particularly on social media, has increased the demand for scalable and accurate automated fact-checking systems. This paper presents our approach to Task 3 of the CheckThat! 2025 lab, which focuses on the verification of numerical claims. We propose a three-stage architecture comprising (i) semantic evidence retrieval using dense bi-encoder representations stored in a FAISS index, (ii) contrastive re-ranking with a fine-tuned cross-encoder leveraging weak supervision from gold evidences, and (iii) claim classification using multi-instance learning (MIL) with various evidence pooling strategies. Experiments on the QuanTemp dataset demonstrate that attention and LogSumExp pooling outperform standard concatenation methods, with our best model achieving a macro-F1 score of 0.5213 on the test set. Additionally, ablation studies confirm the efectiveness of contrastive re-ranking and the practical advantages of dense retrieval, which achieves both higher accuracy and significantly faster retrieval than traditional BM25.
Misinformation is one of today’s greatest challenges in the digital world. Whether they are distributed
intentionally or not, they can have a large influence on our opinions and spread rapidly [
The CheckThat! lab has been addressing diferent parts of the claim verification pipeline in multiple
iterations of its appearance, for instance in its releases in 2018 (check-worthiness detection and claim
verification), 2020 (all subtasks) and 2024 (check-worthiness detection) [
In this paper, we present our approach to the third task, which follows a three-stage architecture: (i) initial evidence candidate retrieval using dense vector similarity from a FAISS index, (ii) fine-grained re-ranking through a contrastively trained cross-encoder, and (iii) final claim classification via a MIL framework based on roberta-large-mnli. Our method emphasizes semantic retrieval over traditional surface-level approaches (e.g., BM25), leverages weakly supervised contrastive signals from gold evidences, and avoids noisy input concatenation by aggregating evidence representations through pooling mechanisms. This modular setup allows the system to flexibly combine retrieval and reasoning, while remaining scalable for real-world application.
Our main contributions are as follows: • We propose a modular pipeline for claim verification that integrates dense retrieval, contrastive re-ranking, and multi-instance classification for numerical claims. • We introduce a contrastive training setup using weak supervision from summarized gold evidences to improve re-ranking quality without relying on manual annotation. • We evaluate multiple evidence aggregation strategies (Max, LogSumExp, Attention), and show that pooling-based MIL improves over standard concatenation approaches.
The paper is structured as follows: Section 2 gives an overview of the fake news detection pipeline
as presented in [
This section provides an overview of the fake news detection pipeline (Subsection 2.1) and presents a formal definition of the claim verification task (Subsection 2.2).
The process of detecting and evaluating fake news was first formulated by Nakov et al. in 2018 [
The first task, check-worthiness estimation, assesses whether a statement needs further verification, i.e., if it needs to be passed on to a verifying authority. The second task, verified claim retrieval , attempts to determine whether the claim has already been fact-checked by searching a database of previously verified claims. If a matching verified claim is found, its label can directly support the classification of the new claim. The third step, evidence retrieval, involves gathering additional relevant evidence from external sources, such as web documents or news outlets, which may help in verifying the claim. Finally, the fourth task, claim verification , combines the original claim with the retrieved evidences to assess its veracity. This is described in more detail in the following Subsection.
The task of claim verification involves determining the veracity of a claim by consulting a set of evidences retrieved from a large corpus. Formally, given a query (claim) and relevant evidences, the goal is to predict whether the claim is true, false, or in contradiction with the available information. Mathematically, this can be described as follows:
Let: • = {1, 2, . . . , } be the set of queries (claims). • = {1, 2, . . . , } be the corpus of evidences. • : × → be the verification model , which maps a query and a set of evidences to a label. • = {True, False, Contradiction} be the label space.
Let be the retrieval function, selecting the top- relevant evidence passages from the corpus for a given query :
This can be formalized as:
: → () = arg
max ⊂ ,||= ∈
∑︁ rel(, ) ˆ = (, ) (1) (2) (3) (4) where rel(, ) is a relevance scoring function.
Given a query ∈ and its retrieved evidence set = (), the classifier predicts: where ˆ ∈ is the predicted label.
Given a training data set {(, , )}=1, where is the ground truth label for query with evidence set , the model parameters are optimized by minimizing the classification loss: ℒ( ) =
1 ∑︁ ℓ( (, ), ) =1 where ℓ is a loss function (cross-entropy).
The relevant dataset for the claim verification task in english is QuanTemp [
As can be seen, the dataset is imbalanced and consists mostly of false claims. It can be argued that the sources of the data, which comes mostly from PolitiFact and Snopes, efects the false classifications, as fact-checking organizations mainly deal with questionable or incorrect claims which need further verification.
Claim verification as a standalone task was first presented in 2018 and the next two editions. It
was then reintroduced in the 2023 CheckThat! lab. It has been held in both english and arabic
language, but it generally has not had as many participants as other tasks in the domain of the full
claim verification pipeline. In most cases, the participants retrieved external information to be able
to satisfactory classify the given claims [
Apart from challenges in the CheckThat! lab, approaches utilized Transformer models like BERT [14] or other language model ensembles [15] for claim verification. More recently, architectures based on graph networks [16] and large language models (LLMs) [17] also showed promising results and outperformed many traditional approaches.
Many of the presented approaches were evaluated using FEVER [18], a dataset consisting of over
185,000 generated claims used for claim verification. In 2024, Venktesh et al. [
Our submitted pipeline (english only) follows a three-stage architecture: (i) evidence candidate retrieval using dense vectors, (ii) re-ranking using a fine-tuned cross-encoder, and (iii) final claim classification using a large NLI model. In the following, each step is described in detail.
Evidence Retrieval We first pre-computed document embeddings using sentencetransformers/all-MiniLM-L6-v23 and stored them in a FAISS index for eficient innerproduct search. The embeddings were L2-normalized to enable cosine similarity retrieval. At inference time, each claim was encoded into a query embedding, normalized, and used to retrieve the top-100 most similar evidence snippets from the index. The goal was to retrieve not only lexically similar evidence snippets, but also semantically relevant ones that may be phrased diferently. 3https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2 where (· ) denotes the choice of pooling operator: 1. Max pooling (hard winner-take-all) picks the single most confident snippet per class: where ,ℓ is the logit of class ℓ. 2. LogSumExp pooling is a smooth approximation of the maximum: 3. Attention pooling learns a soft relevance weight for every snippet:
The initial candidates were re-ranked using cross-encoder/ms-marco-MiniLM-L-6-v24, fine-tuned in a contrastive setup. To improve reranking beyond the vanilla MS MARCO model, we generated weak supervision signals from goldlabeled evidence snippets in the training and validation sets. Further, these snippets were summarized using LLaMA-3.1-8B5 to remove noise and produce cleaner positives. During training, the model was optimized using margin ranking loss6, where each claim served as an anchor, its corresponding summarized gold evidence as the positive, and the top-100 BM25-retrieved candidates as negatives. This training strategy encouraged the model to assign higher relevance scores to gold summaries by pulling them closer to the claim in the embedding space. Crucially, this step addressed the main bottleneck in the task: retrieval quality. prediction was obtained by:
The final decision was produced by FacebookAI/roberta-large-mnli trained in a MIL setup. For every claim ∈ , we took the = 10 re-ranked evidence passages = {1, . . . , } ⊂ , and forwarded each pair (, ) through the encoder, obtaining a class-logit vector ∈ R , with = 3 (True, False, Contradiction). The snippet-level logit vectors were then pooled into a single bag-level vector (· )(, ) ∈ R before applying softmax. Each pooling operator produced a scalar score (· )(, ) for class ℓ. The final ℓ ˆ = arg max (· )(, ) ℓ
ℓ ℓmax(, ) = max ,ℓ,
1≤ ≤ ℓlse(, ) = log︁( ∑︁ exp ,ℓ .
︁) =
exp(︀ ⊤ℎ)︀ ∑︀=1 exp(︀ ⊤ℎ )︀ ( = 1024 for roberta-large).
where ℎ ∈ R is the CLS embedding of (, ) and ∈ R is a single learned parameter vector Max pooling simply takes the highest logit among all snippets for each class and ignores the rest. It is noise-tolerant, but discards any partial support from other evidences. Moreover, it performs poorly when all top- evidence snippets are irrelevant, as no useful evidence contributes to the prediction.
LogSumExp pooling is a smooth approximation of max and amplifies strong evidence while still giving weaker evidences some influence, which makes it more stable than hard max.
Attention-based pooling, used in our final submission, allows the model to focus on the most
informative snippets while ignoring irrelevant ones, without requiring an arbitrary truncation or noisy
concatenation of top-k candidates. Furthermore, it is naturally robust to variable evidence quality across
claims. In essence, it performs an additional, internal ranking of evidence snippets during classification,
letting the model weigh each piece of evidence according to its usefulness for the final decision.
4https://huggingface.co/cross-encoder/ms-marco-MiniLM-L6-v2
6https://pytorch.org/docs/stable/generated/torch.nn.MarginRankingLoss.html
Training Details To ensure comparability with Venktesh et al. [
Training was conducted on an NVIDIA L4 GPU with 22.5 GB VRAM. The model was trained for up to 10 epochs with early stopping (patience = 3), and best validation performance was reached between epochs 5 and 6. We used a batch size of 3 with gradient accumulation steps of 8 (efective batch size of 24). The learning rate was set to 2e-5, with a cosine scheduler and 10% linear warm-up. The model was compiled using torch.compile for performance, and trained using bfloat16 precision via PyTorch AMP. We used a class-weighted cross-entropy loss to handle class imbalance, which led to improved macro-F1, albeit with a slight decrease in weighted-F1. For reproducibility, all experiments were run with random seed 42.
In this section, we present the results of our main method, followed by ablation studies on the test set, and a summary of additional experiments and insights.
Results on Development Set We first evaluated our approach using three diferent pooling strategies: attention-based, LogSumExp, and max pooling. All experiments used the same retriever (all-MiniLML6-v2) and the contrastively fine-tuned cross-encoder described in Section 4. As shown in Table 2, attention-based pooling achieved the best performance, with a macro-F1 of 0.5937 and a weighted-F1 of 0.6682. LogSumExp and max pooling performed comparably but slightly below attention pooling. Based on these results, we selected the attention-based pooling model as our final submission. Ablation Studies on Test Set To better understand the impact of individual components, we conducted ablation studies on the test set (see Table 3). Specifically, we varied the retriever (all-MiniLM-L6-v2 vs. BM25) and the re-ranker (fine-tuned vs. vanilla) across the three pooling strategies. Several key observations emerged: • Retriever quality matters: The bi-encoder (MiniLM) consistently outperformed BM25 across all pooling methods. Furthermore, retrieving the top-100 initial candidates using pre-computed MiniLM embeddings with a FAISS index was approximately six times faster than BM25 retrieval, making it significantly more practical for large-scale applications. • Contrastive fine-tuning helps: In all configurations using MiniLM, the fine-tuned re-ranker outperformed the vanilla version, confirming the efectiveness of our contrastive training strategy with summarized gold evidences. At this point, it is worth noting that we used all top-100 retrieved evidence snippets as negatives during training. This simplification is theoretically suboptimal, since some of these snippets may actually be relevant, and treating them as negatives could push useful evidence away from the claim in the embedding space. Due to the absence of labels (i.e., which specific snippets are truly supportive for a given claim), we adhered to this heuristic approach. • Pooling strategy robustness: LogSumExp pooling achieved the highest macro-F1 (0.5213), suggesting that it generalized slightly better to the test set than attention or max pooling.
Interestingly, when using BM25 for retrieval, the vanilla re-ranker occasionally performed better than its fine-tuned counterpart, likely due to BM25 returning noisier candidates that disrupted the ifne-tuned model’s learned ranking behavior. Overall, the results afirm the benefits of our modular architecture, particularly the fine-tuned re-ranking step and the use of attention-based or LogSumExp pooling in a MIL framework. When inspecting the classwise F1 scores (Table 3) together with the confusion matrices in Figure 2, three clear trends appear:
First, the False label is by far the easiest to recognize. All three pooling strategies achieve an F1 between 0.75 and 0.78, which is driven by the large number of correct False predictions (≈ 1,700 instances in every matrix) and relatively few confusions with the other classes. Because this class accounts for roughly two-thirds of the test set, its high F1 score drives the micro- and weighted-F1 (macro-F1 only proportional).
Second, the True label benefits most from the smoother LogSumExp pooling. Compared with attention pooling, LogSumExp leaves the number of True→False errors virtually unchanged (252 vs. 254) but substantially increases correct True→True hits (264 → 338) and cuts True→Conflicting confusions (201 → 125), raising the class-wise F1 from 0.4087 to 0.4489. In contrast, Max pooling sacrifices True accuracy (F1=0.3594) by sending many true claims to the Conflicting bucket (260 instances). This indicates that its hard selection amplifies noise in borderline evidence.
Third, the Conflicting label remains the hardest case overall. Even the best setting (Max pooling, F1=0.4104) still misclassifies roughly half of the conflicting claims, mostly as False. This corroborates the qualitative observation that distinguishing genuinely contradictory evidence from merely missing or weak evidence is challenging. The models often fall back to the majority False decision when evidence is ambiguous.
In general, LogSumExp ofers the best balance between True and False, whereas Max pooling excels at recognizing Conflicting cases, and attention pooling sits in-between. These findings highlight a trade-of: improving minority classes (True, Conflicting) comes at a slight cost to the False performance, and vice-versa.
We explored a number of additional modeling strategies:
• Claim decomposition: Incorporating decomposed claims (as proposed in Venktesh et al. [
This paper presented a three-stage architecture for numerical claim verification in the CheckThat! 2025 lab. We first retrieve evidence passages with a dense bi-encoder stored in a FAISS index, then refine the candidate list with a contrastively fine-tuned cross-encoder, and finally classify each claim in a MIL framework based on roberta-large-mnli that aggregates snippet-level logits through alternative pooling operators. Evaluated on the QuanTemp dataset, the complete system reaches a macro-F1 of 0.5213 and a weighted-F1 of 0.6280 with LogSumExp pooling, which outperforms concatenation-based baselines and demonstrates that pooling-based MIL ofers a viable alternative to simplistic evidence concatenation. Ablation studies confirm that dense retrieval ofers both higher accuracy and a six-fold speed-up over BM25, while the contrastive fine-tuning of the re-ranker consistently enhances ranking quality.
Despite these gains, the overall macro-F1 is still capped by the recall of the retrieval stage, which remains the main bottleneck of this task. Future work could therefore focus on retrieval-centric improvements, including late-interaction models such as ColBERT that combine token-level matching with eficient vector search. In addition, manually labeling the truly relevant evidence snippets for each claim would allow us to avoid treating helpful passages as negatives during contrastive training, providing much cleaner learning signals for the re-ranker. 8https://sbert.net/examples/sentence_transformer/applications/retrieve_rerank/README.html# pre-trained-cross-encoders-re-ranker
During the preparation of this work, we used ChatGPT-4o to assist with light editing tasks such as grammar correction, writing style improvement, and occasional rephrasing. At no point was any section of the paper fully generated by the tool. All content and ideas originate from us. All outputs from the AI assistant were critically reviewed, revised where necessary, and verified by us. We take full responsibility for the content of this publication and confirm that the use of AI-assisted tools is in accordance with the CEUR-WS policy on AI-assisted writing.
This research work was supported by the National Research Center for Applied Cybersecurity ATHENE.
ATHENE is funded jointly by the German Federal Ministry of Research, Technology and Space and the
Hessian Ministry of Science and Research, Arts and Culture.
[10] K. Yasser, M. Kutlu, T. Elsayed, bigir at clef 2018: Detection and verification of check-worthy
political claims., in: CLEF (Working Notes), 2018.
[11] D. Wang, J. Simonsen, B. Larseny, C. Lioma, The copenhagen team participation in the factuality
task of the competition of automatic identification and verification of claims in political debates of
the clef-2018 fact checking lab, Cappellato et al.[
Conference on Information Retrieval, Springer, 2020, pp. 359–366. [15] S. Gurrapu, L. Huang, F. A. Batarseh, Exclaim: Explainable neural claim verification using rationalization, in: 2022 IEEE 29th Annual Software Technology Conference (STC), IEEE, 2022, pp. 19–26. [16] Y. Chen, H. Liu, Y. Liu, R. Yang, H. Yuan, Y. Fu, P. Zhou, Q. Chen, J. Caverlee, I. Li, Graphcheck: Breaking long-term text barriers with extracted knowledge graph-powered fact-checking, arXiv preprint arXiv:2502.16514 (2025). [17] G. Fenza, D. Furno, V. Loia, P. P. Trotta, Multi-llm agents architecture for claim verification (2025). [18] J. Thorne, A. Vlachos, C. Christodoulopoulos, A. Mittal, Fever: a large-scale dataset for fact extraction and verification, arXiv preprint arXiv:1803.05355 (2018).