The fast and large-scale spread of information through social networks and digital platforms has become an important global issue for today's societies, making automated fact-checking necessary. This paper presents the contribution of the R2I1 and MoFED2 teams from the LIS Laboratory3 to the task of automated claim relevance estimation, in the context of the 2025 CheckThat! Lab, specifically Task 3: Fact-Checking Numerical Claims 4. This task focuses on the verification of claims expressed in Arabic, English, and Spanish, particularly those involving numerical data or temporal references. In this study, we explore the efectiveness of recent open-source large language models (LLMs), such as Mistral5 and Qwen6, for automating the fact-checking of numerical claims. We propose a two-stage pipeline that incorporates these LLMs into the fact-checking process: evidence retrieval and veracity prediction. First, we employ the QwQ-32B model to automatically generate questions from each claim, guiding the retrieval of relevant evidence from the corpus provided for Task 3. Second, we fine-tune the Mistral-Small-24B-Instruct-2501 model using the LoRA (Low-Rank Adaptation) technique to predict the veracity of each claim. This hybrid approach is designed to enhance both the performance and eficiency of the fact-checking pipeline. Despite variations in performance across languages, our method achieved outstanding results, ranking first in all 3 languages : Spanish, English and Arabic. The multilingual nature of the datasets played a crucial role in improving the generalizability of claim validation across linguistic contexts. Our approach obtained macro-F1 scores of 0.503 for Spanish, 0.595 for English and an exceptional 0.960 for Arabic, significantly outperforming the second-best Arabic score of 0.635. These results not only underscore the eficacy of leveraging open-source LLMs for fact-checking, but also contribute to ongoing research in claim detection. They further highlight the importance of language-specific adaptations and the potential of multilingual strategies in the development of robust, automated fact-checking systems.
The issue of Big Data often refers to a situation where the speed at which data is generated and
disseminated far exceeds current computational and processing capacities. This results in a gap
between the massive production of data and the ability to analyze or exploit it efectively in real
time. A similar phenomenon occurs in the field of information verification: the rapid proliferation of
digital content, particularly on social media, makes the swift and systematic validation of circulating
claims extremely dificult [
Fact-checking techniques, initially developed within the field of journalism, aim to evaluate the
veracity of information by systematically comparing claims with verified and trustworthy sources. In
recent years, these techniques have been increasingly adapted and enhanced through computational
approaches. Automated fact-checking generally follows a standardized three-step pipeline: claim
detection, evidence retrieval, and verdict prediction [
However, automating fact-checking presents substantial challenges, especially when dealing with
complex claims that require nuanced reasoning and contextual understanding. This complexity is
further exacerbated when claims include numerical information, which are often perceived as more
credible due to the numeric-truth efect—a cognitive bias where numbers enhance perceived accuracy
[
In this paper, we present our participation in CheckThat! 2025 Task 3: Fact-Checking Numerical Claims3, a challenge that focuses on verifying claims involving numerical quantities and temporal expressions. The main objective of this task is to assess the veracity of claims containing explicit or implicit quantitative or temporal information by classifying them as True, False, or Conflicting, based on evidence retrieved from a predefined corpus. To address this task, we propose a two-stage fact-checking framework specifically tailored to the challenges posed by numerical and temporal claims. The first stage consists of an evidence retrieval module, where we automatically generate natural language questions from each claim using the multilingual large language model (LLM) QWQ-32B. These questions guide the retrieval of the most relevant supporting or contradicting documents from the corpus. In the second stage, we implement a veracity prediction module, in which the model Mistral-Small-24B-Instruct-2501 is fine-tuned on the task’s training data to classify the claims based on the retrieved evidence. Our study builds on a key hypothesis shaped by open-source LLM democratization: Are LLMs capable of performing accurate fact-checking on the dataset provided in CheckThat! 2025 Task 3?
The remainder of this paper is organized as follows. Section 2 provides a short review of some related work, situating our contribution within the current research on fact-checking. Section 3 presents the methodology, detailing the LLMs experimented with and the prompts employed. Section 4 discusses the experimental results, highlighting key findings and observed limitations. Finally, Section 5 concludes the paper by summarizing the main insights and outlining directions for future research.
Although an increasing number of studies have been conducted to enhance fact-checking methods [
To the best of our knowledge, only a few prior works have directly addressed the fact-checking
of numerical claims. For example, Wallat et al. [
Building on the state of the art and existing literature on Fact-Checking Claims, while adopting a somewhat diferent perspective regarding the formulation of our research question, We investigate the capability of open-source LLMs to accurately perform fact-checking on the specific dataset provided in CheckThat! 2025 Task 3.
Our goal is to apply large open-source language models to two key stages of the fact-checking process, evidence retrieval and veracity prediction, for predicting the label of numerical claims. This section introduces the dataset, the Open source LLMs Used in Experiments, and the pipeline of the proposed method followed by our models to fact-check claims. Since we use a shared pipeline across all three languages, we focus on the English setup here; the setups for Arabic and Spanish are analogous, with prompts adapted to each language while maintaining the same format.
The CheckThat! Lab 2025 task 3 [
Open-source LLMs provide significant advantages in terms of cost-efectiveness, transparency and community collaboration. Therefore, we utilize two open-source LLMs, along with an embedding model developed upon LLM foundations, as outlined in Table 4.
QwQ-32B4 is the reasoning-focused model from the Qwen series. Unlike conventional instructiontuned models, QwQ-32B demonstrates enhanced capabilities in reasoning and complex problem-solving tasks. It performs competitively with state-of-the-art models such as DeepSeek-R1 and o1-mini, making
it well-suited for fact-checking tasks that require logical inference. This model is specifically employed to perform the question generation task as part of our inference pipeline.
Linq-Embed-Mistral5 builds upon the foundations of E5-mistral-7b-instruct and Mistral-7B-v0.1. It
demonstrates strong retrieval performance, ranking 2nd on the MTEB benchmark leaderboard [
Mistral Small 36 (also referred to as 2501) is a compact model with 24B parameters, making it one of the most capable models in the sub-70B category. Developed by Mistral AI, it optimizes transformerbased architectures for language tasks, combining high performance with computational eficiency. Within our inference pipeline, this model is specifically responsible for assessing the veracity of the claim based on the retrieved evidences, claim and generated questions.
Figure 1 presents our proposed pipeline for fact-checking, which follows a two-stage architecture: evidence retrieval and veracity prediction. The input to the system consists of a claim to be verified and a corpus of evidence provided by the task organizers, which serves as a knowledge base containing relevant information for evaluating claims. In the first stage, the system retrieves the top-k most relevant pieces of evidence from the corpus by applying a decomposition method to the input claims. In the second stage, veracity prediction, the input includes the retrieved evidence, the original claim, and the generated questions. These are fed into a fine-tuned LLM that has been pre-trained for the veracity prediction task using the training dataset to produce a final verdict, reflecting the claim’s degree of validity. To ensure consistency, the same pipeline described above is used to retrieve evidences for the training, development, and test sets. We do not use the generated questions, retrieved evidences, or golden labels provided by the task organizers at any point in this process.
Although the Arabic dataset contains only two labels (True and False), a three-label prompt structure, as shown in Figure 4, is maintained to ensure consistency across diferent languages. While the prompt suggests the possibility of three output labels, the fine-tuned model, trained solely on two-label data, produces only two labels during evaluation on the development set. This behavior does not result in any errors or require additional normalization, and the model still achieves high performance.
Table 5 presents our model configurations (LIS system) in comparison to the baseline system provided by the organizers. Key diferences lie in the evidence retrieval stage, where we employ claim-augmented retrieval queries (combining the original claim with generated questions), as opposed to the baseline system which uses question-only queries. For retrieval, we use Linq-Embed-Mistral, which selects the most relevant evidence based on cosine similarity, rather than relying on BM25 followed by reranking with paraphrase-MiniLM-L6-v2 as in the baseline. For question generation, our system utilizes the QwQ32B model, while the baseline employs GPT-3.5. In the veracity prediction stage, our system leverages the powerful Mistral-Small-24B-Instruct-2501, whereas the baseline is based on FinQA-Roberta-Large. 3.3.1. Evidence Retrieval
Question Generation
Reranking
Veracity Prediction GPT-3.5 QwQ-32B paraphrase-MiniLM-L6-v2
FinQA-Roberta-Large
Mistral-Small-24B-Instruct-2501
As presented in Figure 1, the claim verification system is based on a two-stage inference pipeline. The
ifrst stage consists of retrieving relevant evidence from the corpus of evidences to either support or
refute a given claim. Inspired by recent advances in generative retrieval methods [
Once the questions are generated, each question is concatenated with its corresponding claim to form a query. This query is then embedded using our embedding model (Linq-Embed-Mistral) and compared against the corpus to retrieve the most relevant piece of evidence (top-1 retrieval) based on cosine similarity. This approach ensures that the selected evidence is contextually aligned with both the claim and the generated question. An example of this retrieval process is illustrated in Figure 3. 3.3.2. Veracity Prediction In the same vein, and as shown in Figure 1, the second stage handles the veracity prediction task based on the generated question, the retrieved evidence, and the original claim as inputs to predict the veracity of the claim. We employ an instruction-following LLM, specifically the fine-tuned Mistral-Small-24B-Instruct-2501 model, to perform this task. This model predicts the final veracity label (True,, False or Conflicting) based on the combined context. The prompt structure used for veracity prediction is illustrated in Figure 4.
Figure 2 illustrates a concrete example of an instruction prompt alongside its generated output for the task of question generation. In this figure, the instruction is highlighted in bold text, providing clear guidance to the model. The italicized text presents the claim or statement on which the question is based. Finally, the question produced by the model is shown in blue, demonstrating how the model interprets and transforms the input prompt into a relevant query. This visual representation helps to clarify the interaction between the instruction, the claim, and the model’s generated response. # Instructions You are a fact-checker. Your general motivation is to verify a given claim. You are at the beginning of the fact-checking process, meaning you have just received the claim, optionally with some additional metadata (such as the date of the claim or the author), if available. Your task now is to prepare the fact-check. That means: 1. Begin with an interpretation of the claim. As part of the interpretation, list the key points of the claim as a list of reformulated sub-claims. 2. Then, analyze what information is missing. 3. Finally, present a complete, numbered list of 5 questions: These are questions that explore the truthfulness of the claim and that we need to answer in order to factually verify the claim. IMPORTANT: Follow these rules: * Phrase each question so that it can be understood independently and without additional context. Be explicit and do not use pronouns or generic terms in place of names or objects. * Enclose each question in backticks like this. # Examples Claim: "The new Food Bill in New Zealand bans gardening" Good Question: “Did the New Zealand government pass a food bill that restricted gardening activities for its citizens?” Bad Question: “Did the government pass a bill?” Bad Question: “Did the bill restrict activities?” # The Claim french president emmanuel macron is around 25 years younger than his wife, brigitte, and they started dating when he was a teenager and she was one of his teachers. her daughter, laurence, was once one of his classmates, and later became his stepdaughter.
Output | Generated questions: “What is the exact age diference between French President Emmanuel Macron and his wife Brigitte Macron?” “Did Laurence, Brigitte Macron’s daughter, legally become Emmanuel Macron’s stepdaughter through the marriage of Brigitte Macron and Emmanuel Macron?” “Did Emmanuel Macron and Brigitte Macron begin their romantic relationship while Macron was still a teenager?” “Was Brigitte Macron serving as a teacher at the same educational institution where Emmanuel Macron was a student when their relationship began?” “Was Laurence, Brigitte Macron’s daughter from a previous relationship, enrolled as a student in the same school class as Emmanuel Macron before he married Brigitte?”
Figure 3 illustrates the process of query formation by integrating a claim with a generated question aimed at retrieving relevant evidence. In this example, the bold text represents the instruction guiding the task, while the italic text shows the claim under verification. The gray text displays the most pertinent evidence retrieved from the data source, and the blue text highlights the questions generated by the model to support the evidence search. This structured approach facilitates targeted evidence gathering for fact-checking purposes.
Query:french president emmanuel macron is around 25 years younger than his wife, brigitte, and they started dating when he was a teenager and she was one of his teachers. her daughter, laurence, was once one of his classmates, and later became his stepdaughter. What is the exact age diference between French President Emmanuel Macron and his wife Brigitte Macron? Output | Relevant evidence 1: Emmanuel, who became president in 2017, and Brigitte first met when the now-president was in high school. In a November 2023 interview with Paris Match — translated by The Independent — Brigitte shared how she never thought of entertaining a relationship with her student, who was 15 at the time. Brigitte and Emmanuel met when he was a student at Le Providence, a Catholic school in northern France. Brigitte worked as a drama teacher and was 39 years old at the time, while Emmanuel was 15. The president’s parents originally thought he was seeing Brigitte’s daughter, but when they found out he had feelings for the teacher, they sent him to boarding school. French President Emmanuel Macron and his wife, Brigitte, have been married for nearly two decades, but their relationship dates back even further.
Query:french president emmanuel macron is around 25 years younger than his wife, brigitte, and they started dating when he was a teenager and she was one of his teachers. her daughter, laurence, was once one of his classmates, and later became his stepdaughter. Was Brigitte Macron serving as a teacher at the same educational institution where Emmanuel Macron was a student when their relationship began? Output | Relevant evidence 2: Emmanuel Macron’s wife and former teacher thought he would “fall in love with someone his [own] age” after he was moved from the school where she taught him. Mrs Macron said her older siblings used to joke about the gossip. She divorced from her estranged husband, André-Louis Auzière, a banker, in 2006 and married Mr Macron a year later, when he was a young civil servant. The French president was 15 when he fell for his married drama teacher, Brigitte Auzière, then 40, at the Catholic Providence school in Amiens in the early 1990s. Her daughter Laurence was the future French president’s classmate.
Query:french president emmanuel macron is around 25 years younger than his wife, brigitte, and they started dating when he was a teenager and she was one of his teachers. her daughter, laurence, was once one of his classmates, and later became his stepdaughter. Question k and ∈ {3, 5}
Figure 4 presents an example of an instruction prompt and output for veracity prediction. The bold text corresponds to the instruction given to the model, the italic text is the claim to be verified, while the gray text represents the extracted evidence deemed relevant. Finally, the blue text includes the questions generated by the model as well as its final prediction regarding the claim’s veracity. This ifgure clearly illustrates the complete process of automatic claim verification based on evidence and generated questions.
# Instructions You are a fact-checker. You have done a fact-check to verify a Claim based on the provided questionanswer pair evidence.
Your task is to predict the verdict of a claim based on the provided question-answer pair evidence whether the Claim is one of the labels: ’True’, ’False’, ’Conflicting’. Do this by following: - Respond "True" only if the relevant evidence fully or almost fully supports and verifies the claim as correct. - Respond "False" if: - The relevant evidence contradicts or disproves the claim. - The claim is misleading based on the relevant evidence. - The evidence is too weak or insuficient to support the claim. - Respond "Conflicting" if the evidence is ambiguous, incomplete, or inconclusive, making it impossible to determine if the claim is fully true or false.
Always adhere to the following rules: - Use information only from the recorded evidence: Avoid inserting information that is not implied by the evidence. You may use commonsense knowledge, though. - Avoid repeating yourself.
Claim: french president emmanuel macron is around 25 years younger than his wife, brigitte, and they started dating when he was a teenager and she was one of his teachers. her daughter, laurence, was once one of his classmates, and later became his stepdaughter.
Q1: “What is the exact age diference between French President Emmanuel Macron and his wife Brigitte Macron?” A1: Emmanuel, who became president in 2017, and Brigitte first met when the now-president was in high school. In a November 2023 interview with Paris Match — translated by The Independent — Brigitte shared how she never thought of entertaining a relationship with her student, who was 15 at the time. Brigitte and Emmanuel met when he was a student at Le Providence, a Catholic school in northern France. Brigitte worked as a drama teacher and was 39 years old at the time, while Emmanuel was 15. The president’s parents originally thought he was seeing Brigitte’s daughter, but when they found out he had feelings for the teacher, they sent him to boarding school. French President Emmanuel Macron and his wife, Brigitte, have been married for nearly two decades, but their relationship dates back even further.
Q2: “Was Brigitte Macron serving as a teacher at the same educational institution where Emmanuel Macron was a student when their relationship began?” A2: Emmanuel Macron’s wife and former teacher thought he would “fall in love with someone his [own] age” after he was moved from the school where she taught him. Mrs Macron said her older siblings used to joke about the gossip. She divorced from her estranged husband, André-Louis Auzière, a banker, in 2006 and married Mr Macron a year later, when he was a young civil servant. The French president was 15 when he fell for his married drama teacher, Brigitte Auzière, then 40, at the Catholic Providence school in Amiens in the early 1990s. Her daughter Laurence was the future French president’s classmate.
Output | Verdict: True
This section presents the results of our implementation, including the hyperparameter configurations used to fine-tune Mistral-Small-24B-Instruct-2501 for veracity prediction and to generate questions with QwQ-32B, as well as the environment setup and our experiment results and discussion.
The hyperparameters used for fine-tuning the LLMs in our experiments are summarized in Table 6.
To enable eficient fine-tuning, we employed the Low-Rank Adaptation (LoRA) technique [
Table 6 summarizes the hyperparameters used for fine-tuning a model for veracity prediction. The training was conducted over 2 epochs with a batch size of 2 and gradient accumulation steps of 4, using the AdamW 8-bit optimizer. Key settings include a learning rate of 2e-4, weight decay of 0.01, and a warmup of 5 steps. A temperature of 0.1 was applied for output control. LoRA-based fine-tuning was used with alpha set to 16, dropout to 0.1, and rank to 64, enabling eficient and scalable parameter adaptation.
4.2.1. English datasets
Results on the development set show that our proposed model achieved nearly equivalent performance in both the 3-question and 5-question scenarios, with the 3-question setup performing slightly better by about 0.2%. Notably, both scenarios outperformed the baseline by approximately 3%.
Due to the task requirement of selecting only one best-performing model for the final evaluation on the test set, we submitted the proposed 3-question model. This model achieved a macro-F1 score of 59.54%, ranking first, ahead of all other participating teams. The full leaderboard is published on the oficial CLEF website 7. Compared to the second-ranked team, our system outperformed by approximately 3%, and exceeded the performance of the lowest-ranked team by about 24%. Notably, this performance was approximately 1.5% lower than its result on the development set. However, we were unable to compare this result against the baseline on the test set, as the oficial baseline results for Task 3 have not yet been released. 4.2.2. Spanish and Arabic datasets Table 10 presents the results obtained from various configurations evaluated on the Spanish and Arabic datasets of the CheckThat! 2025 Task 3, covering both the development and test sets. The evaluation was conducted based on two scenarios regarding the number of generated questions per claim—3 questions versus 5 questions—using the Mistral-Small-24B-Instruct-2501 model, which was fine-tuned on the training set. The evaluation metrics include macro-F1 and class-wise F1 scores for each label. However, unlike the English dataset, the authors did not provide baseline model results for comparison.
On the development set, the macro-F1 scores indicate that, similar to the English results, Arabic achieved nearly equivalent performance in both the 3-question and 5-question scenarios, with the 3-question configuration slightly outperforming by about 1%. Notably, the 3-question setup yielded almost perfect performance, reaching approximately 96%. For Spanish, using 3 questions resulted in a performance improvement of over 10% compared to the 5-question scenario, suggesting that a smaller number of questions may help retrieve more accurate and relevant evidence.
Due to the task requirement that only one best-performing model per language could be submitted for test set evaluation—as was the case with the English dataset—we submitted the 3-question configuration.
In the Spanish, the model attained a macro-F1 score of 59.54%, reflecting a decrease of roughly 7% compared to its performance on the development set. Despite this, it secured first place, outperforming all other competing teams. The complete leaderboard is available on the oficial CLEF website 8. Our system surpassed the second-place team by approximately 13% and outperformed the lowest-ranked team by around 25%.
For Arabic, although a three-label prompt was used as mentioned earlier in Section 3.3, the results on the test set consistently produced only two labels without any errors or the need for normalization, the model achieved a macro-F1 score of 96.15%, showing a slight improvement of about 0.15% relative
8https://codalab.lisn.upsaclay.fr/competitions/22823#results to its development set results. It also ranked first among all participants. The full rankings can be found on the oficial CLEF website 9. Compared to the runner-up, our model exceeded their performance by roughly 33%, and outperformed the lowest-ranked team by approximately 60%.
We chose to generate three questions per claim based on consistent empirical results across Arabic,
English, and Spanish, showing this configuration to be the most efective. This finding aligns with [
This paper presents the experiments conducted by the LIS team for CheckThat! 2025 Task 3, which focuses on verifying numerical claims in English, Spanish, and Arabic. We investigated the application of large open-source language models in the two key stages of the fact-checking pipeline: evidence retrieval and veracity prediction. Our proposed pipeline, which integrates instruction-following LLMs and efective fine-tuning strategies such as LoRA. Our model consistently applies the same methodology across all three languages and is the only team to fully participate in all three. The model achieved strong performance; notably, the fine-tuned model on the training data secured first place on the oficial leaderboard, outperforming submissions from 19 participating teams across all languages. These results highlight the potential of open-source LLMs in multilingual fact-checking tasks. Furthermore, our ifndings emphasize that fine-tuning Mistral-Small-24B-Instruct-2501 model, yields significantly better performance compared to NLI models like FinQA-Roberta-Large, which was used as the baseline on the English development set. This demonstrates the benefit of leveraging both model scale and task-specific adaptation in complex fact-checking scenarios involving numerical claims.
However, many potentially promising open-source LLMs remain unexplored in this experiment. The number of test questions is still limited to only one configuration: 3 questions. Additionally, the current fine-tuning approach has been limited to monolingual language models. In future work, we plan to conduct experiments on a broader range of open-source LLMs as well as with more diverse question configurations, and to explore fine-tuning multilingual models, which may yield better results compared to monolingual ones. Furthermore, although the Arabic dataset includes only two labels (True and False), the current prompt structure uses three possible verdicts to ensure consistency across languages. This design choice may lead the model to generate non-applicable verdicts for certain Arabic claims, potentially afecting system behavior. The impact of this discrepancy will be examined to determine whether normalization techniques—such as mapping the model’s output to the available binary labels—can enhance reliability and alignment with the ground truth.
This work was supported by the Eifel Excellence Scholarship awarded to Quy Thanh Le by the French government, as well as by the R2I and MoFED teams of the LIS Laboratory.
During the preparation of this work, the author(s) used GPT-4 for grammar and spelling checks, as well as for paraphrasing and rewording. After using these tools, the author(s) reviewed and edited the content as needed, and take full responsibility for the publication’s content.