In recent years, Automatic Fact Checking (AFC) has assumed a significant role as an instrument to identify fake news. AFC is a process that verifies the truthfulness and accuracy of information, claims, and data contained in a text or speech. The focus is on debunking disinformation and misinformation, intercepting errors, and verifying sources and facts.

Automated fact-checking uses AI tools to identify, verify, and respond to misleading claims, using techniques based on natural language processing, machine learning, knowledge representation, and databases to automatically predict the truthfulness of claims [1]. This is a complex process that involves searching, interpreting, and assessing information. As discussed in [1] a NLP framework for automated fact-checking consists of three stages: claim detection to identify claims that require verification; evidence retrieval to find sources supporting or refuting the claim; and claim verification to assess the truthfulness of the claim based on the retrieved evidence.

At first, automating the fact-checking process has been discussed in the context of computational journalism in works like [2], and has received significant attention in the computational linguistics and, in general, the artifi-cial intelligence communities, surveyed in [1] and more recently in [3] and [4]. In particular, in [1] the authors expose a survey on the topic, describing the early developments that were surveyed in [5], which is an exhaustive overview of the subject.

As with most machine learning paradigms [1], stateof-the-art methods require datasets and benchmarks.

One of the most impactful campaigns for collecting a large-scale benchmark is FEVER (Fact Extraction and VERification) [6]. In this context, fact-checking involves verifying whether a claim is supported by one or more pieces of evidence. FEVER is a publicly available dataset designed for claim verification against textual sources. It comprises about 180K claims generated by altering sentences extracted from Wikipedia. The claims are classified into three categories: Supported (a piece of evidence exists and it supports the claim), Refutes (a piece of evidence exists and it contradicts the claim), or NotE-noughInfo (there is insufficient evidence to verify the claim). The challenge, therefore, is to retrieve the relevant evidence and verify the accuracy of the claims, categorizing them with the correct label.

Many works like FEVER have recently focused on building data and datasets for the task of Fact Verification, achieving very good results [7,8,9,10,11,12]. However, all of these datasets are designed for the English language. Although multilingual models exist (e.g., in [13,14]), finetuning a model on a specific language, pre-training it for a specific task and use case, could lead to a significant decline in quality if applied to another language. Few studies have worked on training models for languages other than English. An example is the work presented in [15], which focuses on developing automated claim detection for Dutch-language fact-checkers.

In this work, we propose a FEVER-IT dataset in which the FEVER dataset has been translated into Italian to train the model for the Italian language. Inspired by SQUAD-IT [16] and MSCOCO-IT [17], we worked to obtain quality data. Although the training set may be affected by translation errors, the test set will not, as it is composed of manually validated data. Furthermore, while the original FEVER dataset contained evidence only for Supports and Refutes, in this work we have also added and translated examples for the NotEnoughInfo category using the heuristics proposed in [18]. This work extends the experience described in [19], where translations were done using Google API, by using publicly available models ( [20]) and adding data for the NotEnoughInfo category.

The contribution of this work is twofold. Firstly, we release FEVER-IT, a corpus with 228K claims each associated with at least one (possibly useful) piece of evidence, including a test set of 2,000 manually validated claims. In addition, we fine-tuned and validated a state-of-theart model, LLaMA3 [14], on both the original English dataset and the Italian dataset. While this provides a high-performance model ready for the task in both languages, the primary goal is to assess whether the quality of the Italian data is comparable to the English one. By training the model separately on each dataset, we can evaluate its stability: if the model performs similarly on the manually validated Italian dataset and the English test set, we can conclude that the quality of the Italian data is on par with the English data.

Additionally, we want to assess whether using an Italian train dataset, despite the noise from automatic translation, is truly beneficial. LLMs like LLaMA3 can already perform tasks in other languages through zero-shot or few-shot learning, without requiring fine-tuning on a specific dataset, especially if that dataset is noisy. Therefore, we aim to compare the performance on the test set between a LLaMA3 model that hasn't been fine-tuned on the noisy Italian data and one that has been fine-tuned, to determine whether fine-tuning actually improves results or if the model performs on par or better without it.

The experimental results show that the model without fine-tuning achieves an average accuracy of only about 45%. Fine-tuning on the English dataset yields about 90% mean accuracy, while fine-tuning on the Italian dataset results in a percentage quite similar to the fine-tuned English model and much greater than testing without fine-tuning 1 .

The remainder of the paper is organized as follows: Section 2 discusses related work, Section 3 presents FEVER-IT, Section 4 details the experimental measures, and Section 5 provides the conclusions.

One of the pioneering works in autonomous factchecking was conducted by [21], which proposed creating publicly available datasets and developing automated systems using natural language processing technologies. Recent challenges such as CheckThat! at CLEF [10,11,12] and Fever [7,8,9] from 2018 have advanced fact-checking tasks by leveraging advanced approaches and integrating Large Language Models (LLMs) like BERT and GPT. These models represent the current state of the art in many Natural Language Processing tasks, including fact-checking. Notable examples of such technology include FacTeR-Check [22], a multilingual architecture for semi-automated fact-checking and hoax propagation analysis using the XLM-RoBERTa Transformer [13], and FACT-GPT [23], a framework that automates the claimmatching phase of fact-checking using LLMs to identify social media content that supports or contradicts claims previously debunked by fact-checkers.

The success of these systems is largely due to the capabilities of LLMs as summarized in [3], which are neural models based on the Transformer architecture. Specifically, decoder-based architectures, such as GPT [24], GPT-3 [25], and LLaMA [14], generate output sequences in an auto-regressive manner. These models have demonstrated impressive capabilities following pre-training on large collections of documents. One notable outcome is few-shot learning, where models can adapt to new tasks with only a few examples [25], greatly enhancing their flexibility and applicability.

When new annotated data is available, fine-tuning further enhances a model's capabilities. This process involves taking the pre-trained base model and training it on a smaller, specialized dataset relevant to the desired task. Parameter Efficient Fine-Tuning (PEFT) is an optimized technique that involves training only a small portion of the weights, typically by adding a new layer to the model. One widely used technique is LoRA [26], which adds an adapter consisting of two matrices of weights that are relatively small compared to the original model. Extremita [27] is an example of a decoder-based model fine-tuned with LoRA in Italian for multi-task executions.

Several benchmark datasets have been developed to fine-tune and evaluate fact-checking systems, typically collected by organizations like Snopes, FullFact, and Poli-tiFact. The FEVER challenge has produced four major datasets: FEVER (2018) [6], FEVER 2.0 (2019) [8], FEVER-OUS (2021) [9], and AVeriTeC (2024) [28]. These datasets range from labeled claim-evidence associations to verified claims with structured and unstructured evidence. Despite the wealth of resources available, there is a lack of large benchmark datasets in Italian. This work addresses this gap by providing a large-scale Italian resource.

As in [6], the original FEVER dataset is composed of claims that can potentially be verified against an encyclopedic resource, in this case, Wikipedia. The claims are classified into three categories: Supported, Refutes and NotEnoughInfo. For the first two categories, each claim is associated with one or more passages from Wikipedia, each specifying the page from which it was extracted. For the NotEnoughInfo category, no passages are provided because no information was found on Wikipedia to support or refute the claim. For instance, the sentence "Dan Brown is illiterate." is a claim associated with pieces of evidence such as: "Angels and Demons is a 2000 bestselling mystery-thriller novel written by American author Dan Brown and published by Pocket Books and then by Corgi Books.". These pieces of evidence prove that the claim is incorrect, so it can be classified with the label Refutes. In FEVER, a claim is thus a sentence that expresses information (true or mutated) about a target entity.

To generate the Italian dataset, we started from the dataset version2 proposed in [29], which consists of 260k claims. This version extends the original FEVER by adding evidence associated with claims justified as NotE-noughInfo in FEVER, using the heuristics in [18]. The approach involved using a search engine to retrieve potential evidence and a textual entailment system based on GPT [24]. Claims not judged as Supports or Refutes were classified as NotEnoughInfo.

This gives us examples of sentences that are closely related to the claim (according to the search engine) but neither support nor refute it. This makes it more straightforward and efficient to train and/or evaluate a classifier, even though some of the derived examples might be somewhat noisy, as they were generated through heuristics.

For the automatic translation process, we utilized MADLAD400 [20], a machine translation system based on the Transformer architecture3 , trained on MADLAD, a manually audited, general domain 3T token multilingual dataset based on CommonCrawl, spanning 419 languages. Since the Italian data are obtained through machine translation, and thus potentially incorrect as suggested in [16,17], we needed validated test data to obtain a realistic benchmark. Our hypothesis is that an LLM is robust enough to generalize from the 228k examples and recognize the relationships involved in FEVER without inheriting translation errors. However, to prevent these errors from being inherited by the model, we manually corrected the translations of the test set.

Out of the approximately 16k available test examples, three annotators were involved in verifying and correcting 2, 063 translations from the test set. The annotators focused on correcting mistakes related to the proper sentence structure in Italian, the accurate meaning of specific English words that MADLAD had translated literally, any misunderstandings of the intended meaning in Italian, and a few grammatical errors.

In some cases, translation errors do not completely undermine the examples with respect to the task's purpose. For instance, the English sentence from an evidence, "he was booked to win a third world championship at a WWE event on the night of his death" was translated into Italian as "era stato prenotato per vincere un terzo titolo mondiale in un evento della WWE la notte della sua morte". A more accurate translation would be "si pensava avrebbe vinto un terzo titolo mondiale in un evento della WWE la notte della sua morte", better capturing the verb's meaning. In other, more problematic cases, translation errors, loss of information, or introduction of hallucinations could even change the classification in the fact verification task. For example, in the claim "The Thin Red Line (1998 film) has an all-British cast.", the automatic translation was "La sottile linea rossa (The Thin Red Line) è un film del 1998.", which is incorrect because it omits the information about the cast. This detail is crucial, as its absence could lead to incorrect labeling. A quantitative analysis of the translation quality suggests that MADLAD performs well in translating simple assertive sentences such as claims. In fact, 91% of the claims were not altered by the validators, who considered them completely correct. This percentage is lower for the Wikipedia passages, dropping to 76%. This discrepancy may be due to the greater complexity of the evidence compared to the simpler sentence structures in the claims. Additionally, we reported the results in terms of BLEU score [30] for the corrected translations compared to the originals, as shown in Table 1. It should be noted that measuring the translation quality after correcting the sentences introduces a strong bias in the measurements; however, it provides a more specific idea of the translation quality, especially in understanding the potential noisiness of the training and development sentences. In this case, results of over 95% for BLEU-1 and over 92% for BLEU-4 suggest that very few terms were altered during validation, and even the grammatical patterns remained largely unchanged. At most, a few mistranslated terms needed updating, as indicated by the qualitative analysis.

Table 2 summarizes the number of examples created for the Italian dataset. In line with the original English material, the dataset is divided into training, development, and test sets, with claims categorized into Supports, Refutes, and NotEnoughInfo (NEI). The table also distinguishes between silver data (automatically translated) and gold data (manually validated). The training set consists of 228,277 claims, the development set contains 15,935 claims, and the test set has 2,063 claims. Each Italian claim or evidence is aligned with the English counterpart, facilitating future research in cross-lingual fact verification. Language Models for Fact Verification. For addressing the capabilities of Large Language Models in Fact Verification, they can be utilized through In-Context Learning techniques [31] or by directly fine-tuning the model for specific downstream tasks. In-context learning relies on the model's pre-existing knowledge acquired during pretraining and on instructions provided in natural language at inference time. This method does not involve additional training and can be categorized based on the number of examples provided: i) 0-shot Learning, where no examples are given, and the model generates responses based solely on its pre-existing knowledge and the provided instructions; ii) 1-shot Learning, where one example per class is added to provide a more precise context, helping the model better understand the task by offering a concrete reference point; iii) Few-shot Learning, where more than one example per class is provided to give the model additional contextual information during decisionmaking. When the model's pre-existing knowledge is insufficient, we can fine-tune it on the downstream task. Fine-tuning involves training the model in a traditional manner using input-output pairs (training data) to adjust its parameters. This process improves the model's performance on specific tasks, allowing it to learn from a more extensive set of examples. As a result, the model becomes more adept at handling similar queries in the future, with a focus on the specific task at hand. We thus evaluated the application of state-of-the-art LLM, namely LLAMA3 [32], by providing just the definition of the task (zero-shot) or adding an example (one-shot) or by performing fine-tuning, to demonstrate the necessity of a training dataset like the one constructed in this work, as discussed in the following section.

The goal of our experimentation is to assess the performance of a state-of-the-art LLM applied to Fact Verification. Specifically, we aim to determine whether a multilingual model maintains consistent quality when applied to both the English FEVER dataset and our Italian dataset. We utilize LLaMA3-Instruct4 , an instruction-tuned generative text model from META with 8 billion parameters, released in April 2024. This model is trained to execute specific instructions or prompts across various tasks. To ensure alignment, we evaluate the systems on the manually validated Italian test set and the same subset of 2,063 claims in the English counterpart. The model is evaluated in 0-shot and 1-shot settings to assess its capability without fine-tuning. The prompts used in English and Italian are provided in Appendix A. Additionally, we fine-tuned LLaMA3 on the English datasets from [29] and separately on the Italian datasets obtained via machine translation. Fine-tuning was conducted on an NVIDIA A100 using the LoRA technique 5 .

In FEVER, the title of the document associated with each claim often provides crucial context. For example, the claim "The University of Leicester discovered and identified the remains of a king." relies on the document titled "University of Leicester" to correctly classify the claim as Supports. To ensure the model's generalization, we will evaluate the impact of including document titles in prompts. The metrics used to analyze the results are recall, precision, accuracy, and F1 score, calculated globally and for each label (Supports, Refutes, NotEnough-Info).

The results are reported in Tables 3 and 4 for the English and Italian datasets, respectively. Each table shows whether the model underwent fine-tuning (column FT), whether a prompt without examples (0-shot) or with one example per class (1-shot) was used (column Prompt), and whether the document title was included (column Doc). Notably, if no fine-tuning was performed, the original LLaMA3-Instruct model was used. Given that the system's response can consist of multiple words, we search the output for the mention of one of the classes and associate the example with that class. If no class is identified, the result is classified as NotEnoughInfo. In general, the fine-tuned model is extremely stable, consistently outputting one of the three categories for every request. The non-fine-tuned model, on rare occasions-just a few dozen times out of 2000-produces responses that do not correspond to any of the required classes. This highlights the inherent stability of LLaMA3 while also supporting

Performance in terms of Accuracy, Precision, Recall and F1-measure of our systems on Fever-IT dataset the soundness of the results achieved.

A key finding is that the multilingual model generally achieves similar, though modest, results on English and Italian datasets without fine-tuning, with accuracy values around 0.40-0.50 and average F1 scores in the range of 0.35-0.55. This performance is relatively unstable, and the addition of an example in the prompt does not lead to significant improvements. In English, there are some improvements, but in Italian, there are fewer. We believe this is because, although LLaMA is multilingual, the percentage of Italian examples observed during training is less than 1%, making it less performant and less stable in this language.

However, when fine-tuning is applied, the results improve dramatically, with accuracy exceeding 90% in both languages. This demonstrates the utility of the translated dataset, even if it contains some noise. In this scenario, adding an example in the prompt leads to negligible but consistent improvements. Additionally, the inclusion of the document title, while sometimes causing inconsistencies in zero-shot learning, is better utilized by the fine-tuned model, leading to slight but not significant improvements. This is interesting because it suggests that the model not relying on document titles is more broadly applicable. Overall, the fine-tuned models perform significantly better, highlighting the importance of the translated dataset for achieving high accuracy in fact verification tasks in both English and Italian.

The error analysis suggests that the model sometimes inherits the mathematical reasoning limitations of the LLM. For example, the claim "Il Castello di Praga attira oltre 18 milioni di visitatori ogni anno. 6 " was given the evidence "Il castello è tra le attrazioni turistiche più visitate di Praga che attira oltre 1,8 milioni di visitatori all'anno. 7 " The model's predicted label was Refutes, while the true label was Supports. Here, the true label should be Supports since 18 million is indeed greater than 1.8 million, but the model found the numbers inconsistent. In another case, the claim "Ned Stark è stato introdotto nel 1996 in Tempesta di spade. 8 " was paired with the evidence "Introdotto nel 1996 in Il Trono di Spade, Ned è l'onorevole signore di Winterfell, un'antica fortezza nel nord del continente immaginario di Westeros. 9 " The model predicted Refutes, although the true label was Supports. The confusion here is due to the difference in the book titles, which are from the same series but are distinct works. The error analysis revealed that the model occasionally struggled with mathematical reasoning and contextual understanding, highlighting areas for future enhancement. Larger models and further fine-tuning could potentially address these issues, which remain open questions for future research.

In this work, we have introduced FEVER-IT, an Italian version of the FEVER dataset, designed to improve the training and evaluation of models for fact verification in the Italian language. Using a machine translation system, we translated a large-scale dataset of 228,000 claims/pieces of evidence pairs and manually validated 2, 000 test instances to ensure meaningful evaluations. This enabled us to fine-tune a state-of-the-art LLM, specifically LLaMA3, and assess its performance in both English and Italian.

Our experiments demonstrated that the multilingual model, without fine-tuning, performed similarly on both English and Italian datasets, though the accuracy and stability were limited. Fine-tuning significantly improved the model's performance, achieving over 90% accuracy in both languages. This underscores the importance and effectiveness of the translated dataset, even if it contains some noise.

Future work will explore the performance of larger models and further refinement of the dataset to enhance accuracy and generalization capabilities or explore more complex settings such as those described in [9].

This appendix contains the prompts used in the experiments. The prompts are provided in both Italian and English, reflecting the task-specific nature of the experiments. Each prompt begins with an explanation of the task and the meaning of the classes. In the different variants, the 0-shot setting does not include any examples, unlike the 1-shot setting. Where necessary, the name of the document from which the evidence is taken is also specified.

The following prompt is used for 0-shot learning, where the task and classes are presented without additional information.

The following prompt is used for 1-shot learning, where the task and classes are explained, and one example per class is provided. Notice that only the evidence is reported without the title of the original document. The following prompt is used for 0-shot learning, where the task and classes are explained without additional information. Each input evidence is provided with the title of its original document. The following prompt is used for 1-shot learning, where the task and classes are explained, and one example per class is provided. Each input evidence is provided with the title of its original document.