This paper introduces CheckIT!, a resource of expert fact-checked claims, filling a gap for the development of fact-checking pipelines in Italian. We further investigate the use of three state-of-the-art generative text models to create variations of claims in zero-shot settings as a data-augmentation strategy for the identification of previously fact-checked claims. Our results indicate that models struggles in varying the surface forms of the claims.
The pollution of the information ecosystem by means
of misleading or false information has reached
unprecedented levels at a global scale. This has been possible
thanks to a combination of multiple factors, among which
the collapse of (local and national) journalism; an
increasing sense of distrust in science and evidence-based facts;
and the presence of computational amplification tools
such as bots [
Manually fact-checking claims is an expensive oper- viously verified claims retrieval; (iii) claim evidence retrieval;
ation (in terms of time and efort) and in many cases, it (iv) claim veracity assessment. The figure is an adaptation
comes too late. Authors in [
In Figure 1, we present the full fact-checking verifi- This paper presents CheckIT! the first corpus of
precation pipeline [
The remainder of the paper is structured as follows: corresponding label.
Section 2 describes the data collection, the veracity label At the end of this operation, we have the following laharmonization process, and presents an analysis of the bel distributions: 1,255 claims labelled as “Vero” [True], dataset. In Section 3, we discuss the results of our para- 1,512 labelled as “Impreciso” [Imprecise], and 810 laphrases experiments with text generation tools for Ital- belled as “Falso” [“False”]. The label distribution is not ian (mT5, Camoscio, ChatGPT) as a strategy to extend perfectly balanced, with the majority class being “Imprethe variability of expressions of previously fact-checked ciso”. While on the one hand it is comforting to see that, claims. Our eforts have been mainly focused on assess- in absolute terms, politicians do not overtly lie, on the ing the quality of these generative tools. Related work is other hand it is not surprising to observe that politicians discussed in Section 4. Finally, Section 5 concludes the may manipulate data and news as a propaganda strategy paper and draws directions of further development.1 to convince the audience of their arguments. The tendency of the last 16 months is worrying as 61.14% (192 out of 314) of the claims have been fact-checked as false. 2. CheckIT! : Data Collection and The distribution of the claims over time is not well Analysis balanced as illustrated in Table 1. Early years see a rich activities, while this diminishes in more recent times (see also Figure 1 in Appendix B. Election years (2013, 2018, and 2022 for national Parliament elections; 2014 and 2019 for European Parliament elections) contain the majority of the fact-checked claims.
fact-checked claims from Pagella Politica2 and structuring them into a unified representation format. Pagella Politica is a web-based news outlet fully dedicated to fact-checking and analysis of political news in Italy since October 2012. Pagella Politica aims to provide accurate Year True ImVperrdeciciste False YYeeaarr True ImVperrdeciciste False information and it aims to empower readers with knowl- 2012* 143 121 54 2018 25 88 29 edge, fostering a deeper and more informed engagement 2013 315 294 97 2019 79 180 76 with the political landscape. To gather all claims we 22001145 124643 129515 6803 22002201 6539 15834 9952 have obtained access to Pagella Politica’s public APIs, 2016 40 73 24 2022 83 16 136 and scraped claims covering a period from October 3rd 2017 22 42 28 2023* 19 4 46 2012 to April 26th 2023. In our harmonization process, Table 1 we retained 3,577 claims out of 4,547, with 17 common at- CheckIT!: Distribution of verdict labels per year. Years marked tributes (see Table A in Appendix A for details). For each with * cover less than 12 months. claim, the evidence text has been split into sentences and all hyperlinks have been extracted and stored separately. When focusing on the most debated topics, the large
As for the veracity verdicts, Pagella Politica has majority of the claims (79.54%) concern four main areas: changed its labelling scheme since its firsts appearance: Social Issues (983), Economics (919), Institutions (599), they have moved from a five-label scheme (“Vero” [True], Foreign Afairs (350), clearly corresponding to topics of “Ni” [Mostly true], “C’eri quasi” [Half True], “Pinocchio public interest as they directly afect the lives of citizens andante” [False], “Panzana pazzesca” [Pants on fire]) to and the working of the democratic institutions. Some verbose verdicts with explanations (e.g., “the politician of these topics face peaks of fact-checking in corresponis right”). However, mapping the verbose verdicts to the dence of relevant events. For example, 21.71% of the original five labels was impossible, especially for non- claims concerning Foreign Afairs are registered in 2015 experts and for the very nuanced diference between the during the European migrant crisis3; 12.71% of claims labels “Ni” [Mostly true], “C’eri quasi” [Half True]. In ad- related to Social Issues are in 2020 during the first phases dition to this, verbose verdicts are not optimal for training of the COVID-19 pandemics; 10.63% of claims for Ecomachine learning classifiers. To avoid losing 270 of the nomics are in 2019, when the citizens’ income (“Reddito most recent claims, we decided to simplify the labeling di Cittadinanza”) was introduced. An overview of the distribution of the topics and the corresponding verdict
2https://pagellapolitica.it/fact-checking
labels is presented in Table 2. of political orientations, i.e., sovra-representation of a political party or side. The top 10 politicians (Table 3) are
Verdict quite evenly distributed among the three major political Topic True Imprecise False areas that characterizes Italy in the past 10 years: three Environment 44 59 18 for the center-left/left, three for the M5S area, and four Social Issues 291 402 290 for the center-right/right. As a way to estimate the presEconomics 291 459 169 ence of potential biases, we have run a simple machine Justice (Civil and Criminal) 62 63 30 learning experiment to estimate the prediction of the veForeign Afairs 124 163 63 racity labels from the claims themselves. Previous work IOntshtietrutions 26560 23635 9314 has shown that this is not an easy task (if even possiNot Specified 127 64 214 ble) [8, 9]. We have thus split CheckIT! into a Train (80%) and Test (20%) and trained two linear Support Vector Table 2 Machine (SVM) models. We have used a simple TF-IDF CheckIT!: Distribution of verdict labels per topic. vectorization4 in both cases. In the second experiment, we have concatenated the names of the politicians to the
CheckIT! contains 317 unique Italian politicians/public text of their claims. Both SVMs are further compared ifgures. The corpus has a very long tail, with the large with a Dummy classifier implementing majority voting. majority of politicians being attributed only one claim. Results are summarized in Table 4. An aspect to consider in this dataset concerns the popularity and the roles that politicians have. The top 10 Model Label P R Macro-F1 politicians are all prominent figures in the Italian political Dummy ITmruperecise 0.401.06 01..00 0.195 sphere. They are (former) secretary of major political par- False 0.0 0.0 ties, Prime Ministers, ministers, or popular party leaders. True 0.458 0.392 This top 10 covers 52.80% of all the fact-checked claims. SVM - claims only Imprecise 0.457 0.573 0.422 On the other hand, only 18.92% (60) of the politicians False 0.387 0.294 appear in at least 10 claims. A statistic we are not able to True 0.456 0.411 SVM - claims & politicians Imprecise 0.449 0.553 0.425 provide in full given the current version of CheckIT! is the False 0.411 0.300 distribution of the claims per political party. Although we know that there are 16 political parties, more than Table 4 1,000 claims lack this information, i.e., it was not avail- Claim veracity prediction. Underscore figures indicate the able through the APIs. Table 3 shows the distributions of best result. the verdict labels for the top 10 political figures.
Matteo Renzi Matteo Salvini Beppe Grillo Giorgia Meloni Silvio Berlusconi Luigi Di Maio Renato Brunetta Enrico Letta Alessandro Di Battista Laura Boldrini True
As expected, the results are way far from being satisfying. Although the SVMs seem to learn something, when compared to the Dummy classifier, their overall macro-F1 is well below 0.5. A slight improvement in the False class can be observed when the names of the politicians are concatenated with the claims. However, this appears to be an efect of the data split (out of 317 unique entities, 121 appear both in our train and test splits). While on one hand, these results further confirm a limited presence of bias in the data, they further support previous results on the dificulty of assessing the veracity of a claim from the claim itself, especially when it is uttered using formally correct language [10].
This battery of experiments is devoted to evaluate the use of generative language models to enrich fact-checking datasets by varying the expression of the claims. This
a range of 2-5, and stop-word removals. mt5 only available Italian model for paraphrase generation is aiknowyou/mt5-base-it-paraphraser. This model is based on mt5 and fine-tuned on Tapaco and STS Benchmark datasets for Paraphrasing. mt5 [11] is a multilingual variant of T5 [12] that was pre-trained on a new Common Crawl-based dataset covering 101 languages. The TaPaCo Corpus, used for fine-tuning, is a freely available paraphrase corpus for 73 languages extracted from the Tatoeba database. data augmentation approach plays a pivotal role for the for Machine Translation, will assess the overlap of ndevelopment of robust systems for the identification of grams (word sequences) between the claim and the parapreviously fact-checked claims (step (ii) in Figure 1), and phrases as a proxy for text variation. Similarly, ROUGE,6 thus reducing the manual workload of professional fact- which returns the overlap of n-grams and the longest checkers. In particular, we generate five alternative ver- common subsequence, will also assess the variations of sions of the CheckIT! claims using three generative mod- the generated text with respect to the original claims. els, namely mt5, Camoscio, and ChatGPT. Finally, BERTScore, which calculates the similarity between two sentences or texts by utilizing contextualized The embeddings from pre-trained language models and comparing the embeddings of overlapping words between the candidate and reference sentences, will help us to better assess the semantic similarity.
Block I: Machines vs. Claims The summary of the results is in Table 5. Camoscio produced a considerable 3.1. Evaluation Metrics number of empty paraphrases. To ensure fair comparTo assess the goodness of the generated texts, we con- isons, we excluded these empty paraphrases from the ducted a comprehensive evaluation encompassing com- metrics calculation. Overall, we notice a trend of higher parisons between the model-generated paraphrases, the variation in generation for ChatGPT. Despite the high avoriginal sentences, and paraphrases by three human an- erage cosine similarity with the original texts, ChatGPT notators. displayed better performances for creative rephrasing.
In all evaluation settings, we use four automatic met- Surprisingly, mT5 does not perform very well, as indirics, Cosine Similarity (Cos), BLEU [15], ROUGE [16], cated by the high scores across all metrics. Diferences and BERTScore [17], to gain multiple perspectives on the between the training materials and the CheckIT! data models’ performance and gauge both the fidelity and the may have had an impact. Finally, Camoscio is the worst variations with respect to the original claims exhibited performing models. Out of 1,785 possible paraphrases for by the models. In particular, Cos will return the the se- the 357 claims considered, it fails to generate an output mantic similarity between the two texts based on word 1,320 times. The few successful cases are almost exact frequency distributions. BLEU, although commonly used
ROUGE-LSum.
7All annotators are also the author of this paper.
prompting ChatGPT APIs5 with the following text: “Parafrasa le seguenti frasi: claims” (’Paraphrase the following: claims’) where “claims” are the original claims belonging to CheckIT!.
For all models, we have used the default parameters. For ChatGPT, the temperature was left to 1 and max_token to 2,000.
is based on 10% (i.e., 357) of the claims in CheckIT!. In this case, we compared the automatically-generated paraphrases against the original claims.
Camoscio The second method we used to generate The latter three are based on a subset of 50 claims paraphrases is based on instruction-based models. Specif- that have been independently paraphrased by the human ically, we used Camoscio [13], an Italian version of annotators.7 Annotators were given basic instructions Alpaca [14] obtained by instruction-tuning LlAMA on which closely resembled the prompts of Camoscio and Italian data automatically translated with ChatGPT. To ChatGPT: “Provide a paraphrase for each of the following obtain the paraphrases, we used the following prompt: sentences.” In the second evaluation block, we compare “Scrivi 5 parafrasi di questa frase: claim” (‘Write 5 para- human-generated paraphrases (a total of 150 instances phrases of this sentence: claim’) where “claim” is one of corresponding to 3 diferent variants per claim) with the the original claims belonging to CheckIT!. original claims. In the third evaluation block, we evaluate the human-generated paraphrases with respect to each other: for each data point, we compared the four metrics between all the combinations of annotators (e.g., A1 vs.
A2; A2 vs. A3; A1 vs. A3, and so on). Note that some of the metrics (i.e., ROUGE and BERTScore) are not symmetric, thus results may vary. In the fourth evaluation block, we compared the automatically-generated paraphrases against human-generated ones. 5We used GPT 3.5-turbo.
3.2. Evaluation Settings and Results repetitions of the original claims, as highlighted by the scores of the various measures and a manual inspection.
Clear evidence of this parroting behavior is shown by the BLEU score. BERT-P BERT-R BERT-F1 BLEU Cos ROUGE-1 ROUGE-2 ROUGE-L ROUGE-LS
Block II: Humans vs. Claims Scores are reported in Table 6. In general, it seems that humans introduce more superficial variations, as highlighted by BLEU and ROUGE. However, there is an increasing adherence to the original formulation of the claim among the annotators.
Notably, A1 exhibited a greater propensity for variation in their paraphrasing, while A3 tended to produce paraphrases closer to the original texts, as evidenced by the higher BLEU and ROUGE-LS. Clearly, the closer in wording to the original claim, the bigger the impact also on the more semantic oriented measures such as BERTScore and Cos. While A1 and A2 present close performances, A3 achieves the highest results. It appears that divergent interpretations of what a paraphrases of a claim is and how to do it have afected the results, suggesting that more precise instructions will be needed in the future to achieve more varied results. 0.76 Block III: Human vs. Human As we delved into the 0.80 comparison among the annotators (Table 7), we found 0.78 that A1 and A2 produced paraphrases that were notably 0.07 more similar to each other in comparison to those pro0.87 duced by A3. This clearly indicates that distinct stylistic preferences have been adopted.
Block IV: Machines vs. Humans We evaluated the quality of the generated paraphrases by comparing them to the three human-produced paraphrases, considering the latter as references. A summary of these results is presented in Table 8. Surprisingly, the automatically generated paraphrases have a higher degree of similarity and lexical overlap with the manually generated ones.
The results for Camoscio are quite unexpected, as it seems to qualify as the best second system after ChatGPT.
However, this is a distortion due to the measures and the manual paraphrases. As we have seen in Table 6, A3 is very conservative, generating paraphrases close to the original claim. This is also the behavior of Camoscio, as observed in Table 5. On the other hand, mT5 and ChatGPT appears to be more suitable candidates for this task.
and previous work has already investigated multiple
aspects. Early work has focused on detecting rumors in
Social Media [18, 19], or on the identification of the stance
of a document with respect to a claim [20, 21, 22].
Following Figure 1, the claim detection step is one of the
easiest and one of the most controversial subtask. While
the identification of claims is comparable to Attribution
Detection [23, 24], the check-worthiness status of claims
is challenging since it involves some level of
subjectivity. To address this issue, previous work has collected
data from authoritative sources run by professional
factcheckers (e.g., PolitiFact, Snopes) or have seen the
direct involvement of human experts for the veracity
labelling [
Evidence retrieval requires the identification of relevant passages from external resources that can be used to verify the claim. Two mainstream automatic verification methods are employed: Stance Detection and Natural Language Inference (NLI) [25, 31, 32]. They make use of unstructured data (i.e., textual sources) and assume that evidence is available for every claim and make a closed world world assumption, i.e., evidence is available only in one source. Complimentary methods make use of structured data, where evidence can be retrieved inside a knowledge graph [33].
Each of the subtasks involved in the fact-checking pipeline is framed as a classification task, with a varying number of labels: from a binary classification for the check-worthiness, to rich multi-class classification tasks for the veracity of the claim. For CheckIT! , we have opted for a three-way classification of the claim, in line with most of the previous work. The advantage of (more) finegrained veracity classifiers is that it allows to capture also misleading or imprecise information and avoiding to reduce the world into a black or white picture.
This work has introduced CheckIT!, an expert-curated fact-checked repository of claims by politicians and prominent public figures in Italy. CheckIT! covers 10 years of claims and it is the first publicly available dataset for fact-checking in Italian. In our analysis of CheckIT!, we have observed a drop in the numbers of fact-checked claims suggesting that manual fact-checking is increasingly dificult to conduct and that automated assisted tools are more and more needed.
We have conducted a preliminary investigation of three state-of-the-art automatic text generation tools for claim paraphrases. By combining multiple automatic measures, it appears that ChatGPT and mT5 are the two best candidate to further explore, while Camoscio presents non-trivial issues with respect to failure to produce an output and variations of the generated texts.
Future work will focus on three aspects: conduct a
qualitative (human-based) evaluation of the two best
models; evaluate the generated paraphrases for
previously fact-checked claim retrieval on the line of [
Giovanni Zagni, for the access to the APIs that made the data collection for CheckIT! possible.
This work has been partially supported by the EU H2020 TAILOR project, GA n. 952215. for Computational Linguistics: NAACL 2022, As- method for automatic evaluation of machine transsociation for Computational Linguistics, Seattle, lation, in: Proceedings of the 40th Annual MeetUnited States, 2022, pp. 1619–1631. URL: https: ing of the Association for Computational Linguis//aclanthology.org/2022.findings-naacl.122. doi: 10 tics, Association for Computational Linguistics, .18653/v1/2022.findings-naacl.122. Philadelphia, Pennsylvania, USA, 2002, pp. 311– [7] P. Nakov, A. Barrón-Cedeño, G. da San Mar- 318. URL: https://aclanthology.org/P02- 1040. tino, F. Alam, J. M. Struß, T. Mandl, R. Míguez, doi:10.3115/1073083.1073135. T. Caselli, M. Kutlu, W. Zaghouani, C. Li, S. Shaar, [16] C.-Y. Lin, ROUGE: A package for automatic evalG. K. Shahi, H. Mubarak, A. Nikolov, N. Bab- uation of summaries, in: Text Summarization ulkov, Y. S. Kartal, M. Wiegand, M. Siegel, J. Köh- Branches Out, Association for Computational Linler, Overview of the clef–2022 checkthat! lab guistics, Barcelona, Spain, 2004, pp. 74–81. URL: on fighting the covid-19 infodemic and fake news https://aclanthology.org/W04-1013. detection, in: A. Barrón-Cedeño, G. Da San Mar- [17] T. Zhang, V. Kishore, F. Wu, K. Q. Weinberger, tino, M. Degli Esposti, F. Sebastiani, C. Macdon- Y. Artzi, Bertscore: Evaluating text generation with ald, G. Pasi, A. Hanbury, M. Potthast, G. Faggioli, BERT, in: 8th International Conference on Learning N. Ferro (Eds.), Experimental IR Meets Multilin- Representations, ICLR 2020, Addis Ababa, Ethiopia, guality, Multimodality, and Interaction, Springer April 26-30, 2020, OpenReview.net, 2020. URL: https: International Publishing, Cham, 2022, pp. 495–520. //openreview.net/forum?id=SkeHuCVFDr. [8] H. Rashkin, E. Choi, J. Y. Jang, S. Volkova, Y. Choi, [18] L. Derczynski, K. Bontcheva, M. Liakata, R. ProcTruth of varying shades: Analyzing language in ter, G. Wong Sak Hoi, A. Zubiaga, SemEval-2017 fake news and political fact-checking, in: Proceed- task 8: RumourEval: Determining rumour veracings of the 2017 Conference on Empirical Methods ity and support for rumours, in: Proceedings of in Natural Language Processing, Association for the 11th International Workshop on Semantic EvalComputational Linguistics, Copenhagen, Denmark, uation (SemEval-2017), Association for Computa2017, pp. 2931–2937. URL: https://aclanthology.org tional Linguistics, Vancouver, Canada, 2017, pp. /D17-1317. doi:10.18653/v1/D17-1317. 69–76. URL: https://aclanthology.org/S17-2006. [9] S. Volkova, K. Shafer, J. Y. Jang, N. Hodas, Separat- doi:10.18653/v1/S17-2006. ing facts from fiction: Linguistic models to clas- [19] G. Gorrell, E. Kochkina, M. Liakata, A. Aker, A. Zusify suspicious and trusted news posts on Twit- biaga, K. Bontcheva, L. Derczynski, SemEval-2019 ter, in: Proceedings of the 55th Annual Meeting task 7: RumourEval, determining rumour veracof the Association for Computational Linguistics ity and support for rumours, in: Proceedings (Volume 2: Short Papers), Association for Compu- of the 13th International Workshop on Semantic tational Linguistics, Vancouver, Canada, 2017, pp. Evaluation, Association for Computational Linguis647–653. URL: https://aclanthology.org/P17-2102. tics, Minneapolis, Minnesota, USA, 2019, pp. 845– doi:10.18653/v1/P17-2102. 854. URL: https://aclanthology.org/S19- 2147. [10] T. Schuster, R. Schuster, D. J. Shah, R. Barzilay, The doi:10.18653/v1/S19-2147.
Limitations of Stylometry for Detecting Machine- [20] D. Küçük, F. Can, Stance detection: A survey, ACM Generated Fake News, Computational Linguistics Computing Surveys (CSUR) 53 (2020) 1–37. 46 (2020) 499–510. URL: https://doi.org/10.1162/co [21] M. Hardalov, A. Arora, P. Nakov, I. Augenstein, li_a_00380. doi:10.1162/coli_a_00380. A survey on stance detection for mis- and disin[11] L. Xue, N. Constant, A. Roberts, M. Kale, R. Al-Rfou, formation identification, in: Findings of the AsA. Siddhant, A. Barua, C. Rafel, mt5: A massively sociation for Computational Linguistics: NAACL multilingual pre-trained text-to-text transformer, 2022, Association for Computational Linguistics, arXiv preprint arXiv:2010.11934 (2020). Seattle, United States, 2022, pp. 1259–1277. URL: [12] C. Rafel, N. Shazeer, A. Roberts, K. Lee, S. Narang, https://aclanthology.org/2022.findings-naacl.94.
M. Matena, Y. Zhou, W. Li, P. J. Liu, Exploring the doi:10.18653/v1/2022.findings-naacl.94. limits of transfer learning with a unified text-to- [22] J. Zheng, A. Baheti, T. Naous, W. Xu, A. Ritter, text transformer, The Journal of Machine Learning Stanceosaurus: Classifying stance towards multiculResearch 21 (2020) 5485–5551. tural misinformation, in: Proceedings of the 2022 [13] A. Santilli, Camoscio: An italian instruction-tuned Conference on Empirical Methods in Natural Lanllama, https://github.com/teelinsan/camoscio, 2023. guage Processing, Association for Computational [14] R. Taori, I. Gulrajani, T. Zhang, Y. Dubois, X. Li, Linguistics, Abu Dhabi, United Arab Emirates, 2022, C. Guestrin, P. Liang, T. B. Hashimoto, Stanford pp. 2132–2151. URL: https://aclanthology.org/2022. alpaca: An instruction-following llama model, 2023. emnlp-main.138. [15] K. Papineni, S. Roukos, T. Ward, W.-J. Zhu, Bleu: a [23] S. Pareti, PARC 3.0: A corpus of attribution relations, in: Proceedings of the Tenth International G. Venturi (Eds.), Proceedings of the Eighth EvaluaConference on Language Resources and Evaluation tion Campaign of Natural Language Processing and (LREC’16), European Language Resources Associ- Speech Tools for Italian. Final Workshop (EVALITA ation (ELRA), Portorož, Slovenia, 2016, pp. 3914– 2023), Parma, Italy, September 7th-8th, 2023, vol3920. URL: https://aclanthology.org/L16-1619. ume 3473 of CEUR Workshop Proceedings, CEUR[24] C. Scheible, R. Klinger, S. Padó, Model architectures WS.org, 2023. URL: https://ceur-ws.org/Vol-3473/ for quotation detection, in: Proceedings of the 54th paper32.pdf .
Annual Meeting of the Association for Computa- [31] J. Maillard, V. Karpukhin, F. Petroni, W.-t. Yih, tional Linguistics (Volume 1: Long Papers), Asso- B. Oguz, V. Stoyanov, G. Ghosh, Multi-task retrieval ciation for Computational Linguistics, Berlin, Ger- for knowledge-intensive tasks, in: Proceedings of many, 2016, pp. 1736–1745. URL: https://aclantholo the 59th Annual Meeting of the Association for gy.org/P16-1164. doi:10.18653/v1/P16-1164. Computational Linguistics and the 11th Interna[25] A. Hanselowski, C. Stab, C. Schulz, Z. Li, tional Joint Conference on Natural Language ProI. Gurevych, A richly annotated corpus for diferent cessing (Volume 1: Long Papers), Association for tasks in automated fact-checking, in: Proceedings Computational Linguistics, Online, 2021, pp. 1098– of the 23rd Conference on Computational Natural 1111. URL: https://aclanthology.org/2021.acl-long. Language Learning (CoNLL), Association for Com- 89. doi:10.18653/v1/2021.acl-long.89. putational Linguistics, Hong Kong, China, 2019, pp. [32] M. Arana-Catania, E. Kochkina, A. Zubiaga, M. Li493–503. URL: https://aclanthology.org/K19-1046. akata, R. Procter, Y. He, Natural language inferdoi:10.18653/v1/K19-1046. ence with self-attention for veracity assessment [26] I. Augenstein, C. Lioma, D. Wang, L. Chaves Lima, of pandemic claims, in: Proceedings of the 2022 C. Hansen, C. Hansen, J. G. Simonsen, MultiFC: Conference of the North American Chapter of the A real-world multi-domain dataset for evidence- Association for Computational Linguistics: Human based fact checking of claims, in: Proceedings of Language Technologies, Association for Computathe 2019 Conference on Empirical Methods in Nat- tional Linguistics, Seattle, United States, 2022, pp. ural Language Processing and the 9th International 1496–1511. URL: https://aclanthology.org/2022.na Joint Conference on Natural Language Processing acl-main.107. doi:10.18653/v1/2022.naacl-m (EMNLP-IJCNLP), Association for Computational ain.107.
Linguistics, Hong Kong, China, 2019, pp. 4685– [33] J. Kim, K.-s. Choi, Unsupervised fact checking by 4697. URL: https://aclanthology.org/D19- 1475. counter-weighted positive and negative evidential doi:10.18653/v1/D19-1475. paths in a knowledge graph, in: Proceedings of [27] P. Atanasova, P. Nakov, L. Màrquez, A. Barrón- the 28th international conference on computational Cedeño, G. Karadzhov, T. Mihaylova, M. Mohtarami, linguistics, 2020, pp. 1677–1686.
J. Glass, Automatic fact-checking using context and discourse information, Journal of Data and Information Quality (JDIQ) 11 (2019) 1–27. [28] N. Kotonya, F. Toni, Explainable automated factchecking for public health claims, in: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Association for Computational Linguistics, Online, 2020, pp. 7740–7754. URL: https://aclanthology.org/2020. emnlp-main.623. doi:10.18653/v1/2020.emnlp -main.623. [29] L. C. Passaro, A. Bondielli, P. Dell’Oglio, A. Lenci,
F. Marcelloni, In-context annotation of topicoriented datasets of fake news: A case study on the notre-dame fire event, Inf. Sci. 615 (2022) 657– 677. URL: https://doi.org/10.1016/j.ins.2022.07.128.
doi:10.1016/j.ins.2022.07.128. [30] A. Bondielli, P. Dell’Oglio, A. Lenci, F. Marcelloni,
L. C. Passaro, M. Sabbatini, Multi-fake-detective at EVALITA 2023: Overview of the multimodal fake news detection and verification task, in: M. Lai,
S. Menini, M. Polignano, V. Russo, R. Sprugnoli,
id link statement_date timestamp of the claim statement verdict_ext political_party politicians_in tags versione the claim verbose veracity judgment of the claim Political party membership at the time of the claim the name(s) of any politician(s) mentioned in the claim (other than the owner of the claim) keywords to describe the content of the claim versioning of the dataset
date content source verdict politician platform
timestamp of fact-checking fact-checking evidence URL of the news outlet/platform where the claim has appeared veracity label of the claim full name of the politician or public figure owning the claim Name of the news outlet/platform where the claim has appeared
broader topic of the claim links list of URLs used to retrieve evidence, write the content, and the verdict