The rapid dissemination of information through social media and the Internet has posed a significant challenge for fact-checking, among others in identifying check-worthy claims that fact-checkers should pay attention to, i.e. filtering claims needing fact-checking from a large pool of sentences. This challenge has stressed the need to focus on determining the priority of claims, specifically which claims are worth to be fact-checked. Despite advancements in this area in recent years, the application of large language models (LLMs), such as GPT, has only recently drawn attention in studies. However, many open-source LLMs remain underexplored. Therefore, this study investigates the application of eight prominent open-source LLMs with fine-tuning and prompt engineering to identify check-worthy statements from political transcriptions. Further, we propose a two-step data pruning approach to automatically identify high-quality training data instances for efective learning. The eficiency of our approach is demonstrated through evaluations on the English language dataset as part of the check-worthiness estimation task of CheckThat! 2024. Further, the experiments conducted with data pruning demonstrate that competitive performance can be achieved with only about 44% of the training data. Our team ranked first in the check-worthiness estimation task in the English language.
With the significant development of the Internet and social media over the past decades, the practical
challenges associated with fact-checking have become more complex [
Since the beginning of the CheckThat! competition, traditional machine learning models and neural
network models have been commonly employed for the task of claim check-worthiness detection. At
CheckThat! 2018, the top submission was achieved by a team using Support Vector Machines and
Multilayer Perceptrons [
Beyond the application of machine learning models, various techniques have been explored
throughout the CheckThat! competition. Feature representation methods, including word embeddings, Bag
of Words, Named Entity Recognition, and Part of Speech tagging [
Large language models (LLMs) have seen remarkable advancements in recent years, with GPT [
This paper presents the experiments conducted for CheckThat! 2024 task 1 [
The remainder of the paper is structured as follows. Section 2 presents the methodology with the introduction to the LLMs experimented, prompts used and the data pruning techniques proposed. Section 3 discusses the experiment results, followed by section 4 concluding the key findings and future directions.
Our goal was to automatically refine the training data to obtain high-quality training data instances and fine-tune open-source Large Language Models (LLMs) to identify check-worthy statements from political transcriptions. This section introduces the dataset, LLMs used in the experiments, prompt engineering carried out, fine-tuning process, and the two-step data pruning we applied to the training data for efective learning.
The dataset provided by CheckThat! 2024 comprises the train, dev, and dev-test partitions, containing 23,849 sentences from political transcriptions, along with a later release of test partition, bringing the total to 24,163 sentences. Table 1 presents the statistics for each partition. From this table, it is evident that the dataset is imbalanced, posing a challenge for the check-worthy statement detection task. Furthermore, Figure 1 illustrates the distribution of text lengths across each partition, revealing that the median length of the sentences is approximately 10-14 words in each partition, with 28%-42% of sentences containing fewer than 10 words. This indicates that the dataset not only sufers from class imbalance but also contains predominantly short sentences, hence implying a limited amount of information.
Open-source LLMs ofer substantial advantages in terms of cost, transparency, community support, and ethical considerations. Consequently, we investigated eight prominent open-source LLMs, as detailed in Table 2 by fine-tuning them for check-worthy statement detection. 1https://github.com/isyufeng/FactFinders 2https://huggingface.co/Rrubaa/factFinders-checkworthy-estimation
Llama2-7b, Llama2-13b [17], and Llama3-8b are part of the Llama family, developed by Meta, and are available in various sizes. The most recent release, Llama3-8b, was made available in April 2024. These models have been optimized for text generation and dialogue applications. Similarly, Mistral [18] and Mixtral [19] are both developed by Mistral AI. Mistral mainly focuses on optimizing transformer models for language tasks, achieving high eficiency and performance in a compact form. Mixtral, with its hybrid approach, aims to integrate the best of various AI methodologies, ofering flexibility and scalability for complex applications. Compared to this bigger model, Phi3-Mini-4K [20] is a smaller variant of the Phi-3-mini, developed by Microsoft, designed to provide capabilities similar to its larger counterparts but with a reduced number of parameters, making it more accessible and easier to run on less powerful hardware. Similarly, Falcon [21], developed by TII, stands as one of the most powerful open-source models and consistently achieves top positions on the OpenLLM leaderboard hosted on Hugging Face. One of the latest models we experimented with, Gemma-7b [22] belongs to the Gemma family developed by Google DeepMind, which is designed to ofer a balance between computational eficiency and advanced capabilities in generating text and understanding complex language queries. We fine-tuned these eight open-source LLMs published in Huggingface platforms (links listed in Table 2) for check-worthy statement detection from political transcriptions.
Given the critical role of prompts in the performance of LLMs, we initially came up with a simple yet direct prompt, as illustrated in Prompt 2. Observing that this initial prompt resulted in lengthy responses with redundant information in the zero-shot setting, and lacked a clear definition of checkworthiness, we employed ChatGPT-4 to refine and improve the prompt, resulting in Prompt 1. All eight LLMs were fine-tuned using this refined prompt to generate ‘Yes’ or ‘No’ answers indicating the check-worthiness of the input statement. Furthermore, we observed that a significant proportion of sentences in the training data utilized pronouns to refer to political entities, thereby increasing uncertainties and ambiguities. Therefore, we experimented with an expanded version of Prompt 1 (i.e. Prompt 3) to evaluate whether explicitly indicating that the pronouns in the input statement may refer to political entities could enhance the performance of the fine-tuned model. However, the initial experiments on prompt engineering revealed that neither the compressed prompt (Prompt 2) nor the expanded (Prompt 3) improved the performance of fine-tuned models (refer to Table 6). 3https://huggingface.co/meta-llama/Llama-2-7b-hf 4https://huggingface.co/meta-llama/Llama-2-13b-hf 5https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct 6https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2 7https://huggingface.co/mistralai/Mixtral-8x7B-v0.1 8https://huggingface.co/microsoft/Phi-3-mini-4k-instruct 9https://huggingface.co/tiiuae/falcon-7b 10https://huggingface.co/google/gemma-7b Prompt 1 Check-worthy Statement Detection ### Instruction: Read the statement provided as below.
Your task is to evaluate whether the statement contains information or claims that are worthy to be verified through fact-checking. If the statement presents assertions, facts, or claims that would benefit from verification, respond with ’Yes’. If the statement is purely opinion-based, trivial, or does’not contain any verifiable information or claims, respond with ’No’. ### Response: <Yes/No>
Fine-tuning an LLM is a challenging task due to the resource requirements, especially the memory demand. While this challenge can be escalated by training only certain layers of the LLM, still the computational requirement associated with gradient updates requires a lot of GPU memory. Therefore we use the Low-Rank Adaption technique (LoRA) for fine-tuning the LLMs. Instead of updating the weights directly, LoRA keeps track of the changes through low-rank perturbations requiring only minimal GPU memory. The LoRA configuration used for the fine-tuning is listed in Section 3.1. To ensure experimental control, consistent hyperparameters were applied across all eight LLMs (see Table 3).
The performance of each model on check-worthy statement detection is presented in Table 4. Unfortunately, we could only compare the performance of the Llama family, Mistral, and Mixtral models during the testing phase of the competition. Therefore, the Phi-3-Mini-4k, the best-performing model in the Dev-Test partition wasn’t considered for the remaining experiments. Considering the competitive performance of the Llama2-7b model and the time and memory required, the rest of the experiments were carried out by fine-tuning the Llama2-7b using Prompt 1.
As previously mentioned, the training data is highly imbalanced, mostly containing shorter sentences
with limited information. Recent studies [
The pronouns in the statement may refer to political entities.
Your task is to evaluate whether the statement contains information or claims that are worthy to be verified through fact-checking. If the statement presents assertions, facts, or claims that would benefit from verification, respond with ’Yes’. If the statement is purely opinion-based, trivial, or does’not contain any verifiable information or claims, respond with ’No’. a two-step data pruning approach to automatically identify high-quality data instances for efective learning.
We began with identifying informative sentences in the training data that could potentially convey meaningful information for the training. In other words, we intended to remove the noisy instances from the training data as the first step of the data-pruning process. We define a political statement as informative if it meets one of the following four criteria: • Check-worthy status is "Yes": If the class label of the statement is "Yes", then it is informative. • Contains a named entity: If the statement contains a named entity, it is highly likely to discuss information related to that entity. Hence the statement is informative. • Contains an informative verb: If the statement contains an informative verb, it is highly likely to discuss an informative action. Hence the statement is informative.
• Lengthy enough: If the statement is lengthy enough, it is likely to convey meaningful information.
While the first criteria check-worthy status is straightforward, the other criteria require extracting further information to determine the informative status of a sentence. We used the BERT model [24] ifne-tuned 11 for the Named Entity Recognition (NER) task to identify the presence of a named entity. The NER model identifies four types of named entities, person, organization, location, and miscellaneous from the input text. We noticed that only 37.8% of the training data contains at least one named entity and only 12.2% of the training data mentions a person’s name. This indicates the prevalent use of pronouns to refer to the political entities in the transcriptions, which makes the task more challenging.
In order to identify informative verbs, we first extracted all the verbs presented in the training data using the Part-of-Speech tagger from NLTK library.12 Extracted verbs were lemmatized further to bring them to their base form. This resulted in 3838 verbs in the training data to be identified as informative or not. We wanted to automatically classify each verb as either informative or not based on whether it conveys any check-worthy action. However, performing this binary classification in a zero-shot setting 11https://huggingface.co/dslim/bert-base-NER 12https://www.nltk.org/api/nltk.tag.pos_tag.html using a language model is a challenging task as explicitly defining an informative verb in the prompt may result in ambiguous classification. Therefore, we performed a fine-grained categorization of the verbs into the following 10 categories.
1. Physical Actions: e.g. Run 2. Mental Actions: e.g. Think 3. Changes in State: e.g. Grow 4. Creation or Destruction: e.g. Build 5. Communication: e.g. Discuss 6. Movement: e.g. Walk 7. Emotion: e.g. Hope 8. Perception: e.g. See 9. Linking verbs : e.g. is, has 10. None: Any verb that does not fit into the other categories
The first 8 categories were obtained by prompting ChatGPT with the question "What are the types of action verbs?". In addition to the 8 action verb categories, we added Linking verbs and None resulting in 10 categories of verbs. The option "None" was added to the categories to indicate that the verb does not ift into any of the other 9 categories.
We utilized Mixtral [19] in a zero-shot setting to classify a verb into one of the 10 categories using the Prompt 4. Among the 10 categories, we chose the verb types Physical Actions, Changes in State, Creation or Destruction, Communication, and Movement as the informative verbs, as the other verb types are less likely to represent a check-worthy action.
Figure 2 shows the word cloud of verbs present in the training data. Informative verbs are highlighted in green color and non-informative verbs are highlighted in red color in the Figure. It can be observed that most of the verbs are classified into the two groups correctly. Figure 3 presents the verb type distribution in the training data. Interestingly the occurrences of informative verb categories are relatively high compared to non-informative verb categories. We further noticed that 88.2% of the check-worthy statements contain at least one informative verb in the training data, whereas this value drops to 77.2% for non-check-worthy statements.
The final factor determining an informative sentence, minimum length is dificult to define explicitly. Therefore, we choose the minimum length value that reaches the optimal F1-score in the dev-test data by varying the value from 3 - 10. We excluded the stop words13 while calculating the length of a statement. 13https://www.nltk.org/search.html?q=stopwords We observed that the most informative sentences are obtained when the minimum length factor is set to 8 (refer to Section 3.6 for the results). With this optimal setting, the first step of data pruning resulted in a reduced training data with 20,141 sentences. In other words, 2358 sentences (10.5% of original training data) were filtered out as non-informative sentences at this stage of the data pruning process.
The informative sentences identified in the previous stage are still imbalanced in class. Therefore, we executed an under-sampling technique, Condensed Nearest Neighbour (CNN) [23], to generate class-balanced training data. We retained all the minority data instances (check-worthy statements) and sampled only the majority data instances (non-check-worthy statements). The idea behind CNN sampling is to identify a subset of data instances that can be used to correctly classify all the other unsampled data instances using the 1-Nearest Neighbour rule. This makes sure that the sampled data (a) Original training data (before applying step 1 & 2) (b) Original training data with uninformative sen-(c) High-quality training data obtained after data pruntences (filtered during step 1) highlighted ing (after step 1 & 2) distribution is the same as the original data distribution without any information loss.
CNN requires the input data to be represented as vectors to iteratively sample data points from a vector space and perform the 1-Nearest Neighbour classification in the unsampled data points. We used BERT [24] embeddings to convert the sentences in the training data to a vector representation of length 768. The imbalanced14 python library was used to perform CNN sampling. This resulted in a sampled data with 9907 sentences (44% of the original training data) as the high-quality training data. Since we retained all the positive data instances (check-worthy statements) during the data pruning process, the resulting high-quality training data comprised 54.6% positive instances and 45.4% negative instances.
Figure 4 visualizes the training data points in 2 dimensions at each stage of the data pruning process. It can be observed that uninformative data points filtered during step 1 (Figure 4b) are concentrated around the bottom-left corner of the plot. Further, the CNN samples a similar distribution of non-check-worthy data points (Figure 4c) for obtaining balanced training data. 14https://imbalanced-learn.org/stable/references/generated/imblearn.under_sampling.CondensedNearestNeighbour.html
We discuss the experiment results in this section, especially the hyper-parameters used to fine-tune the LLMs, the environment setting used, the performance of various LLMs with the consistency analysis, efect of prompt engineering, and the impact of training data pruning on check-worthy statement detection task.
Hyper-parameters used to fine-tune the LLMs in our experiments are listed in Table 3. Most of the hyper-parameters were the same for all the LLMs we experimented with, except the training batch size which was reduced to 1 for Mixtral, due to its memory demand. Consequently, the gradient accumulation steps for this model was increased to 4. All the experiments were conducted using Queen Mary’s Apocrita HPC facility, supported by QMUL Research-IT [25]. Specifically, 1 GPU (Volta V100 or Ampere A100) with 8 CPU cores, each composed of 11 GB memory was used to train and test all the models.
While generating the predictions by the fine-tuned models for evaluation, we noticed that a language model may generate diferent predictions for the same prompt and input sentence. Therefore, we ran each fine-tuned language model 5 times and obtained the majority prediction as the final prediction of the model. Further, we fine-tuned each model 3 times and reported the average performance of 3 ifne-tuned models. We use the partitions train and dev for training and validation of the models, and the performance of the models is reported on the other two partitions dev-test and test.
The oficial evaluation metric for CheckThat! 2024 task 1, check-worthiness estimation is F1-Score over the positive class. However, since one metric could have a bias toward the comparison, we report average accuracy, precision, and recall along with the F1-score. Further, we computed consistency@K of the fine-tuned models indicating the fraction of data instances for which the model generated the same Test output class in all K iterations. Consistency score report in the following subsections was computed
We compare the performance of the eight open-source LLMs in test and dev-test partitions and Table 4 reports their average accuracy, precision, recall, F1-score, and consistency@5. Since the class distribution of dev-test and test partitions are diferent, we can observe that the performance in each partition varies for all the LLMs. While Phi3-Mini-4K obtains the highest F1-score in the dev-test portion, Llama2 models stand out as the best-performing models in the test partition based on F1-score. Further, both Llama2-7b and Phi3-Mini-4K demonstrate greater consistency in predicting class labels across both partitions compared to other models. On the other hand, Llama3-8b, one of the latest models from the Llama family reaches the highest accuracy and precision in the test partition. However, the model fails to outperform the other models in terms of F1-score due to poor recall. Similarly, Mixtral, the largest model we compared, fails to give a consistent performance across both partitions. Moreover, both Mistral and Falcon remain as the lower-performing models in both partitions.
Table 5 lists the fine-tuning time of each LLM. Except for Mixtral and Phi3-Mini-4K, the largest and smallest models compared respectively, all the other models’ fine-tuning time remains between 6-12 hours. As expected, the fine-tuning time increases with the number of parameters.
As we already mentioned, we were able to compare only the Llama models, Mistral, and Mixtral during the testing phase of the competition. Therefore, considering the performance of these models in terms of F1-score in the dev-test partition and the time and memory required to fine-tune the model, we chose Llama2-7b as the optimal LLM for the remaining experiments.
Since LLMs are text generation models, they tend to generate diferent output for the same input text even in a fine-tuned environment. Therefore, we analyze their consistency in predicting the same output class label over K iterations with the metric consistency@K. Figure 5 presents the change in consistency with the number of iterations varied from 2 to 25 in the test and dev-test partition. It can be observed that the consistency declines with the increase in the number of iterations and becomes stable after around 11-12 iterations. While the model could reach a stable consistency in both partitions, the percentage of drop in consistency (diference between initial consistency when K=2, and the stable consistency) is nearly double for the test partition (3.1% vs 5.9%).
We conducted experiments using three proposed versions of prompts discussed in Section 2.2.2 along with a prompt without any instruction to analyze the impact of prompt engineering on Llama2-7b model performance. Table 6 presents the evaluation results on the test and dev-test partitions. While all three prompts proposed reach a similar F1-score in the test partition, their impact is quite evident in the dev-test partition. We can observe that Prompt 1 achieves the best overall scores across all metrics. It is Prompt Prompt 1 Prompt 2 Prompt 3 No Instruction worth noting, that the expanded prompt Prompt 3, shows a notably high recall score and consistency in the test partition possibly due to the attention given to the pronouns in the instruction. Moreover, a substantial performance decline is observed when no instructions are included in the prompt highlights the importance of prompt engineering in achieving optimal results from LLMs.
In addition to performance metrics, Table 7 indicates that the fine-tuning time increases with the length of the instruction. According to the competition’s settings, we primarily consider the F1-score performance of each prompt in the dev-test partition. Consequently, we selected Prompt 1 as the optimal one for the remaining experiments.
As we discussed earlier, we leave the minimum length factor as a parameter of the data pruning process. Therefore we varied the minimum length from 3-10 and observed the performance of the Llama2-7b model fine-tuned on the pruned dataset. Figure 6 shows the F1-score of the fine-tuned model in the dev-test partition. It can be observed that Step 1 alone is not suficient enough to identify high-quality training data, and Step 2 always boosts the performance when combined with Step 1 except when the minimum length factor is set to very low (3-4). The optimal performance for the two-step data pruning is obtained when the minimum length factor is set to 8.
Table 8 presents the performance of Llama2-7b on the original and pruned training data. It can be observed that the model trained using only the step 2 data pruning approach yields the highest F1-score and accuracy in the dev-test partition and its performance in the test partition is close to the model trained without any data pruned. While the two-step data pruning approach reaches a slightly lower F1 score in the test partition, it stands out as the high recall model in both partitions. Further, it is worth noting that this model resulted in a better precision-recall trade-of in the dev-test partition compared to the models trained with individual pruning steps (step 1 only and step 2 only). Moreover, the consistency of the models in predicting the same output class ranged from 0.9-0.95, while the lower range is always observed in the dev-test partition. During the testing phase of the completion, we could not compare the average performance of the models due to time limitations. Therefore, we submitted the class labels predicted by the Llama2-7b model fine-tuned without the data pruning process for the CheckThat! 2024 task1 leader board, as it yielded a slightly higher F1-score compared to other approaches. This submission was ranked 1st place in the leaderboard with the highest F1-score of 0.802. While this score is lower than the average F1-score reported in Table 8, the standard deviation indicates that the model performance could vary from 0.801 to 0.839.
Table 9 reports the training data size and their corresponding average fine-tuning time. This demonstrates that using data pruning strategies allows competitive performance on both test and dev-test data while utilizing only about 44%-44.5% of the original training data. Further, the fine-tuning time is reduced in a similar proportion, cutting the training time by more than half compared to using the original training data. This indicates that obtaining high-quality training data is crucial for developing efective check-worthy statement detection models from political transcriptions.
This paper demonstrates the experiments conducted by the FactFinders team for CheckThat! 2024 task 1, check-worthiness estimation in English. We experimented with eight open-source LLMs with fine-tuning and prompt engineering to identify check-worthy statement detection from political transcriptions. Results using our Llama2-7b model fine-tuned on the training data secured the 1st position in the leaderboard among a total of 26 participants, with F1-scores surpassing the baseline for the task. This demonstrates that open-source models are powerful in check-worthy statement detection in the English language. Further, we demonstrated the role of data pruning in identifying high-quality training data for efective learning. Our results show that competitive or better performance can be obtained by utilizing only about 44% of training data by saving the fine-tuning time in a similar proportion. Apart from the fine-tuned LLMs for check-worthy statement detection, we utilized LLMs for refining prompts and identifying informative verbs in zero-shot setting.
The key challenges we faced while utilizing LLMs for check-worthy statement detection were the memory requirements and their inconsistent response in predicting the class label for the same input statement. We used the Low-Rank Adaption technique (LoRA) for efective fine-tuning with low GPU memory usage to conquer the memory requirements. While we tried to overcome the consistency issue by running the fine-tuned models 5 times and obtaining the majority prediction, our consistency analysis reveals that the consistency score itself is unstable during early iterations, and may have a drop of around 6% while reaching stability. This behavior of LLMs highly questions their adaptation for classification tasks in general as inconsistent responses may result in less reproducible results.
Rrubaa Panchendrarajan is funded by the European Union and UK Research and Innovation under Grant No. 101073351 as part of Marie Skłodowska-Curie Actions (MSCA Hybrid Intelligence to monitor, promote, and analyze transformations in good democracy practices). Yufeng Li is funded by China Scholarship Council (CSC). and adversarial robustness, in: L. Goeuriot, P. Mulhem, G. Quénot, D. Schwab, L. Soulier, G. M. Di Nunzio, P. Galuščáková, A. García Seco de Herrera, G. Faggioli, N. Ferro (Eds.), Experimental IR Meets Multilinguality, Multimodality, and Interaction. Proceedings of the Fifteenth International Conference of the CLEF Association (CLEF 2024), 2024. [16] M. Hasanain, R. Suwaileh, S. Weering, C. Li, T. Caselli, W. Zaghouani, A. Barrón-Cedeño, P. Nakov, F. Alam, Overview of the CLEF-2024 CheckThat! lab task 1 on check-worthiness estimation of multigenre content, in: G. Faggioli, N. Ferro, P. Galuščáková, A. García Seco de Herrera (Eds.), Working Notes of CLEF 2024 - Conference and Labs of the Evaluation Forum, CLEF 2024, Grenoble, France, 2024. [17] H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, et al., Llama 2: Open foundation and fine-tuned chat models, arXiv preprint arXiv:2307.09288 (2023). [18] A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. d. l. Casas, F. Bressand,
G. Lengyel, G. Lample, L. Saulnier, et al., Mistral 7b, arXiv preprint arXiv:2310.06825 (2023). [19] A. Q. Jiang, A. Sablayrolles, A. Roux, A. Mensch, B. Savary, C. Bamford, D. S. Chaplot, D. d. l. Casas,
E. B. Hanna, F. Bressand, et al., Mixtral of experts, arXiv preprint arXiv:2401.04088 (2024). [20] M. Abdin, S. A. Jacobs, A. A. Awan, J. Aneja, A. Awadallah, H. Awadalla, N. Bach, A. Bahree, A. Bakhtiari, H. Behl, et al., Phi-3 technical report: A highly capable language model locally on your phone, arXiv preprint arXiv:2404.14219 (2024). [21] E. Almazrouei, H. Alobeidli, A. Alshamsi, A. Cappelli, R. Cojocaru, M. Debbah, É. Gofinet, D. Hesslow, J. Launay, Q. Malartic, et al., The falcon series of open language models, arXiv preprint arXiv:2311.16867 (2023). [22] G. Team, T. Mesnard, C. Hardin, R. Dadashi, S. Bhupatiraju, S. Pathak, L. Sifre, M. Rivière, M. S.
Kale, J. Love, et al., Gemma: Open models based on gemini research and technology, arXiv preprint arXiv:2403.08295 (2024). [23] P. Hart, The condensed nearest neighbor rule (corresp.), IEEE transactions on information theory 14 (1968) 515–516. [24] J. Devlin, M. Chang, K. Lee, K. Toutanova, BERT: pre-training of deep bidirectional transformers for language understanding, CoRR abs/1810.04805 (2018). URL: http://arxiv.org/abs/1810.04805. arXiv:1810.04805. [25] T. King, S. Butcher, L. Zalewski, Apocrita - High Performance Computing Cluster for Queen Mary University of London, 2017. URL: https://doi.org/10.5281/zenodo.438045. doi:10.5281/zenodo. 438045.