This paper discusses the participation of the SINAI team in the CLEF-2024 CheckThat! lab Task 1 for English. The task involves assessing whether claims extracted from transcribed texts should be fact-checked. We explored two approaches to address this challenge: adjusting a Transformers-based model and using prompting-based techniques. In order to address imbalances within the data provided by the organizers, the method of class weighting is employed. Our best-performing system achieved an F1 score of 0.761 for the positive class and was ranked seventh among all twenty-six submissions in the competition.
The CheckThat! task proposed in the CLEF forum [
For the task, the organisers provide multi-genre textual data in Arabic, Dutch, Spanish and English
taken from tweets and transcriptions [
In order to feed the data to the systems described in Section 3, an analysis of the data obtained for each language was necessary, showing notable distinctions among the diferent languages. Whilst the Arabic and Dutch sets only contain texts sourced from tweets, those in English are based on transcriptions, whereas the Spanish collection comprises texts from both data sources. In order to use data that accurately reflects the desired outcome, the sets in Arabic and Dutch have been discarded, and only the texts from the Spanish set based on transcriptions have been chosen to increase the amount of available data.
In supervised learning tasks, analyzing the class proportion within the dataset is crucial, since the distribution of classes can significantly impact the learning algorithm. An analysis of this ratio has been conducted for each data split, revealing a clear imbalance among classes 1.
Sequence length is an important feature to take into account since most transformer-based models have a limited amount of tokens per sequence to be trained on. Thus, we used the Tiktoken1 library from OpenAI to calculate the average length of the provided texts. Figure 1 shows an histogram that reveals that most sequences contain less than a hundred tokens, which is short enough for most state-of-the-art models. The average tokens in the dataset is 21.06 and the standard deviation 14.38.
In this section, we describe the diferent approaches presented for the oficial evaluation of the first task.
Leveraging the power of models based on Transformers like RoBERTa-base [
As previously mentioned, the imbalance among classes in the data might negatively impact the performance of this fine-tuning. To address the class proportion disparities, the method of class weighting is employed. This technique rectifies imbalances within datasets by assigning greater importance to underrepresented classes, leading to more equitable and accurate predictions. This value is calculated as the ratio of negative samples to positive samples.
Two finetunings were conducted using the same base model. The first finetuning was performed exclusively with the original English dataset, whereas the second expanded the original training data by incorporating translated texts from the Spanish set. These experiments were performed with a learning rate of 1e-06 and a batch size of 8, using both train and dev sets combined as training data and dev-test for evaluating the checkpoints obtained during the execution. To safeguard against overfitting and unnecessary training cycles, the strategy of early-stopping was implemented. This technique stopped the training process if the model shows no improvement over three consecutive epochs. The results from the best checkpoints of both experiments showed minimal diferences 2, obtaining F1 score of 0.896 for the positive class by using the original dataset.
In addition to text and class labels, the organizers provided a sentence_id field that can be sorted to get consecutive sentences in a longer paragraph transcription. An example of this is shown in Table 3 where all the sentences belong to the same intervention from a presidential debate between candidates George Bush and Michael Dukakis in September 19882: 2https://www.presidency.ucsb.edu/documents/presidential-debate-winston-salem-north-carolina I think we’ve seen a deterioration of values.
I think for a while as a nation we condoned those things we should have condemned.
For a while, as I recall, it even seems to me that there was talk of legalizing or decriminalizing marijuana and other drugs, and I think that’s all wrong.
So we’ve seen a deterioration in values, and one of the things that I think we should do about it in terms of cause is to instill values into the young people in our schools. ...
We’ve been dealing with him; he’s been dealing drugs to our kids.
We’ve been dealing with him; he’s been dealing drugs to our kids.
We are better of than we were four years ago.
We are better of than we were four years ago.
That will not help us compete with China.
That will not help us compete with China.
I do not say that.
I do not say that.
We don’t know who the rebels are.
We don’t know who the rebels are. [BUSH:]“I think we’ve seen a deterioration of values. I think for a while as a nation we condoned those things we should have condemned. For a while, as I recall, it even seems to me that there was talk of legalizing or decriminalizing marijuana and other drugs, and I think that’s all wrong. So we’ve seen a deterioration in values, and one of the things that I think we should do about it in terms of cause is to instill values into the young people in our schools. We got away, we got into this feeling that value- free education was the thing.”
Further analysis highlights the need of utilizing context to determine the model prediction as there are similar sentences labeled diferently as shown in Table 4.
With this in mind, we took two diferent prompting approaches, both with GPT-3.5-turbo [
The results obtained with both approaches in the dev-test dataset for the positive class are shown in Table 5. Prompting with context underperformed significantly with respect to the no-context version, which scored an F1-score of 0.7059.
In this section, we report the results obtained during the evaluation cycle. As only the last submission was selected, the best result from all the experiments we performed was submitted. This result corresponds to the RoBERTa-base fine-tuning approach using the original dataset 2.
Surprisingly, the metrics obtained using the gold labels provided by the organizers 6 show a considerable drop in the F1 score of the positive class compared to the results we obtained with the dev-test set.
The fine-tuned models showcased superior eficacy in determining which texts are worth factchecking, as it is reflected in the F1 score of the positive class. This is due to the fundamental diference in training processes: while fine-tuning adjusts the model parameters to a specific task through iterative learning, the prompting approach relies solely on inference without any training process. This crucial distinction underpins the disparities in performance observed between the two methods. The fine-tuning process allows the model to adapt and specialize, leveraging task-specific information and fine-grained adjustments.
In this paper we presented the systems developed by the SINAI team at the CheckThat! Lab to tackle Task 1: Check-worthiness detection. We compare two diferent approaches: A RoBERTa-based finetuning with and without data augmentation from the Spanish dataset provided by the organizers, and GPT-3.5turbo prompting approaches including a baseline and a context-aware prompt. Results show transformer ifnetuning as a promising technique to tackle this task as we ranked 7th out of 26 participants scoring 0.761363 F1-score for the positive class.
Regarding future work, we aim to use balancing techniques such as downsampling of the majority
class or data augmentation with external resources. We would also like to further explore the
contextaware prompting approach as we believe extra context is important given the duplicated examples with
diferent labels, even though it did not achieve good results the way we applied it. Other prompting
techniques such as few-shot learning [
This work has been partially supported by projects CONSENSO (PID2021-122263OB-C21), MODERATES (TED2021-130145B-I00), SocialTOX (PDC2022-133146-C21) funded by Plan Nacional I+D+i from the Spanish Government.