This article outlines the NLPIR-UNED team's strategies for Task 1B in the CLEF 2023 CheckThat! Lab. The goal of this task is to determine if a text fragment from a tweet or a debate/speech is worth fact-checking. Our team devised three main approaches based on ensemble models for this binary classification task. For the English version of subtask 1B, which involves classifying text fragments from debates/speeches, we utilized an ensemble classifier composed of three transformer models that were fed diferent sentences from the debate/speech. On the other hand, for the Spanish version of subtask 1B, which requires classifying tweets, we have tried two more strategies, an ensemble classifier composed of three diferent transformer models in Spanish that receive the same tweet, and the one that we finally use: an ensemble classifier that combined a transformer model and two feed-forward neural networks (FFNN). The transformer model processes the tweet's text, while the two FFNNs receive as input TF-IDF vectors and LIWC features extracted from the text, respectively. With these approaches, our team achieved the fourth position in subtask 1B English and the same position for subtask 1B Spanish.
Fake news is a growing problem that has been amplified by the rise of social media and the ease of spreading misinformation online. This phenomenon can have serious consequences, such as influencing political elections, spreading harmful health information, and causing social unrest. Traditional fact-checking methods are often slow and labor-intensive, making them inefective at keeping up with the speed at which false information can spread. This has led to the development of automated methods to detect and combat fake news, such as machine learning algorithms that can quickly analyze large volumes of data and identify suspicious patterns. These automated methods have the potential to provide a more eficient and efective approach to combating fake news.
One of the fundamental tasks to perform if we want to detect fake news in news or message
lfows on a social network is the selection of the statements to check. This is precisely what task
1 of the CheckThat! Lab [
We have organized the rest of the article as follows: in Section 2 we make a brief review of the diferent approaches carried out in recent years to the task of estimating the check-worthiness of a statement, in Section 3 we explain our diferent approaches to this task, Section 4 discuss the results obtained, and Section 5 contains our conclusions and future work.
The task of estimating check-worthiness of a sentence has had diferent approaches as the
available models and tools have evolved. The approaches considered include word embeddings,
bag-of-words representations, and heuristic rules to classify claims, and multilayer perceptron
or support vector machine models [
Subsequently, more sophisticated representation models have been applied, such as training
a feed-forward neural network with Standard Universal Sentence Encoder embeddings and
using diferent variations of embeddings and training epochs [
Finally, with the generalization of transformer models as a basic tool in almost any task
related to general language processing, most approaches have used this strategy [
In this 2023 edition of the CheckThat! Lab, the goal of subtask 1B is to determine if a given tweet is worth checking (binary classification), taking into account whether this tweet contains a factual statement that can be verified and whether it could be harmful. This task is ofered in three languages: English, Spanish and Arabic. The organizers provide three diferent datasets for each language with which the models can be developed, in addition to the test dataset used for the competition. In Table 1 we can see the number of instances of each dataset for the languages in which our team has participated.
To tackle this subtask, our team has evaluated three strategies, all based on ensemble classifiers.
The first of them is an ensemble classifier composed of a transformer model, a feed forward
neural network (FFNN) whose inputs are TF-IDF vectors, and a FFNN whose inputs are text
analysis indicators. Here, the objective is to complement the latent features that a transformer
model is able to extract from plain text, with other types of features such as TF-IDF vectors
extracted from that same text, and the features provided by the text analysis tool Linguistic
Inquiry and Word Count (LIWC) [
The other two strategies are also ensemble classifiers but this time they contain three transformer models although used diferently. In the subtask proposed for the English language, the dataset is composed of sentences extracted from a debate and in the first column appears an identifier that, after carefully examining the instances of training and test, we have assumed is the order in which the sentences appeared in the debate. That is why we wanted to explore the possibility of taking advantage of this information, making the ensemble classifier receive in its input 3 the instance to be evaluated, and in its inputs 2 and 1 the two instances immediately prior to the current one existing in the dataset, taking into account that there are gaps we assume generated when making the training-dev-test partition. Our hypothesis is that providing context information (previous sentences) to the sentence to be evaluated can be useful in determining the check-worthiness of that sentence.
For the subtask proposed in Spanish the dataset is composed of tweets, which we cannot identify as related following any type of order. That is why we have chosen to use an ensemble composed of three diferent transformer models pre-trained all of them in Spanish, hoping that the diferent behaviors that each of them may have complement each other, achieving a superior performance to each of them separately.
We detail each of these approaches below. All used pre-trained transformer models have been downloaded from https://huggingface.co/.
3.1.1. Method To check whether diferent types of input generated from the same text can complement each other and lead to greater eficiency in detecting whether that text deserves to be verified, we need to be able to handle these three types of input simultaneously for each instance of the dataset. Therefore, we have developed an ensemble model (Figure 1) composed of a transformer
TF-IDF Vectors
Transformer Pretrained Model [CLS] First Token Last Hidden Layer
FFNN 1 Pretrained Model
FFNN 2
Pretrained Model Hidden Layer
Hidden Layer that processes the text as a sequence, a FFNN classifier that admits as input that text in the form of a TF-IDF vector, and a second FFNN classifier that has as inputs the discrete features generated by the LIWC text analysis tool (93 features for the English language and 90 features for the Spanish language). The hidden layers of the FFNNs and the first token of the last hidden layer of the transformer (classification token) are concatenated and form the first layer of the ensemble classifier. Behind this concatenation layer are two hidden layers and one output layer. It is also possible to disable one of the hidden layers by configuration. Before training the ensemble classifier, the transformer and the two FFNN models are trained separately on the same dataset and stored in binary files. These models are then loaded in evaluation mode in the ensemble classifier to prevent their parameters from being modified during ensemble training.
To determine the configuration with the best performance, the system was configured to use deterministic algorithms and the tests were repeated for 10 diferent random seeds, obtaining the average of the precision, recall, and F1 measures. An early stopping mechanism has also been implemented. This mechanism stores the updated state of parameters after each epoch and stops training when there have been no improvements in the F1 measure over the dev dataset in the last n epochs (default value 2), then selecting the saved configuration with the best F1 measure during that interval. After performing a grid search for each component separately and for the ensemble classifier, the following hyperparameters have been selected for the English language: • FFNN hidden layer size: 1000. • FFNN seed value: 0. • FFNN max. epochs: 250. • FFNN (TF-IDF) activation function: relu. • FFNN (LIWC) activation function: sigmoid. • Transformer pre-trained model: bert-base-uncased. • Transformer max. sequence lenght: 128. • Transformer max. epochs: 10. • Transformer seed value: 63. • Ensemble activation function: relu. • Ensemble hidden layers: 2. • Ensemble dropout: 0.
• Ensemble max. epochs: 10.
And for the Spanish language the same hyperparameters have been used except for the
following:
• FFNN seed value: 96.
• Transformer pre-trained model: bertin-project/bertin-roberta-base-spanish [
3.2.1. Method In the English subtask, as discussed above, we wanted to take advantage of what appears to be a flow of sentences to provide context for the sentence to be evaluated. For this, we have developed an ensemble classifier (Figure 2) composed of three transformer models so that each one can receive a diferent sentence. During the processing of the training and test datasets, we look for the two instances that have the identifiers immediately before i - n, i - n - m to the instance i to evaluate. These three sentences form the input of the ensemble classifier using the class value of the instance i. As a pre-trained model, we use the same in each of the three transformer components: bert-base-uncased.
For the Spanish subtask, since the datasets contain tweets, we have assumed that they had
no relationship between them by adopting the strategy of providing the same sentence (tweet)
in the three inputs of the ensemble classifier, and use a diferent pre-trained model on each
transformer component. With this, we hoped that the diferent pre-trainings that each one has
had are somehow complementary and allow us to determine more precisely if the sentence to
evaluate is worth checking. Specifically, we have used the following models:
• Transformer 1: PlanTL-GOB-ES/roberta-large-bne [
• Transformer 3: bertin-project/bertin-roberta-base-spanish.
The hyperparameters used in both languages were as follows: • Transformer max. sequence lenght: 128.
Sequence i
Sequence i - n
Sequence i - n - m
Transformer 1 Pretrained Model [CLS] First Token Last Hidden Layer
Transformer 2 Pretrained Model [CLS] First Token
Last Hidden Layer • Transformer max. epochs: 10. • Transformer seed value: 63. • Ensemble activation function: relu. • Ensemble hidden layers: 2. • Ensemble dropout: 0.
• Ensemble max. epochs: 10.
This section describes the results obtained with the three strategies described. The F1 score on the positive class is the evaluation measure used by the organizers in the competition.
The organizers provided three datasets for both languages: training, dev-test and test. Since the values of F1 measure obtained for the English subtask with the test dataset were very high even for the n-gram baseline, we also performed the evaluation joining the dev-test and test datasets. In all cases the training has been done using only the training dataset. Table 2 shows the averaged results of the models after 10 runs with diferent random seeds.
The last column shows the F1 measure calculated on the test dataset, along with the reference values provided by the baselines. As we can see, the FFNN with input in the form of TF-IDF vectors is not able to overcome the baseline n-gram. On the other hand, the transformer model bert-base-uncased outperforms the two ensemble classifiers, indicating that, at least in this test dataset, the combination of models and inputs does not bring any improvement to the classifier’s performance. In principle, ensemble models would be expected to outperform the solo transformer model as they should be able to select the best information present in each of their three inputs to evaluate a given sentence. It is possible that the context search that we intended to use by looking for sentences before the current one is not working in this test dataset because there is too much distance between them. Remember that we have assumed that the identifier of each sentence is the order it has within the debate. Thus, having partitioned the debate by randomly extracting sentences to create the train, test and dev-test datasets, the sentences are no longer consecutive in these datasets. Regarding the other type of ensemble classifier, we can also assume that in this test dataset the inputs of TF-IDF vectors and LIWC features do not provide enough information to improve the behavior of the ensemble model.
If we look at the evaluation carried out on the dev-test + test dataset, we see that in this case the two ensemble models surpass to the transformer model alone, although by a small margin. For the ensemble model that searches for previous sentences, performance may be improving as two datasets have been rejoined, making the distances between the current sentence and the two immediately preceding sentences smaller and contributing to this context information more efectively.
For the main submission, we selected the ensemble model that seeks context information in two previous sentences (shown in Table 2 as Ensemble of Transformers). Although it does not have the highest average F1 score, the diferences with the other models are minimal, and we wanted to see how this approach performs in the competition. With this configuration we have achieved the fourth best result among the eleven participating teams with an F1 measure of 0.851 (Table 3), being 0.898 the value obtained by the team classified in first position and 0.462 the one obtained by the baseline.
To tackle the task of estimating the check-worthiness of a sentence or tweet, in this edition of the CheckThat! Lab our team has evaluated several strategies that involve the use of ensemble classifiers.
One of them, has been based on the use of an ensemble classifier containing a transformer model, a feed-forward neural network (FFNN) with TF-IDF vectors at the input, and a second FFNN with features extracted by the text analysis tool LIWC. This model, as we expected, has been able to surpass the solo transformer models in the Spanish subtask and is the one we have used to make the main submission in this language, obtaining the fourth best result (F1 = 0.589) among the seven participants. In this same subtask we have also evaluated an ensemble model that contained three diferent transformer models in Spanish, obtaining similar results.
In subtask in English language, the diferences between the results obtained by the ensemble models and the transformer models alone have been much smaller, the latter surpassing the ensemble in the test dataset. The main diference between both subtasks is the content of the datasets: sentences of a debate in the subtask in English, and tweets in the subtask in Spanish. This, in principle should not be the reason for this similarity in performance since ensemble models usually perform better than solo transformers. Still, we have selected the ensemble classifier composed of three transformer models fed with the current sentence and two previous sentences to provide context information. With this configuration, our team has obtained the fourth best F1 measure (0.851) among the eleven participating teams. We think that this model can give good results when, for example, we are analyzing a text to identify the sentences that are worth checking, because in this case we could select the two sentences immediately prior to the current one, unlike what happens with the datasets of this subtask where not all the sentences of the debate are available.
In the future, we intend to further explore alternative methods of integrating diverse models into an ensemble classifier, thereby expanding the range of features utilized in identifying sentences that need to be verified.
This work has been partially supported by the Spanish Ministry of Science and Innovation within the DOTT-HEALTH Project (MCI/AEI/FEDER, UE) under Grant PID2019-106942RB-C32 and OBSER-MENH Project (MCIN/AEI/10.13039/501100011033 and NextGenerationEU/PRTR) under Grant TED2021-130398B-C21 as well as project RAICES (IMIENS 2022). Pre-Training of a Spanish Language Model using Perplexity Sampling, Procesamiento del Lenguaje Natural 68 (2022) 13–23. URL: http://journal.sepln.org/sepln/ojs/ojs/index.php/ pln/article/view/6403.