This article describes the different approaches used by the NLPIR@UNED team in the CLEF2021 Check-That! Lab to tackle the tasks 1A-English, 1A-Spanish and 3A-English. The goal of Task 1A in English is to determine which tweets within a set of COVID-19 related tweets are worth checking. Task 1A in Spanish is similar but in this case the tweets are related to political issues in Spain. In both tasks, transformer models have been used to identify check-worthy tweets, obtaining the first place in the task in English and the fourth place in the task in Spanish. Task 3A is focused on determining the veracity of a news article. It is a multi-class classification problem with four possible values: true, partially false, false, and other. For this task we have used two different approaches: a gradient-boosting classifier with TF-IDF and LIWC features, and a transformer model fed with the first tokens of each news article. We got the fourth place out of 25 participants in this task.
Despite the efforts carried out in recent times to combat the proliferation of fake news, these have not stopped growing, taking advantage of events conducive to its dissemination, such as the current pandemic, or the events that occurred in the last presidential elections in the United States. Therefore, the existence of initiatives such as this CheckThat! Lab [1] [2], which give researchers in this area of natural language processing the opportunity to propose and share different ideas that can help mitigate this problem, are appreciated.
In this article, we present the approaches used by our team in the tasks of check-worthiness and fake news detection. Since transformer models have become a fundamental tool that adapts to many of the tasks related to natural language processing obtaining state-of-the-art results, we have chosen to take them as our first option in each of the tasks. However, in Task3a we decided to also use more classical approaches since the size of the news articles to be checked exceeded the input sequence size that is reasonable to define in a transformer model.
We have organized the rest of the article as follows: in section 2 we briefly describe the transformer models, the approach we have used in tasks 1A-English and 1A-Spanish and we comment on the results obtained, in section 3 we explain our approach in the fact-checking task and discuss the results obtained, and section 4 contains our conclusions and future work.
Among the approaches that have been used to tackle this task we can highlight the initial work carried out by [3] where they make use of classifiers such as Random Forest, SVM or Multinomial Naive Bayes, and features based on TF-IDF representations, parts of speech tags, sentiment scores, and entity types. To the aforementioned methods [4] add features such as average embedding vector of the sentence, linguistic features that count the number of words in the sentence that belong to a certain lexicon, contextual features such as the position of a sentence with respect to others in a segment of text, discourse features such as the detection of contradictions, and as a classifier uses a Deep Feed-Forward Neural Network. Already within this Check That! Lab we have seen in past editions the use of recurrent neural networks by [5] where each token is represented in three ways: through embeddings, and with part of speech tags and syntactic dependencies encoded as one-hot vectors. In the same edition [6] makes use of character n-gram features with a k-nearest neighbors classifier. More recently in this same Lab, transformer models began to be used for the check-worthiness task by many of the participants [7][8] [9]. In the next section we will see a short description of this architecture.
Since its appearance as an alternative to neural machine translation models, transformer models [10] have become a preferred model when compared to other natural language processing techniques, not only in machine translation, but in other tasks such as sequence classification, summarization, named entity recognition, text generation, extractive question answering or language modeling.
A transformer is a deep learning model that "translates" input sequences into output sequences using an encoder-decoder architecture. It uses an attention mechanism to identify the most relevant parts of the input and output sequences. Previous models such as RNNs also use an attention mechanism but are limited by their sequential nature when processing input data. Transformers, by relying solely on the attention mechanism, do not need to process the input sequences in a specific order, allowing them to process these sequences in parallel and thus reducing training times.
The model is fed with training data in the form of sequence pairs (input, target). The first is applied in the encoder block and the second in the decoder block.
In recurrent models, sequences are introduced token by token, thus providing the relative position of each of these tokens in the sequence. Since transformers do not process sequences in this way, this positional information is provided to the model as a mask added to the input and target sequences.
The encoder block is made up of a stack of n identical encoders, each of them with a selfattention layer and a feed fordward neural network. The decoder block is made up of the same number n of decorders and each of them is composed of a self-attention layer, an encoder-decoder attention layer and a feed forward neural network.
The self-attention layers allow to identify within the same sequence, which tokens are more relevant for another token that is being considered at that moment. On the contrary, the encoder-decoder attention layer relates tokens of the input and target sequences. The attention layers are not monolithic, but are composed of several attention heads that focus on different portions of the sequence.
The output of the encoder block is the one that feeds all the encoder-decoder attention layers of the decoder block, while the output of the decoder block links with a linear layer and this with a softmax layer that maps each position of the target sequence with the output vocabulary.
What is described above is the original model however, after its presentation a large number of models derived from the transformer architecture have appeared. For example, one of the most successful is BERT [11], which basically eliminates the decoder block present in transformers, and in its training the input sequences are masked in such a way that it processes them bidirectionally.
Another point to highlight is that as part of these architectural-models a series of data-models pre-trained in an unsupervised manner with large datasets have been released. This allows us to easily apply transfer-learning to different tasks such as those mentioned at the beginning of this section.
Next, we will describe how we have used some of these models in the check-worthiness and fake news detection tasks.
The objective of Task1a-English [12] is, given a set of tweets in English language related to the COVID-19 topic, to identify which tweets are worth checking by assigning a score to each of them.
To tackle this task we eliminated any metadata present in the tweets and have focused only on the textual information provided.
Taking into account that all the tweets to be evaluated are about COVID-19, we have searched a well-known repository of pre-trained models1 , and we have found one that is trained in tweets related to this topic.
Finally, we have used the BERTweet model [13], a BERT-architecture model initially pretrained with 850 million tweets in English using the RoBERTa [14] pre-training procedure, to which the same authors performed a second 40-epoch pre-training with 23 million English tweets related to the COVID-19 topic.
To check if actually using a pre-trained model for the same topic and document type had a superior behavior to other pre-trained models and architectures in more neutral datasets, we implemented a grid search procedure in which we varied the number of periods, the size of the lot and the model/architecture used. The rest of the hyperparameters have been kept in the default values that each model has.
Among the transformer models we have tested are BERT, ALBERT [15], RoBERTa, DistilBERT [16], and Funnel-Transformer [17]. Table 1 shows the best results obtained for each model for the mean average precision, F1, precision-recall curve and ROC curve measurements, sorted by mean average precision.
As we can see, the best behavior is obtained with the model that is pre-trained in tweets related to the COVID-19 topic.
Therefore we select the first two models bertweet-covid19-base-uncased and bertweet-covid19base-cased and we test various values of the epsilon parameter obtaining the best results with the value 2.5 × 10 −9 . These results are shown in Table 2.
We also found that although we always initialized the Python, NumPy, and PyTorch random number generators with the same seeds, the same results did not always appear for a given set of parameters. Therefore, to make the final shipments, we do not join the training and dev datasets to have a larger one with which to train the models, but we train the models with the training dataset and evaluate them with the dev dataset, repeatedly executing the same configurations of parameters and selecting the test files to send from the best results obtained on the dev dataset, assuming that an initial random configuration that behaved well in the dev dataset would also do so in the test dataset.
In this version of Task 1A, the set of tweets is defined in Spanish language and these tweets are related to issues of Spanish politics.
As in Task 1A English, we have used several transformer models to evaluate which one best suits these types of tweets. The tested models have been BERT, Electra [18] and RoBERTa.
After a preliminary grid search with different pre-trained models in Spanish and different values of batch size and epochs, keeping the rest of the hyperparameters in their default values, we obtained the results shown in Table 3. The best results are shown for each pre-trained model.
Since the model Electra mrm8488-electricity-base-discriminator 2 is the one with a slightly higher result, it is the one we selected for a more exhaustive search for parameters. This Electra model is pre-trained with 20GB of the Spanish-language Oscar corpus [19].
We also realized, extracting the vocabulary from this pre-trained model, that among the first 1000 tokens there were 971 unused tokens of type [unusedNNN].
To see if these tokens could be useful, we pulled all the out-of-vocabulary tokens of the training dataset. From this set of words, we manually selected those that seemed most relevant to us and had three or more appearances, mainly the names of politicians, political parties, the media, and hashtags used in electoral campaigns. In total, the list consisted of 197 tokens.
With this list, we create a dictionary to group tokens that correspond to the same concept. For example, #PINParental, pin and parental were matched with the same PINParental token.
In this dictionary, we substitute the tokens on the right side by tokens [unusedNNN] to obtain a match between the out-of-vocabulary tokens with the unused tokens of the model, and both in the training loop and in the evaluation loop we did the replacement of the out-of-vocabulary tokens using this dictionary.
Unfortunately, the results obtained with this strategy were not as expected, obtaining better results without substituting out-of-vocabulary tokens. The best results obtained after repeated Task MAP MRR RP P@1 P@3 P@5 P@10 P@20 P@30 1A Spanish 0.492 1.000 0.475 1.000 1.000 1.000 0.800 0.800 0.620 1A English 0.224 1.000 0.211 1.000 0.667 0.400 0.300 0.200 0.160 runs with different batch sizes and epochs are shown in Table 4, along with the best results obtained by substituting tokens.
To send the submissions to this version in Spanish of subtask 1A, the same strategy was used as in the English version: training the model repeatedly for the same parameters and send the configurations with the best values in the dev dataset.
Finally, two submissions were made for the Spanish version of Task 1A and three submissions for the English version. The official evaluation measure was mean average precision (MAP). In Spanish we obtained the fourth position among six participants while in English we obtained the first position among ten participants. The results are shown in Table 5.
The approaches to the detection of fake news that have been made so far can be divided into three groups: knowledge-based methods, content-based method and context-based methods.
In the former, each claim is compared with a source of evidence that supports that claim. The source of evidence can be a knowledge graph [20] in which case we must extract subjectpredicate-object triples from the claim and verify their existence in the graph, or we can be use as a source of evidence the information retrieved from a query to a search engine [21], having then to compare the information obtained with the claim using techniques such as similarity, stance detection, contradiction detection, etc.
Content-based methods only use the textual information in the document. The features obtained can be latent, such as word or sentence embeddings, or explicit such as TF-IDF vectors, bag of words vectors, word counts [22], psycho-linguistic features [23], etc. Transformers and RNNs can also be considered as a content-based method that uses latent features.
In context-based features the information surrounding the claim is used to verify its degree of truthfulness. Examples of these features can be those based on propagation [24], based on the user's reputation [25], based on their profile [26], etc.
For the fake news detection task in English [27], from a set of news articles we have to classify each item in one of the following categories: true, partially true, false, or other [28][29] [30], taking into account the main claim of the news article.
The organizers provided three different training datasets [31], so we joined these three datasets and left 20% as a dev dataset for a total of 760 training instances and 190 validation instances.
To tackle this task we have used two different approaches. The first of them is, as in the tasks dedicated to determining the check-worthiness of a sentence, to use transformer models to check if the latent features that these models extract from the documents can be related to their veracity.
The second approach is to use the more classical ensemble methods together with various types of features such as TF-IDF and LIWC.
A grid search has been carried out with four different transformer models: ALBERT, BERT, DistilBERT and Funnel-Transformer, and different batch sizes and number of epochs.
Given that one of the limitations of the transformer models is the length of the sequence that they accept as input, we have assumed that the relevant information for each news article is likely to be found at the beginning of it. In this way we have extracted the first 150 and 200 tokens as input for the models. We have also tried to use the first 150 tokens of the article title as input. As some instances had no title, in those cases we have used the first 150 tokens of the article text. The four possible class values have been converted to integer values so that they could be processed correctly.
The Table 6 shows the best results obtained for each transformer model. Given that this is a multi-class classification, we have used precision, coverage and F1 as evaluation measures, taking this last measure as the main one. As can be seen, the title of the article does not seem to contain enough information about its veracity, and a longer sequence length provides better results, as expected.
In this second approach we use the random forest [32] and gradient boosting [33] classifiers. We extracted the text of each article and processed it with the LIWC2015 [34] text analysis tool, obtaining a total of 93 discrete features3 such as Analytic, Clout, Authentic, Tone, etc. The use of LIWC in this task is motivated by the premise that false articles may have certain linguistic features that are not present in legitimate articles, and this can be reflected in the results offered by this tool. We also extract the TF-IDF vectors as features from the text of the articles.
To build the latest feature set, for each article we do a Google search using the article title as query terms.
From the first 20 results obtained, we extract the domain names from each URL and concatenate them, separating them with spaces, constructing text strings with the shape "www.politifact.com www.reuters.com www.nytimes.com apnews.com ... ". With these strings we also build a TF-IDF representation. Thus, we assume that if domain names of sites dedicated to fact-checking appear among the first 20 results, that article is at least suspected of containing some controversy. On the other hand, if the domain names are from prestigious media, the original article, true or false, may be important.
To select the proper configuration, we keep the LIWC features fixed, and we try to optionally concatenate the text TF-IDF features and the domain names TF-IDF features.
In Random Forest the number of estimators has been established at 100, the maximum depth of the tree at 1000 and as a criterion to evaluate the split quality gini has been used. In Gradient Boosting the number of estimators has also been set to 100 and as a loss function deviance has been used. The result of these tests is shown in Table 7.
As can be seen, the Gradient Boosting classifier is superior to Random Forest in all feature configurations. It is also able to take advantage of the information provided by all the concatenated features, while the Random Forest classifier obtains the best result when only the LIWC features are used.
In this task we have made three submissions. The first one has been generated by Gradient Boosting with the three types of features: LIWC, domain names TF-IDF and text TF-IDF. The second submission we have done with Albert transformer with albert-base language model and the article text as input with a sequence length of 150. Moreover, for primary submission we have used the same type of transformer but with a sequence length of 200.
With the best of these submissions we have achieved an F1-macro measure of 0.468 which places us in fourth position among 25 participants.
Table 8 shows our reproduction of the results obtained by the three submissions. Unlike what happened in the dev dataset, with the test dataset the best model has been the Gradient Boosting classifier that uses the features based on LIWC, domain names TF-IDF and text TF-IDF. This tells us that although transformer models can perform well in the fake news detection task with little or no feature engineering, the use of text analysis tools like LIWC along with other handcrafted features can still be useful for profiling fake news.
In this edition of CheckThat! Lab, our team has explored the two main tasks in detecting fake news: the selection of sentences or tweets to verify and the verification of these elements themselves.
Regarding the check-worthiness task, we have verified that the transformer models can extract the latent features present in the tweets more efficiently than other methods, although is necessary to carefully choose the appropriate data model for the task, with large performance differences between some models and others.
Our participation in the English version of this task has been very positive, obtaining the first position, while in the Spanish version we have been in fourth place. We have also detected that in Spanish the mean average precision on the dev dataset (0.495) was much lower than that obtained in English (0.849). This may be due to the fact that the dataset used is not specifically pre-trained on tweets or on Spanish politics.
In the task of detecting fake news we have participated with two different approaches. On the one hand, we have used transformer models trying to extract linguistic features that identify fraudulent articles, and expecting good behavior from them. On the other hand, we have used a fairly simple Gradient Boosting classifier that uses linguistic features extracted through the LIWC tool, TF-IDF text features, and a TF-IDF representation of domain names retrieved from a Google search. We have used this second system as contrastive submission since its results were inferior to those of the transformer models. However, in the test dataset the best performance was obtained with this last model.
Being our first participation in a fake news detection task, the result was positive, obtaining fourth place among 25 participants.
We think that although it can always be improved, the check-worthiness task can be approached reasonably well by means of transformers models, so our future work will be mainly devoted to investigating alternative methods to those used in this laboratory to tackle the task of fact-checking and detection of fake news, for example using knowledge methods to verify claims.
This work has been partially supported by the Spanish Ministry of Science and Innovation within the DOTT-HEALTH Project (MCI/AEI/FEDER, UE) under Grant PID2019-106942RB-C32, as well as project EXTRAE II (IMIENS 2019) and the research network AEI RED2018-102312-T (IA-Biomed).