Checkworthy content refers to information that must be confirmed for accuracy, as it may have the potential to shape the opinions and decisions of others. The rise of social networks has led to an exponential growth in textual data on the internet, sometimes resulting in the spread of false claims that can be detrimental to society if left unaddressed. These claims can include political, religious, and health-related misinformation, which can cause discord in society. Fact-checking is a time-consuming task that requires extensive research, identification, verification, and expert analysis. Automating this entire process is a significant challenge, and the first step towards this goal is to determine whether the information is worth checking in the first place.
With the proliferation of communication and social media platforms, such as Facebook, Twitter, and Reddit, the dissemination of false information has become increasingly prevalent. A recent study has suggested that people struggle to differentiate facts from false news [1]. Intelligent technologies can be used to support human fact-checkers to identify claims worth fact-checking [2]. Many studies have been devoted to developing a fully automated system for fact-checking [3], [4], [5], [6], [7]. As social media data continues to expand daily, it is impractical to monitor everything efficiently by human experts. Therefore, developing an automatic system has emerged as the ultimate solution to this problem. This work proposes a solution to classify English, Arabic, and Dutch texts or tweets as checkworthy, harnessing the power of BERT-based approaches. The critical contributions of this study are:
• Introducing a fine-tuned transformer-based model to classify checkworthy texts for three languages (English, Arabic, and Dutch). • Exploring various machine learning (LR, SVM, and MNB), deep learning (CNN, CNN + LSTM, and CNN + BiLSTM), and transformer-based models for finding a suitable method for detecting checkworthy texts in multiple languages.
Inaccurate news is quickly spreading throughout social media. Checking the authenticity of any post that surfaces on social media becomes crucial. Intelligent fact-checking systems have emerged as a significant area of research to tackle this problem. Several domains allow for the detection of trustworthiness, such as digital scam [8], the healthcare sector [9], politics [10], and many more fields. An overview of Task 1 in the fourth edition of the CheckThat! The lab was provided by Shaar et al. [11]. Their job was anticipating which tweets involving politics and COVID-19 needed to be verified. Williams et al. [12] presented a transformer-based solution with data augmentation for this problem, and it received an mAP (mean average precision) of 0.66 in the Arabic language. Checkworthiness in multimodal [13] is another popular research area these days; in addition to unimodal, Sadouk et al. [14] proposed a multimodal transformer-based model (BERT+ResNet50) to identify checkworthiness in English which recorded F1 score of 0.71 and transformer based model (MarBERT) with downsampling recorded F1 score of 0.61 in Arabic for image dataset. Meanwhile, Ivanov et al. [15] proposed audio datasets from past political debates and ensemble techniques for detecting checkworthiness. Their audio model (wav2vec2.0) received a mAP of 0.34 when extra noise was eliminated. Ensembles using BERT and an audio model outperformed BERT alone, with a mAP of 0.38. This work addresses a significant gap in the existing literature by comprehensively comparing machine learning (ML), deep learning (DL), and transformer-based solutions. In addition, it investigates the use of few-shot models like SetFit for determining check worthiness in Dutch, Arabic, and English. This study improves the understanding of the various models' performance in these distinct languages.
The dataset consists of tweets or texts in English, Arabic, and Dutch languages, along with their corresponding labels ('Yes' for texts worth checking, 'No' otherwise). Table 1 shows the distribution of train, dev, dev-test, and test sets. We trained all models using the training set and evaluated the model's performance based on the test set. CLEF 2024 -CheckThat! Lab [16,17,18] consists of six tasks [19,20,21,22,23]. We participated in task-1 of this shared task. Task-1 [19] focuses on assessing whether a claim in a tweet or transcription requires further investigation for fact-checking. The traditional approach for such decisions involves human experts, either professional fact-checkers or annotators, who evaluate the claim based on various criteria. Table 2 illustrates an example of training data for the different languages.
This task exploited various ML, DL, and transformer-based approaches across all three languages. MLbased techniques used include linear regression (LR), support vector machine (SVM), and multinomial naive Bayes (MNB). DL-based techniques involve CNN, CNN+long short-term memory (LSTM), and CNN+bidirectional LSTM (BiLSTM). Lastly, various BERT-based transformers are fine-tuned for each language for the given task. Figure 1 illustrates the schematic process of checkworthy text detection. Textual Feature Extraction: Textual feature extraction is one of the essential steps in natural language processing, which involves transforming raw textual data into numerical representations. This numerical representation aids the models in understanding and processing textual data. A Count Vectorizer is used in the ML models examined in this work. It is a widely used technique for textual feature extraction that transforms text data into a matrix of token counts. In DL models, tokenization and padding combine to convert raw texts into structured numerical data. These numerical representations are then passed through an embedding layer, which captures more advanced features such as semantic relationships. This study uses the embedding layer instead of Word2Vec [24] or GloVe [25] to allow the model to learn task-specific embedding during training. Finally, BERT-based tokenizers are employed for transformer-based models to exploit the BERT architecture. ML Models: Various ML models are examined in this work, such as LR, SVM, MNB, KNN, and RF. All the hyperparameter settings for these models are illustrated in Table 3.
This work employed a CNN model comprising an embedding layer with an output dimension of 200. The model features two Conv1D layers with 64 and 128 filters, respectively. Both layers used a kernel size of 2 and ReLU activation. For downsampling, the model incorporates a GlobalMaxPooling1D layer. Subsequently, a dense layer with 128 units and ReLU activation is followed by a dropout layer with a rate of 0.5 to prevent overfitting. The output layer has a single unit with sigmoid activation. The model utilizes the 'binary_crossentropy' loss function and 'Nadam' optimizer and has trained with a batch size of 32 for three epochs.
CNN+LSTM: The CNN+LSTM model used in this work has almost the same architecture as the CNN model, incorporating a single LSTM layer comprising 64 units and a dropout rate of 0.2 for sequence modeling. Furthermore, the dense layer included in this design features 64 units and utilizes the ReLU activation function. The remaining hyperparameter configurations are consistent with those employed in the CNN model.
CNN+BiLSTM: This model has an architecture similar to the CNN+LSTM model but replaces LSTM with a Bidirectional LSTM.
Transformer models for English: This study fine-tuned three transformer-based models for a specified task in the English dataset. The models employed were BERT-Base-Uncased [26], SetFit [27], and RoBERTa [28]. The necessary text preprocessing steps were followed before feeding the data into the transformers. These text preprocessing steps include lowercasing, emoji removal, stop word removal, stemming, contraction expansion, simple Unicode spelling correction, and HTML tag removal. For stop word removal, the NLTK stopwords list is used. The main agenda of the text preprocessing steps was to reduce the noise in the dataset and focus on meaningful words. The BERT-Base-Uncased used in this task is a pre-trained transformer model with exceptional performance across various natural language processing (NLP) tasks. This model demonstrated satisfactory performance on the specified task. On the other hand, SetFit leverages pre-trained transformers with limited labeled data. We explored the potential of this few-shot learning framework for the given task. SetFit does not require manual prompts for classification, in contrast to LLMs. Finally, RoBERTa, another optimized version of BERT, is used here for the specified task and outperforms other models.
Transformer models for Arabic: This study also exploited three transformer-based models and fine-tuned them in the Arabic dataset. Models used for the Arabic dataset were AraBERTV2 [29], SetFit (Few-shot) [27], and Dehate-BERT [30]. Similar to the English dataset, some text-preprocessing steps were also performed. Again, the preprocessing steps are lowercasing, emoji removal, stop word removal, stemming, contraction expansion, simple spelling correction using Unicode, HTML tag removal, punctuation removal, URL removal, whitespace removal, and number removal. Stemming was performed here using ArabicLightStemmer. Finally, normalization was used to convert similar characters to a standard form.
AraBERTV2 is the improved version of AraBERT, which leverages the BERT architecture. This model was trained on a sizeable Arabic dataset and has demonstrated effectiveness in various downstream NLP tasks, including sentiment analysis, NER, and Arabic question answering. Dehate-BERT is a pre-trained transformer model primarily designed for hate speech detection, and it outperformed all other models in Arabic in this specific task.
Transformer models for Dutch: This study investigated Dutch RoBERTa [31], SetFit, and Dutch-BERT in the Dutch dataset. Rather than undertaking extensive text preprocessing, this work limited its processing to removing non-Dutch characters from the texts. Table 4 illustrates the hyperparameters of transformer-based models.
Table 5 illustrates an in-depth analysis of the performance of ML, DL, and transformer-based models in English, Arabic, and Dutch on the test set.
In evaluating English language data, logistic regression (LR) achieved a precision of 75.00%, recall of 37.50%, and F1-score of 50.00%. However, support vector machine (SVM) emerged as the top-performing ML model, with a precision of 68.97%, recall of 45.45%, and the highest F1-score of 54.79%. Multinomial Naive Bayes (MNB) showed a precision of 64.81%, recall of 39.77%, and F1-score of 49.29%. Among DL models, CNN+BiLSTM demonstrated the best performance with a precision of 66.10%, recall of 44.32%, and F1-score of 53.06%. Among Transformers, RoBERTa showcased remarkable performance, achieving a precision of 89.23%, recall of 65.91%, and the highest F1-score of 75.82%. The SetFit model achieved a precision of 52.34%, recall of 63.64%, and an F1-score of 57.44%. This model shows comparable performance to well-known models such as SVM and LR.
Arabic language evaluations revealed LR achieves a precision of 38.52%, recall of 21.55%, and F1-score of 27.65%. However, SVM followed closely with a precision of 40.57%, recall of 32.57%, and F1-score of 36.13%. MNB showed a precision of 36.00%, recall of 24.77%, and the highest F1-score of 29.35%. Among DL models, CNN+BiLSTM showed the best performance with a precision of 34.27%, recall of 27.98%, and F1-score of 30.81%. Among Transformers, Dehate-BERT emerged as the top-performing model with a precision of 40.24%, recall of 75.69%, and the highest F1-score of 52.55%. The SetFit model attained a precision of 37.75%, recall of 69.26%, and an F1-score of 48.86%. Although it does not match the performance of the leading transformer-based model, Dehate-BERT, it still demonstrates potential in tackling Arabic language classification issues.
For Dutch language evaluations, LR achieves a precision of 50.98%, recall of 32.75%, and F1-score of 39.88%. SVM showed a precision of 43.86%, recall of 37.78%, and F1-score of 40.60%. MNB attained a precision of 46.91%, recall of 9.57%, and F1-score of 15.89%. Among DL models, CNN+BiLSTM In general, transformer-based models surpass both ML and DL models across various languages, demonstrating the effectiveness of pre-trained language models in numerous natural language processing applications. Additionally, within each category, specific models show superior performance, emphasizing the necessity of choosing the appropriate model based on the specific task and language.
A comprehensive quantitative and qualitative error analysis is conducted to provide detailed insights into the proposed model's performance.
Figure 2 illustrates the confusion matrix of the best-performing models across English, Arabic, and Dutch.
From a total of 341 test cases in English, RoBERTa demonstrates strong performance in identifying the positive class, with 247 True Positives and only 6 False Positives. This indicates a high precision, meaning the model is highly accurate when it predicts "Yes." Additionally, with 57 True Negatives, it correctly identifies many negative instances. However, there are 31 False Negatives, which indicates that some positive instances are being missed. RoBERTa shows a balanced approach with notable proficiency in minimizing incorrect optimistic predictions, resulting in a high F1 score of 75.82% over
Table 6 presents some actual labels (AL) and predicted labels (PL) of the developed models.
Few predictions with actual and predicted label.
It is clear that the models accurately predicted the labels for examples 2, 3, and 5 but made errors with examples 1 and 4. For the first example, the sentence's intent is ambiguous, leading to an incorrect label prediction by the model. In the case of example 4, although the sentence is checkworthy, the model mislabeled it due to inadequate training data in the Dutch language, which hindered proper learning.
This work investigated the various ML, DL, and transformer-based models for identifying checkworthy tweets or texts in English, Arabic, and Dutch. The results indicate that transformer-based models shine in this task and exhibit exceptional capability in detecting checkworthy text. Specifically, RoBERTa excels in English, Dehate-BERT for Arabic, and Dutch-BERT for Dutch, achieving the highest F1 scores of 75.82%, 52.55%, and 58.42%, respectively. The study recommends that further advancements be made by increasing the training data and incorporating advanced LLMs and GPT models.