Many people nowadays tend to explore social media to obtain news and find information about various events and activities. However, an abundance of misleading and false information is spreading every day for many purposes, dramatically impacting societies. Therefore, it is vitally important to identify false information on social media to help individuals distinguish the truth and protect communities from the harmful effects of false information. For this reason, determining which information has the priority to be scrutinized is a significant prior step that several studies have considered. In this paper, we have addressed Subtask-1A(Arabic) of CLEF2021 CheckThat! Lab. We have done that in two steps. The first involved preprocessing the provided dataset with text segmentation and tokenization. In the second step, we implemented different models on the Arabic tweets in order to binary classify them according to whether a specific tweet is worth being considered for fact-checking or not. We mainly compared two versions of the pre-trained AraBERT model with some of the traditional word encoding methods, including the Linear SVC model with TF-IDF. The results indicate that the AraBERTv2 version outperforms the other models. Consequently, we used it for our final submission, and we were ranked third among eight other participating teams.
Social media has become a key tool for disseminating news as it allows news providers to distribute
and propagate their messages to a broader range of users. The colossal number of people using social
media every day has accelerated the rate at which fake news spreads [
Subtask 1A in the Arabic language aims at classifying real-world tweets into predefined categories. Each tweet has been labelled as to whether the tweet is worth checking or not. Hence, this is a supervised text classification problem that aims to classify upcoming tweets based on their content. We used a pretrained model that can make more accurate predictions. The main objective of this task is to develop a system that would automatically identify whether a tweet is worth checking or not.
This subtask comes with three CSV files containing collections of tweets and divided into the following three categories: • A training dataset that includes 3439 labelled tweets. • A development dataset with 661 labelled tweets.
• A testing dataset with 600 un-labelled tweets.
The training and development datasets contain four features (topic_id, tweet_id, tweet_url, tweet_text) and two labelling features (claim, claim_worthiness) . The second labelling feature classifies the tweet's check-worthiness, which is defined as whether this tweet includes any concerning claims that may have a significantly detrimental impact.
The remaining sections of this paper are as follow. The next section gives a review of related work. Section 3 details the adopted approach in our experiment, including the pre-processing phase of the provided text and the fine-tuning stage of the used pre-trained model. Section 4 provides the results of the experiments, and Section 5 provides the conclusions. Finally, Section 6 contain some acknowledgements for this work.
This section briefly describes other work which we considered while developing our experiment.
ClaimRank is a multilingual system developed for the purpose of detecting claims that are worth
checking [
Atanasova et al [
The winner team from the previous year contest CheckThat! Lab proposed a system that introduces
a claims fact-checking approach employing two models: BERT and RoBERTa [
One of the best models that delivered state-of-the-art outcomes in various NLP tasks is the AraBERT
model [
Various models can be applied for solving a binary classification problem. For example, linear machine learning models such as SVM have been used by many researchers and showed excellent results. Additionally, deep learning is considered one of the most effective domains in tackling such problems. It has demonstrated satisfactory performance in solving diverse tasks in natural language processing (NLP), such as text classification. However, the scarcity of big labelled textual datasets may lower the accuracy of the results. A transformer model such as BERT was introduced in the NLP domain to support various state-of-the-art performance tasks. Since we are dealing with Arabic tweets, we pre-trained a BERT-based model designed explicitly for Arabic called AraBERT. The following subsections cover text pre-processing and explain the model used in this experiment.
In the first stage of this phase, all sentences were split using a pre-processing function called ArabertPreprocessor.preprocess. The main idea here is to divide words from punctuation symbols and numbers by a space, as in the following example: ن رقلا _ ةقفص # ن رقلا_ةقفص#
Because Arabic has a complicated lexical structure and same-meaning vocabularies may have
various forms, we considered pre-processing steps such as segmentation. To explain this, take the
Arabic word "AlElm - ملعلا" which consists of the word "ملع " (that means science) and the prefix "لا "
(that means "the" in English). This word with and without the definite article can be duplicated and
counted as two different words [
There are a number of options for tokenization when working with a pre-trained BERT model. We used the base model ‘aubmindlab/bert-base-arabertv2’, which is more common in practice. Also, we utilized a helpful tool, called AutoTokenizer, which is essential for splitting inputs into tokens with each token encoded into a corresponding id. Then, we built a batch to feed the model using this tokenizer by setting the maximum length to 128 and the batch size to 32, padding inputs that include fewer tokens than the full length, and truncating longer inputs to the specified length. # ةيدبلأ ا _ ينطسلف _ ةمصاع _ س دقلا # ن رقلا _ ةقفص
input_ids 2 10 5987 68 token_type_ids 0 0 0 0 attention_mask 1 1 1 1 decoded text [CLS] # ةقفص _
The table above illustrates how the tokenizer function converts a given input to produce a dictionary. Due to the narrowed space of this table, we changed the max_length to 16 to exhibit this example. The keys of the returned dictionary are the strings: input_id, token_type_id, and attention_mask. Each token has a unique input id. For instance, the character hashtag (#) corresponds to the id 10. The attention_mask key determines what tokens should get noticed by the model. If the attention mask is zero, there is no input, and the padding argument fills the batch to complete the maximum length.
We used a pre-trained model called AraBERT as reported by Antoun et al. [
In this experiment, we used BERT for sequence classification to fine-tune the model with the help
of the transformer’s library. The used classifier includes 12 BERT layers and fed into a hyperbolic
tangent activation function to return a probability distribution across all the tokens. Training smaller
datasets from scratch can cause overfitting. Therefore, it is good to use models previously trained on a
vast dataset and fine-tune the model using a smaller dataset [
We tried setting a high number of epochs during the fine-tuning step to look out for overfitting and take appropriate measures. Each time we trained the model using the training and validation datasets, there was overfitting in the fourth epochs. Since there is no accurate way to determine the number of epochs, we decided to set it to 3. The table below demonstrates the average training loss and the accuracy for the validation run.
Since AraBERT has many versions, we explored different practices to determine which version we
should use for our submission. Although a study compared different AraBERT models and found that
some larger models performed better than others in terms of accuracy [
We examined the results of both versions on the development set. The results show that using the AraBERTv2-base model may increase the accuracy and the weighted average of precision, recall, and f1-score. In another attempt, we executed a baseline model that can be used for classification, particularly the Linear Support Vector Classification model (SVC) with TF-IDF. The experiment of the validation datasets indicates that this traditional word encodings method yields a weighted average F1score of 0.67 compared to the AraBERTv0.2 and AraBERTv2 with 0.83 and 0.84, respectively. These results suggest that these methods might be insufficient to model the complicated Arabic tweets.
After releasing the test gold labels dataset, we compared our predictions to the correct values. Table 3 shows a comparison between AraBERTv0.2-base and AraBERTv2-base for both labels using a classification report function. The main observation from Table 3 is that the two versions approximately performed the same. There is a slight improvement in general using AraBERTv2-base, which we decided to use for the official submission.
Finally, the submitted predictions for Subtask1A on Check-worthiness of tweets in Arabic has achieved a 0.612 Mean-Average-Precision, and our team SCUoL ranked third among eight participating teams. Although the results were encouraging, the main concern is the insufficient improvement when examining the two versions of the AraBERT model. Another contributing factor is contrasting base models with larger-size models to provide more comprehensive selections.
It is essential to gain information from reliable sources. Therefore, a considerable number of models are used to solve this problem. Due to the massive number of tweets published online, moderators want to concentrate on tweets worth checking only. This experiment examined an Arabic dataset that is labelled as to whether a tweet is worth checking. We took two steps toward our goal, including preprocessing and fine-tuning the datasets. We first approach the task by pre-processing the dataset and then applied two versions of the transformer-based model AraBERT as well as the traditional machine learning method Linear SVC model with TF-IDF. The comparison indicated that AraBERT beats the regular ML model. However, there was not a significant improvement when comparing the base versions of this model.
In the future, investigating additional features should be considered. Some related highlights for this task can be tackled not only by transformer-based embeddings but also by word-vector based models. In addition, the larger versions of the AraBERT may give a better performance.
The research behind this paper would not have been possible without the extraordinary assistance of the team and supervisors, Mohammad Ammar Alsalka and Eric Atwell. Their knowledge and consideration kept my work on track. Moreover, my colleague Abdullah Alsaleh has helped me outline the work and added a significant enhancement. Similarly, we appreciate the insightful comments offered by the reviewers.