With today's modern technology, breaking news, from the latest celebrity gossip to updates on unprecedented events like the COVID-19 pandemic, are now available with just a few taps on your smartphone. As the availability and volume of readily available information has grown, so has the rise of misinformation. Fake news is specifically designed to plant a seed of mistrust and exacerbate existing social and cultural dynamics by misusing political, regional, and religious undercurrents [1]. "In 2019, 8 percent of engagement with the 100 top-performing news sources on social media was dubious. In 2020, that number more than doubled to 17 percent" [3]. Twitter's purpose has been advertised to the public as a platform that "uniquely provides its users the opportunity to discover what's happening in the world" [4]. Unique includes fake, so the Twitter platform has become an easy target for the rapid dissemination of skewed facts to the world, as seen with the attribution of the current COVID-19 pandemic to novel 5G technology. Topical automated classification systems with potent predictive power for innumerable conspiracies are urgently needed to curb the spread of inaccurate news. In this paper, we focus on content-based fake news detection strategies.
This problem of misinformation in social media is universally faced by any user of a social media site. These users, as well as the private companies who run these social media sites, have a vested interest in ensuring that the information on the platform is beneficial to the consumer (the users). For most users, this means that information is accurate and can be trusted as valid. For example, rumors have surfaced in the past about McDonald's use of worm filler in its food. This has caused tremendous boycott threats [2]. retrieved using TwitterAPI. We used several pre-processing methods on the data. First, we used the baseline pre-processing [5], which included converting to lowercase; removing punctuation; preserving URLs; removing stop words; and normalizing terms ("u.k" to UK). Our pre-processing enhancements to the pipeline this year include removing usernames (Twitter handle); removing all special characters; removing hashtags; removing contractions (e.g., convert "won't" to "will" and "not"); removing non-English Tweets if present, removing links (which not only incorporates "https://t.co/", but also "http" and "www"), and removing Emojis. When we looked at the dataset for Subtask 2 and 3, the Tweet was divided into several parts, and each part was present in a separate column. To deal with this, we merged them into one column in the data frame separated by a space. The validation size was set to 0.2 to partition our dataset for the sake of evaluating our model's predictive power according to a set of predefined metrics.
The objective is to build a multi-class classifier that can flag whether a Tweet promotes, supports or discusses at least one (or many) of the conspiracy theories. 41.88% Pre-Processing. Links that contain or start with "https://t.co/" are removed, but links such as the ones beginning with "http" and "www" are still present even after applying the control's normalization. Username handles are also not filtered out. Data Integration. Combining two datasets requires them to have the same dimensions, as well as consistent and meaningful class labels. We observed that there are some discrepancies between these two datasets, which would impede the flow of the integration process. For this reason, before integration, we began by carefully selecting class labels from fine-grained classification that would make sense in the new dataset. We replaced the class label of tuples that is 1 with 3 and 3 with 1. We also came to a consensus that label 2 is irrelevant in our new context. Thus, we excluded all tuples having this class label. To form a uniform dataset with a uniform number of dimensions, we extracted only the "Tweet" and the "Label" dimensions from the old dataset, finally rendering the previous dataset integrable with the new one (no missing Tweets detected). Before fusion, the number of tuples of the old dataset was 5,946 rows. After integration and removing rows that are labeled as 2, we got a total of 6,769 tuples.
Modeling We chose Logistic Regression as a baseline model because it performed the best in the control experiment [5]. However, we modified it by applying some hyperparameter tuning to adjust to our new, fully integrated dataset. For example, we altered the class weight attribute to 1: 0.1, 2: 0.7, 3: 0.2. We also increased maximal iterations from 2000 to 4000 because there were some instances in which the model did not converge. We kept the same feature extracting technique (CountVectorizer) and utilized the spacy to tokenize our text. In addition, the test size was set to 0.2 to partition our dataset for the sake of evaluating our model's predictive power according to a set of predefined metrics. We utilized the voting classifier to combine several selected models. The selected models were based on similar related works on Tweets [7]. They are SVC, Multinomial NB, Logistic Regression, and Random Forest Classifier. The voting type was set to "hard."
BERT for Tweets "BERT-large was trained on 64 TPU chips for four days at an estimated cost of $7,000." [8]. Selecting a pretrained model for BERT is a crucial step when fitting your model. For instance, we initially used the pretrained model offered by Google (BERT-Large, Uncased) [9]. We ended up with dismal results. As it turns out, the BERT-Large pretrained model was trained and is based on conversational English text. However, we know that the structure and nature of Tweets are very different from those of any other text. For this reason, we searched to find a pretrained model for Tweets, and we stumbled upon BERTweet [10]. We based our code on a similar work that was already done on Kaggle, but for disaster Tweets. This code has utilized BERTweet pretrained model from "Vinai" [11] with some modifications. For example, shifting our class labels (1 to 0, 2 to 1, and 3 to 2) was a requirement for BERTweet to work. We kept the same hyperparameters (5 epochs and batch set size to 8) and changed the num_classes parameter.
Encoding & Decoding Since our data contains multiple target variables, it was beyond the models' inference capabilities of more than one dependent variable at once. So, we came up with an idea to encode every occurrence of a combination of binary target variables into a single target variable. For example, 0,0,0,0,0,0,0,0,0 has 754 occurrences and we encoded it with 0. For the sake of simultaneously reducing the number of class labels and improving generalization, we decided to apply a threshold to remove any rare combination of binaries that has an occurrence less than the threshold. We found that the ideal threshold, 20, would capture the most frequent occurrences. A total of 10 encodings (class labels) were produced after using this threshold. When the model produces a label 3, the decoding process is going to translate it back into 0,0,0,0,0,0,1,0,0. The same experiments were applied to subtasks 2 and 3, except for the data integration, as the class labels of the old dataset are irrelevant in this context.
BERTweet outperforms all models in terms of multiple metrics on validation set for subtask 1, as illustrated in Figure 2.