In the age of the internet, people interact with each other more often than ever. Almost everybody with internet access has an afiliation with a social media website. With this popularity, spreading of misinformation has inevitably become a huge problem of the current age. In recent years, 2016 US Presidential Election brought the attention to fake news. With the Coronavirus Pandemic misinformation became an increasingly popular area to research in academia. To be a part of the research on detecting misinformation in the internet, we participated in task 3: Fake News Detection of the Checkthat! Lab at CLEF2022. In this paper, we show the details of our system consisting of data collection, transformer based pre-trained models and extensive preprocessing methods. We achieved an F1-score (macro) of 0.328 against a top score of 0.339 on the oficial test set.
In the information age, internet became the main source of news on what is happening in the
world. Individual access to the internet became easy and afordable which gave people a massive
freedom to obtain and share information online. Although there have been major benefits of
this freedom, it often came with a cost that is called misinformation. Misinformation is seen in
variety of forms [
In recent years, misinformation became a significant research area in natural language
processing. Some of the past studies focused on rumor detection [2, 3, 4, 5, 6], fake news
detection [7, 8, 9, 10, 11, 12], spam detection [13, 14, 15, 16, 17] and bot detection [
Task 3 of the Checkthat! Lab at CLEF2022 [
In the following sections, we show the features of the datasets we used in this work, our methods, experiments, results, error analysis and our conclusion.
There are multiple datasets with mixed domains that focus on fake news detection. In this section, we show features of the used datasets in our work. In table 2, you can find the label count for all the datasets below. Label counts are calculated after dropping NaN values, bad lines and duplicates except for oficial test set. In table 3, you can find the nfial distribution of labels in the training set, dev set and test set that have been used in our work.
Introduced in [
This combined dataset consists of 2 seperate datasets, each includes article titles and article texts. In true dataset, all samples are considered as true, in fake dataset all samples are considered as fake. We randomly chose 30k of the samples and included them in our training set. The exact version we used in our work can be found at here and here. Unfortunately we could not find the original source that introduced these datasets.
This dataset has only article texts and their labels. Fake label is defined as "potentially unreliable". We randomly chose 1k of the samples and included them in our training set. The dataset can be found from Kaggle and it was provided by Kai Shu to the competition organizers.
This is the oficial dataset that is released by task organizers privately for the participants [
This is oficial test set released by task organizers. It contains article id, article text, and all 4 labels. It can be found at Zenodo. We calculated our final scores based on our predictions on this dataset.
In this section, we give details about the methods we employed for the task. They consist of text preprocessing and fine-tuning pre-trained language models. 3.1. Preprocessing • Concatenating title and content when available: In some datasets, there exists article title column that contains the title of the articles. In this case, we merged title and article content in one sequence of text. • Converting to lower case: It is usually unhelpful to keep the characters in both lowercase and uppercase form. • Removing stop words: Stop words are usually the most frequently occurring words in natural language and they do not contribute much to the meaning. It’s a widely used practice to remove them from text before feeding the sequences to the model. • Removing punctuation: As stop words, punctuation marks are usually unnecessary to keep. • Standardizing certain words: In order to make the text as clear as possible, we used some pre-defined tokens to replace urls, email addresses, phone numbers, names, numbers, digits and currency characters. • Lemmatizing: In order to simplify the words we used lemmatizing over stemming to avoid creating words that are not in the dictionary or lost their meaning. • Shortening: We shortened the sequences to 500 tokens and 4000 tokens in order to fit them into the models, depending on the model’s capacity.
We used the Natural Language Toolkit (NLTK) [
Transformer based pre-trained language models have become significantly popular in recent
years due to the fact that they led to state of the art improvements in many natural language
processing tasks [
In the model development stage, we used our custom split of training and dev sets that are explained in section 2. We used 4 32GB Nvidia V100 GPUs. After the initial experiments on dev
set, we decided to choose 2e-05 as learning rate, 64 as batch size to focus more on number of epochs for the rest of the experiments. For the longformer model, we had to reduce the batch size to 2 due to GPU limitations. The results obtained in the development stage can be found in table 5 with metrics micro f1 and macro f1.
• It can be said that using longformer did not lead to the results we anticipated. • Longer training had diminishing returns. • Macro f1 scores are lower than micro f1 scores because the performance on certain label(s) is significantly worse.
As seen from Table 5, best model setup is obtained by distilbert model trained for 20 epochs with batch size of 64 and learning rate of 2e-05. Furthermore, we explored how the model would perform with no additional data. We used the best model setup and trained on just oficial training set with no hyperparameter search. Apart from that, we explored the efect of preprocessing methods. For this, we used the same setup and trained on just oficial training set with no preprocessing. In table 6, we show the results we obtained on oficial test set in terms of accuracy, macro precision, macro recall, macro f1 metrics.
Results show that additional data hurt the performance the most. One cause might be the
diference in domains of the oficial dataset and additional datasets. We anticipated that for
sequence classification, combining multiple domains might lead the model to make better
predictions as shown in [
In this section we present the confusion matrix on oficial test set for the model that was trained on oficial training set after preprocessing (second model in table 6) in table 7. The table shows the model managed to learn the most detecting false claims and struggled to see other labels. This can be explained by the challenging nature of the data and the dominance of "False" label in the training set.
We participated in task 3: Fake News Detection of the Checkthat! Lab at CLEF2022 and developed models to detect and classify misinformation in the internet. We applied extensive preprocessing methods and fine-tuned several pre-trained language models with the released dataset and additional datasets. We found that being able to feed longer sequences and additional data with mixed domains did not improve performance, preprocessing and smaller model led to better predictions.
For the future work, one might explore extra pre-training an already pre-trained language model with data that has similar nature to the oficial training set to develop better models. In this task, class imbalance seems to be a significant issue. Therefore another direction might be exploring data augmentation methods to increase in domain data or modifying loss function to increase penalty for misprediction of samples of the underrepresented label(s). [2] J. Yu, J. Jiang, L. M. S. Khoo, H. L. Chieu, R. Xia, Coupled hierarchical transformer for stanceaware rumor verification in social media conversations, Association for Computational Linguistics, 2020. [3] S. Kwon, M. Cha, K. Jung, Rumor detection over varying time windows, PloS one 12 (2017) e0168344. [4] Q. Zhang, S. Zhang, J. Dong, J. Xiong, X. Cheng, Automatic detection of rumor on social network, in: Natural Language Processing and Chinese Computing, Springer, 2015, pp. 113–122. [5] S. Hamidian, M. T. Diab, Rumor detection and classification for twitter data, arXiv preprint arXiv:1912.08926 (2019). [6] T. Takahashi, N. Igata, Rumor detection on twitter, in: The 6th International Conference on Soft Computing and Intelligent Systems, and The 13th International Symposium on Advanced Intelligence Systems, IEEE, 2012, pp. 452–457. [7] V. Pérez-Rosas, B. Kleinberg, A. Lefevre, R. Mihalcea, Automatic detection of fake news, arXiv preprint arXiv:1708.07104 (2017). [8] J. C. Reis, A. Correia, F. Murai, A. Veloso, F. Benevenuto, Supervised learning for fake news detection, IEEE Intelligent Systems 34 (2019) 76–81. [9] K. Shu, A. Sliva, S. Wang, J. Tang, H. Liu, Fake news detection on social media: A data mining perspective, ACM SIGKDD explorations newsletter 19 (2017) 22–36. [10] R. K. Kaliyar, A. Goswami, P. Narang, Fakebert: Fake news detection in social media with a bert-based deep learning approach, Multimedia tools and applications 80 (2021) 11765–11788. [11] C. Liu, X. Wu, M. Yu, G. Li, J. Jiang, W. Huang, X. Lu, A two-stage model based on bert for short fake news detection, in: International Conference on Knowledge Science, Engineering and Management, Springer, 2019, pp. 172–183. [12] J. C. B. Cruz, J. A. Tan, C. Cheng, Localization of fake news detection via multitask transfer learning, arXiv preprint arXiv:1910.09295 (2019). [13] A. Gupta, R. Kaushal, Improving spam detection in online social networks, in: 2015 International conference on cognitive computing and information processing (CCIP), IEEE, 2015, pp. 1–6. [14] T. Wu, S. Liu, J. Zhang, Y. Xiang, Twitter spam detection based on deep learning, in:
Proceedings of the australasian computer science week multiconference, 2017, pp. 1–8. [15] G. Jain, M. Sharma, B. Agarwal, Spam detection on social media using semantic convolutional neural network, International Journal of Knowledge Discovery in Bioinformatics (IJKDB) 8 (2018) 12–26. [16] G. Jain, M. Sharma, B. Agarwal, Optimizing semantic lstm for spam detection, International
Journal of Information Technology 11 (2019) 239–250. [17] J. Cao, C. Lai, A bilingual multi-type spam detection model based on m-bert, in: GLOBE
COM 2020-2020 IEEE Global Communications Conference, IEEE, 2020, pp. 1–6. [18] M. Heidari, J. H. Jones, Using bert to extract topic-independent sentiment features for social media bot detection, in: 2020 11th IEEE Annual Ubiquitous Computing, Electronics & Mobile Communication Conference (UEMCON), IEEE, 2020, pp. 0542–0547. [19] S. Feng, Z. Tan, R. Li, M. Luo, Heterogeneity-aware twitter bot detection with relational graph transformers, arXiv preprint arXiv:2109.02927 (2021). domain and multi-label uncivil language detection, in: Proceedings of the Fourth Workshop on Online Abuse and Harms, 2020, pp. 28–33.