This paper describes the results of the CheckThat! Lab 2022 Task 3. This is the fifth edition of the lab, which concentrates on the evaluation of technologies supporting three tasks related to factuality. Task 3 is designed as a multi-class classification problem and focuses on the veracity of German and English news articles. The German subtask is ought to be solved using an cross-lingual approach while the English subtask was ofered as mono-lingual task. The participants of the lab were provided an English training, development and test dataset as well as a German test dataset. In total, 25 teams submitted successful runs for the English subtask and 8 for the German subtask. The best performing system for the mono-lingual subtask achieved a macro F1-score of 0.339. The best system for the cross-lingual task achieved a macro F1-score of 0.242. In the paper at hand we will elaborate on the process of data collection, the task setup, the evaluation results and give a brief overview of the participating systems.
During the COVID-19 pandemic, the World Health Organization did not only speak of a
pandemic, but even an Infodemic, due to the vast amount of false information about the
disease1. Regarding the huge societal impact of misinformation, research on the topic received
a lot of attention in the past years. To counter the spread of wrong and potentially harmful
information, researchers explored diferent approaches to detect fake news in diferent media
forms such as social media [
The remainder of this paper is organized as follows: Section 2 gives an overview on the state-of-the-art research, section 3 provides detailed information on the task, while section 4 explains the process of data collection. Section 5 presents the evaluation results and gives an overview of the participants approaches. Section 6 describes our baseline classifier and lists detailed descriptions of the diferent approaches used the by the individual participating teams. Finally, we provide a brief conclusion and an outlook on potential future work in section 7.
Much work was recently dedicated towards the identification of misinformation in general
[12, 13, 14] and in particular in social media [15, 16]. Fake news detection for social media
poses several challenges which require more research. Among them are visual content [
Several other initiatives related to the CheckThat! Lab at CLEF aim to advance research on fake news detection. MediaEval 2021 ofered a text-based fake news and conspiracy theory detection task with three subtasks [21]. Their data originates from Twitter posts and news articles and the main topic in the data is COVID-19. A similar task was already hosted by MediaEval 2020 [22] focusing on COVID-19 and 5G conspiracy theories.
RumourEval [23, 24], as part of SemEval, addressed stance detection and also classifying tweets according to the truthfulness. Other SemEval tasks concerned stance [25], and propaganda detection [26] as well as fact-checking in community question answering forums [27]. FEVER [28, 29] focused on Wikipedia data for supporting or invalidating claims.
In 2019, the Qatar International Fake News Detection and Annotation Contest 2 was conducted.
The task description is similar to last year’s iteration of CheckThat! ’s Task 3 [17]. The
ifrst subtask focused on the classification of news articles, detecting if an article is fake or
legitimate [
2https://sites.google.com/view/fakenews-contest
Another related shared task is the Fake News Challenge Stage 1 (FNC-I)3, which centered around stance detection. The aim was to develop automatic systems that, given a random pair of a title and the body of two diferent or the same article, classify the pair in one of four stance classes: Agrees (if the body text agrees with the title), Disagrees (if the body text disagrees with the title), Discusses (if the body text discusses the title without taking a position), Unrelated (if the body text and title are unrelated). Participants were given a dataset that consists of titles and bodies of news articles as well as a the stance dataset 4. In the latter, a pair of title and body were allocated the according stance. The most successful submission5 applied both a XGBoost classifier and a 1D convolutional neural network classifier. The weighted average of those two classifiers was taken as output.
The technology for detecting misinformation can be broadly categorised into knowledge based
approaches which use knowledge bases and compare claims to them in some way (e.g. [
The CheckThat! Task 3 evaluates systems which predict the veracity of news articles and is designed as a multi-class classification problem. In 2022, the second iteration of the task was conducted. As in 2021, the task is ofered as a monolingual task in English. Additionally – in line with the general CLEF mission – the task was also ofered as a cross-lingual task this year, providing English training and German test data. The overall problem definition is equivalent to Subtask 3A from last year’s task:
a news article, determine whether the main claim made in the article is true, partially true, false,
or other. The four categories were proposed based on Shahi et al. [
False: The main claim made in an article is untrue.
Partially False: The main claim of an article is a mixture of true and false information. It includes articles in categories like partially false, partially true, mostly true, miscaptioned, misleading etc., as defined by diferent fact-checking services.
True: This rating indicates that the primary elements of the main claim are demonstrably true.
Other: An article that cannot be categorised as true, false, or partially false due to lack of evidence about its claims. This category includes articles in dispute and unproven articles. 3http://www.fakenewschallenge.org/ 4https://github.com/FakeNewsChallenge/fnc-1 5https://github.com/Cisco-Talos/fnc-1
For this task, we aimed for two high-quality, real-life fake news datasets that address a wide range of topics in two languages: English and German. Fact-checking claims is not only a time-consuming task, but also requires training and experience. Therefore, to ensure high data quality, we relied on expert evaluations of claims in news articles that were documented on fact-checking services’ websites. In the following section, the process of data collection and the dataset itself will be described in detail. A summary of the approach is depicted in Figure 1.
The general procedure for data crawling was adopted from last year’s iteration of the task [17]:
First, fact-checking sites with an appropriate structure for crawling and focus had to be found.
For the English data, the same fact-checking sites were used as last year. For the German data,
an analysis of available fact-checking sites was conducted, and seven websites were judged as
suitable for crawling regarding the purposes of this task. The crawling process was based on
the AMUSED framework [
If available, the metadata was accessed via the JSON format using the ClaimReview-type defined by Schema.org. However, many websites did not make use of this format and a clear general position of a source link did not exist. Therefore, the first three links of a report were collected, based on the observation that one of those usually referred to the claim source. A manual check of the collected links given in the reports followed to identify the correct source link. For the German data around 1,300 links from roughly 650 reports were manually examined this way. For the English data the same was done for around 1,100 links from roughly 780 reports.
Furthermore, to generate more articles for the category true, we made use of sources the fact-checking experts referred to, in order to validate and proof their judgements. This decision was made, because those references were implicitly judged as reliable. On top of that, these articles covered similar topics as the original claim, thus counteracting a topical bias between classes. The collection of the according URLs was done manually as well.
After the collection and evaluation of links, roughly 1,500 articles (779 German, 711 English) remained for scraping.
From the remaining article candidates and their corresponding links; title and text were extracted in an automatic scraping process. Due to very diverse websites, the creation of tailored scrapers was not feasible. Instead, the h1-tags were extracted as titles and the contents of the p-tags as text, excluding footer contents. For data with missing titles and texts or with a text length below a threshold of 500 characters, the according articles were checked manually again. Often, those articles were not extracted correctly, sitting behind a paywall, or having already removed for the public. If possible, the missing values were added to the data manually or otherwise the article was deleted. Since many fact checking sites relied on archived versions of an article, diferent URLs sometimes lead to the same content. Those duplicates were also removed from the corpus.
In total, we relied on data from 20 diferent websites with AFP being used for both languages). Each fact-checking agency made use of their customized labels (see Table 1 for examples). Thus, instead of providing the original labels, we merged the labels with a similar meaning to one of our four categories: false, partially false, true or other.
The participants of Task 3 were provided an English training set that consisted of last year’s
training set (900 articles) and an English development set that served as test set in the last
iteration (364 articles). The newly collected data was given as test set without labels to the
participants. The test sets consist of the title and text from 612 English and 586 German articles.
An overview about the distribution of the diferent classes in the fake news detection corpus,
CT-FAN-22 [
In this section, we present an overview of all submissions for Task 3 of the CheckThat! lab 2022. Each team could submit up to 200 runs. Yet, only the last submission was taken into account for the evaluation. In total, there were 32 submissions evaluated for detecting English fake news and 6https://deutschlandkurier.de/2021/09/geheime-planspiele-von-gruenen-und-spd-linken-esken-soll-an-stellevon-scholz-kanzlerin-werden/
7https://montrealgazette.com/news/local-news/quebecs-liquor-and-cannabis-stores-will-require-vaccinepassport-as-of-jan-18
Training
Development
Test 16 submissions for the German part of the task. Out of the 32 submissions for English detection, 7 were rejected due to the submission being either incorrectly formatted or incomplete. The same is true for half, that is, 8 of the submissions for the German detection. However, 6 out of those 8 flawed submissions were rejected, because they contained classification results for the English instead of the German test data.
Out of 26 teams that successfully submitted a solution that fully complied with our format specifications for at least one of the two subtasks, most teams only participated in the English monolingual subtask (25 accepted submissions). Yet, 8 teams also successfully submitted runs to the cross-lingual subtask.
Both subtasks are classification tasks. Therefore, we used accuracy and the macro-F 1 score for evaluation and ranked the systems by the latter.
Table 4 provides an overall summary of system performances in terms of macro-F1 score. In both subtasks the best system score is still fairly low. This underlines the general dificulty of both tasks. It comes as no surprise that the best score in the cross-lingual task is below that of the monolingual task, since the latter is a more dificult problem.
Table 4 shows that the baseline classifier, further described in Section 6.1, is very strong for both subtasks, this is particularly true for the monolingual task. In both cases, it is notably above the median. The median in the monolingual task is a bit closer to the best score than in the cross-lingual task. This suggests that the number of stronger systems is greater in the monolingual task.
In both tasks, the range between worst and best score is still not insignificant. Despite the fairly similar system design of the diferent participants within the same subtask as outlined in section 6 (i.e. fine-tuning a transformer), there still seem to be several degrees of freedom which have a major impact on overall performance (e.g. hyperparameter settings or the particular choice of the language model).
In Tables 5 and 6 the latest submission of each team for both subtasks is given.
In total, 25 teams attempted to solve the first task, which was the the monolingual English task.
The best system for the monolingual subtask was submitted by team iCompass [
The second-best system in this subtask was submitted by team NLP&IR@UNED [
A more detailed discussion of the diferent approaches is given in section 6.
In total 8 teams attempted to solve the second subtask, which was the English-German
crosslingual task. Team ur-iw-hnt [
Team NITK-IT_NLP [
A more detailed discussion of the diferent approaches is given in section 6.
Before we give a summary of the diferent classification approaches in section 6.2, we will first describe the baseline classifier that was used in this year’s shared task (section 6.1).
To have a starting point for the participants, we created a baseline system. The model used for the CheckThat! 2022 Task 3 baseline is a standard bert-base-cased model from HuggingFace (no lower-casing during training). The downloaded pre-trained model is originally trained on English data and was fine-tuned on the 900 articles from the CheckThat! training set. The training parameters were: a batch size of 8, a maximum sequence length of 512, 10 epochs and a learning rate of 3e-5.
Since only one article was longer than the maximum sequence length, we did not use passage classification or windows for training. We adopted AdamW as an optimizer with a linear scheduler without warm up. For the training/validation split we used 90% of the training set for training and 10% for validation. The training loss was 0.04 after 10 epochs. The outputs on the validation set (training data) after completion of training were the following: • Accuracy: 0.56 • Precision: 0.44 (macro-averaged) • Recall: 0.44 (macro-averaged) The results on the test set show that there was a problem with overfitting (see Table 5).
The baseline model is trained on the title and text content of the articles. It is based on former work in [51], where also a bert-base-cased was used on another fake news detection dataset. In that work, it was shown that using the titles in front of the body content boosted the accuracy. This boost could also be observed on the CheckThat! Task 3 data. For the German data, we used automatic translation of German texts to English and then applied the baseline model. The results of the baseline system for German can be found in Table 6.
Most experiments involved deep learning models (16 teams), especially applications of BERT (12 teams), RoBERTa (6 teams) or other BERT versions (in total 8 teams) were popular. However, almost as many teams (14 teams) experimented with feature-based supervised-learning approaches as well. Examples are SVMs (10 teams), Logistic Regression (9 teams), Random Forests (8 teams) and Naive Bayes (7 teams). Yet, the majority merely fine-tuned a pre-trained language model and only very few experimented with other approaches.
Although still quite many participants also experimented with feature-based approaches,
only very few teams incorporated a more non-standard feature design, i.e. features other than
bag of words, n-grams or word embeddings. One team (NLP&IR@UNED [
Surprisingly, only two participants (BUM [45] and NLP&IR@UNED [
Only very few teams used additional processing techniques which are not part of standard
text classification algorithms. BUM [45] exploited Wikipedia for evidence retrieval. Team
ur-iw-hnt [
In order to bridge the language gap between the English training data and German test data in the cross-lingual subtask, either a multi-lingual language model (such as XML-RoBERTa or mDeBERTa) was used or the data was automatically translated into the other language by using services such as Google Translate8. Among the participants, there were no approaches that went beyond these well-established procedures.
Since all participants pursued a supervised-learning approach, the choice of training data is also an issue that has been addressed by several participants. About half of them used some training data in addition to the one provided as part of the lab. A popular complement were data from Kaggle.9 10
In this subsection, we provide a description of the individual participant papers to ofer deeper insight into the individual approaches applied to the tasks.
Team AI Rational (monolingual:25 cross-lingual:8) employed a RoBERTa classifier and made use of other English training data in addition to the provided dataset.
Team AIT_FHSTP [50] (monolingual:22 cross-lingual:4) primarily experimented with diferent transformers for this task being T5 and XLM-RoBERTa. For the evaluation, they used XLMRoBERTa. For the cross-lingual subtask the given English training data was translated into German.
Team Awakened [
Team boby232 (monolingual:11) exclusively experimented with feature-based supervised
classiifcation, more specifically, nearest neighbors. The focus was on tuning the parameters of the
classifier. The training data provided by the previous edition of this task was used.
Team BUM [45] (monolingual:10) described an approach to the monolingual fake news detection
task. Numerous additional datasets were added for training. In addition, a Bag-of-Words
approach was used to extract text passages from Wikipedia data that match claims from the
training and test data. A T5 transformer approach checked whether a claim was a logical
consequence of these passages. As a result, the authors found that the approach worked better
for detecting the false class than for the other classes. They attribute this to unbalanced data.
Team CIC [
Team DIU_Carbine (monolingual:14) first augmented the dataset with additional instances of class true from Kaggle so that the dataset is more balanced for training. TF-IDF was used as a method to generate features for supervised learning. Four diferent traditional learning algorithms were tested, Logistic Regression turned out to be the one that worked best. Team DIU_Fire71 (monolingual:24) experimented with several traditional learning algorithms, namely XGBoost, KNN, Gradient Boosting Classifier, Random Forest, Support Vector Machines, Naive Bayes, and Decision Trees. They also used other English training data than the one provided in this year’s task and the dataset from the last iteration of the shared task. For the best classification model, they used TF-IDF vectorization. XGBoost and the Gradient Boosting Classifier algorithm achieved the best results.
Team DIU_Phoenix (monolingual:21) employed a multitude of traditional supervised learning algorithms (i.e. Support Vector Machines, Logistic Regression, Random Forests, Decision Trees, XGB Boosting, Gradient Boosting, Naive Bayes, KNN) and deep learning classfiers (LSTM). They used other English training data than the one provided in this year’s task and the dataset from the last iteration of the shared task as well.
Team DIU_SilentKillers (monolingual:23) experimented with Support Vector Machines, Random Forests, XGBoost, and LSTM. No additional dataset was used.
Team DIU_SpeedOut (monolingual:13) experimented with several traditional learning algorithms, i.e. Naive Bayes, Logistic Regression, and Stochastic Gradient Descent, and deep learning, more specifically, LSTM.
Team FoSIL [48] (monolingual:18) employed an SVM in combination with a feature selection algorithm whose concept is based on human behaviour-based optimization. Team HBDCI [46] (monolingual:12) compared two diferent classification approaches: a featurebased approach using traditional supervised learning and a deep-learning approach that combines BERT, CNN, non-contextual embeddings and stylometric features. Both classifiers were evaluated in diferent configurations (i.e. diferent subsets of features). Overall, the deep-learning approach outperformed the feature-based approach. Including a subset of stylometric features was also helpful.
Team iCompass (monolingual:1) employed two concatenated parallel fine-tuned BERT models. One of the models processed the title of an article and the other the main text. Team MNB (monolingual:16) experimented with Support Vector Machines, Logistic Regression, Random Forests, and Naive Bayes. Tuning the parameters of their classifiers was the main focus of their research.
Team NITK-IT_NLP [
Team NLP&IR@UNED [
Team NLytics [
Team NoFake (cross-lingual:7) also made use of other English non-specified training data
and training data provided by the previous edition of this task. They experimented with two
traditional supervised classifiers (Support Vector Machines, Logistic Regression) and one BERT
deep learning classifier. They focused on exploring the diferent training data in their research.
Team SCUoL [
Team TextMinor [49] (monolingual:19) pursued a deep-learning approach based on RoBERT. Next to exploiting the information contained in that language model, the authors also included overlap features, a singular value decomposition between text and title, and cosine similarity between text and title based on their TF-IDF representation.
Team UNED (monolingual:4 cross-lingual:3) experimented with BERT, RoBERTa, and ALBERT deep learning classifiers. They relied on the following publicly available models: bert-base-cased, bert-base-uncased, albert-base-v2, multilingual-bert-base, roberta-base. Their focus was on the ifne-tuning of those classifiers.
Team ur-iw-hnt [
We have presented a detailed overview of Task 3 of the CheckThat! Lab of CLEF 2022. It focused on the classification of news articles with respect to the correctness of their main claims. The results give a realistic estimate of the current state-of-the art for fake news detection. Most of the participants used transformer-based models like BERT or RoBERTa. Systems based on such technology could be applied within the fact checking community. However, the results show that more work is required in order to improve the current systems. The marco F1 scores are not suficient for a satisfying multi-class classification of news articles according to their factuality. It is a limitation that the provided dataset was unbalanced. Yet, this shared task is one of the few research initiatives that focuses not binary, but multi-class classification. On top of that we ofer a dataset in two languages: German and English. Future research should continue our eforts to provide high-quality multilingual real-world dataset in multiple languages and also broaden the scope by including diferent kinds of meta data (e.g. social factors). Thus, enabling research beyond textual features.
We are thankful for the CheckThat! organizers for supporting this task. We also thank the student volunteers for helping with the annotation of the data.
Part of this work has been funded by the BMBF (German Federal Ministry of Education and Research) under the grant no. 01FP20031J. The responsibility for the contents of this publication lies with the authors. Notes of CLEF 2022—Conference and Labs of the Evaluation Forum, CLEF ’2022, Bologna, Italy, 2022. [10] P. Nakov, G. Da San Martino, F. Alam, S. Shaar, H. Mubarak, N. Babulkov, Overview of the CLEF-2022 CheckThat! lab task 2 on detecting previously fact-checked claims, in: Working Notes of CLEF 2022—Conference and Labs of the Evaluation Forum, CLEF ’2022, Bologna, Italy, 2022. [11] P. Nakov, A. Barrón-Cedeño, G. Da San Martino, F. Alam, J. M. Struß, T. Mandl, R. Míguez, T. Caselli, M. Kutlu, W. Zaghouani, C. Li, S. Shaar, G. K. Shahi, H. Mubarak, A. Nikolov, N. Babulkov, Y. S. Kartal, J. Beltrán, M. Wiegand, M. Siegel, J. Köhler, Overview of the CLEF-2022 CheckThat! lab on fighting the COVID-19 infodemic and fake news detection, in: Proceedings of the 13th International Conference of the CLEF Association: Information Access Evaluation meets Multilinguality, Multimodality, and Visualization, CLEF ’2022, Bologna, Italy, 2022. [12] X. Zhou, R. Zafarani, A survey of fake news: Fundamental theories, detection methods, and opportunities, ACM Comput. Surv. 53 (2020) 109:1–109:40. URL: https://doi.org/10. 1145/3395046. doi:10.1145/3395046. [13] X. Zhang, A. A. Ghorbani, An overview of online fake news: Characterization, detection, and discussion, Inf. Process. Manag. 57 (2020) 102025. URL: https://doi.org/10.1016/j.ipm. 2019.03.004. doi:10.1016/j.ipm.2019.03.004. [14] M. Hardalov, A. Arora, P. Nakov, I. Augenstein, A Survey on Stance Detection for Misand Disinformation Identification, in: Findings of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL ’22 (Findings), Association for Computational Linguistics, Seattle, WA, USA, 2022. URL: https: //arxiv.org/abs/2103.00242. [15] S. I. Manzoor, J. Singla, et al., Fake news detection using machine learning approaches: A systematic review, in: 2019 3rd International Conference on Trends in Electronics and Informatics (ICOEI), IEEE, 2019, pp. 230–234. doi:https://doi.org/10.1109/ICOEI. 2019.8862770. [16] A. Kazemi, K. Garimella, G. K. Shahi, D. Gafney, S. A. Hale, Research note: Tiplines to uncover misinformation on encrypted platforms: A case study of the 2019 indian general election on whatsapp, Harvard Kennedy School Misinformation Review (2022). [17] G. K. Shahi, J. M. Struß, T. Mandl, Overview of the CLEF-2021 checkthat! lab: Task 3 on fake news detection, in: G. Faggioli, N. Ferro, A. Joly, M. Maistro, F. Piroi (Eds.), Proceedings of the Working Notes of CLEF 2021 - Conference and Labs of the Evaluation Forum, Bucharest, Romania, September 21st - to - 24th, 2021, volume 2936 of CEUR Workshop Proceedings, CEUR-WS.org, 2021, pp. 406–423. URL: http://ceur-ws.org/Vol-2936/paper-30.pdf. [18] M. Hardalov, I. Koychev, P. Nakov, In search of credible news, in: C. Dichev, G. Agre (Eds.), Artificial Intelligence: Methodology, Systems, and Applications, Springer International Publishing, Cham, 2016, pp. 172–180. [19] G. K. Shahi, D. Nandini, Fakecovid–a multilingual cross-domain fact check news dataset for covid-19, arXiv preprint arXiv:2006.11343 (2020). [20] D. Röchert, G. K. Shahi, G. Neubaum, B. Ross, S. Stieglitz, The networked context of covid-19 misinformation: Informational homogeneity on youtube at the beginning of the pandemic, Online Social Networks and Media 26 (2021) 100164. [21] K. Pogorelov, D. T. Schroeder, S. Brenner, J. Langguth, Fakenews: Corona virus and 5g conspiracies multimedia analysis task at mediaeval 2021, in: Working Notes Proceedings of the MediaEval 2021 Workshop, 2022. [22] K. Pogorelov, D. T. Schroeder, L. Burchard, J. Moe, S. Brenner, P. Filkukova, J. Langguth, Fakenews: Corona virus and 5g conspiracy task at mediaeval 2020, in: S. Hicks, D. Jha, K. Pogorelov, A. G. S. de Herrera, D. Bogdanov, P. Martin, S. Andreadis, M. Dao, Z. Liu, J. V. Quiros, B. Kille, M. A. Larson (Eds.), Working Notes Proceedings of the MediaEval 2020 Workshop, Online, 14-15 December 2020, volume 2882 of CEUR Workshop Proceedings, CEUR-WS.org, 2020. URL: http://ceur-ws.org/Vol-2882/paper64.pdf. [23] L. Derczynski, K. Bontcheva, M. Liakata, R. Procter, G. Wong Sak Hoi, A. Zubiaga, SemEval2017 task 8: RumourEval: Determining rumour veracity and support for rumours, in: Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), Association for Computational Linguistics, Vancouver, Canada, 2017, pp. 69–76. URL: https://aclanthology.org/S17-2006. doi:10.18653/v1/S17-2006. [24] G. Gorrell, E. Kochkina, M. Liakata, A. Aker, A. Zubiaga, K. Bontcheva, L. Derczynski, SemEval-2019 task 7: RumourEval, determining rumour veracity and support for rumours, in: Proceedings of the 13th International Workshop on Semantic Evaluation, Association for Computational Linguistics, Minneapolis, Minnesota, USA, 2019, pp. 845–854. URL: https://aclanthology.org/S19-2147. doi:10.18653/v1/S19-2147. [25] S. Mohammad, S. Kiritchenko, P. Sobhani, X. Zhu, C. Cherry, SemEval-2016 task 6: Detecting stance in tweets, in: Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016), Association for Computational Linguistics, San Diego, California, 2016, pp. 31–41. URL: https://aclanthology.org/S16-1003. doi:10.18653/ v1/S16-1003. [26] G. Da San Martino, A. Barrón-Cedeño, H. Wachsmuth, R. Petrov, P. Nakov, SemEval2020 task 11: Detection of propaganda techniques in news articles, in: Proceedings of the Fourteenth Workshop on Semantic Evaluation, International Committee for Computational Linguistics, Barcelona (online), 2020, pp. 1377–1414. URL: https://aclanthology.org/2020. semeval-1.186. doi:10.18653/v1/2020.semeval-1.186. [27] T. Mihaylova, G. Karadzhov, P. Atanasova, R. Baly, M. Mohtarami, P. Nakov, SemEval-2019 task 8: Fact checking in community question answering forums, in: Proceedings of the 13th International Workshop on Semantic Evaluation, Association for Computational Linguistics, Minneapolis, Minnesota, USA, 2019, pp. 860–869. URL: https://aclanthology. org/S19-2149. doi:10.18653/v1/S19-2149. [28] J. Thorne, A. Vlachos, O. Cocarascu, C. Christodoulopoulos, A. Mittal, The fact extraction and VERification (FEVER) shared task, in: Proceedings of the First Workshop on Fact Extraction and VERification (FEVER), Association for Computational Linguistics, Brussels, Belgium, 2018, pp. 1–9. URL: https://aclanthology.org/W18-5501. doi:10.18653/v1/W18-5501. [29] J. Thorne, A. Vlachos, C. Christodoulopoulos, A. Mittal, FEVER: a large-scale dataset for fact extraction and VERification, in: Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), Association for Computational Linguistics, New Orleans, Louisiana, 2018, pp. 809–819. URL: https://aclanthology.org/N18-1074. doi:10. at CheckThat! 2022: multi-class and cross-lingual fake news detection, in: Working Notes of CLEF 2022 - Conference and Labs of the Evaluation Forum, CLEF ’2022, Bologna, Italy, 2022. [45] D. La Barbera, K. Roitero, J. Mackenzie, S. Damiano, G. Demartini, S. Mizzaro, BUM at CheckThat! 2022: a composite deep learning approach to fake news detection using evidence retrieval, in: Working Notes of CLEF 2022 - Conference and Labs of the Evaluation Forum, CLEF ’2022, Bologna, Italy, 2022. [46] C. P. Capetillo, D. Lecuona-Gómez, H. Gómez-Adorn, I. Arroyo-Fernández, J. Neri-Chávez, HBDCI at CheckThat! 2022: fake news detection using a combination of stylometric features and deep learning, in: Working Notes of CLEF 2022 - Conference and Labs of the Evaluation Forum, CLEF ’2022, Bologna, Italy, 2022. [47] O. Blanc, A. Pritzkau, U. Schade, M. Geierhos, CODE at CheckThat! 2022: multi-class fake news detection of news articles with BERT, in: Working Notes of CLEF 2022 - Conference and Labs of the Evaluation Forum, CLEF ’2022, Bologna, Italy, 2022. [48] A. Ludwig, J. Felser, J. Xi, D. Labudde, M. Spranger, FoSIL at CheckThat! 2022: using human behaviour-based optimization for text classification, in: Working Notes of CLEF 2022 - Conference and Labs of the Evaluation Forum, CLEF ’2022, Bologna, Italy, 2022. [49] S. Kumar, G. Kumar, S. R. Singh, TextMinor at CheckThat! 2022: fake news article detection using RoBERT, in: Working Notes of CLEF 2022 - Conference and Labs of the Evaluation Forum, CLEF ’2022, Bologna, Italy, 2022. [50] M. Schütz, J. Böck, M. Andresel, A. Kirchknopf, D. Liakhovets, D. Slijepčević, A. Schindler, AIT_FHSTP at CheckThat! 2022: Cross-lingual fake news detection with a large pre-trained transformer, in: Working Notes of CLEF 2022 - Conference and Labs of the Evaluation Forum, CLEF ’2022, Bologna, Italy, 2022. [51] M. Schütz, Detection and identification of fake news: Binary content classification with pre-trained language models, in: Information between Data and Knowledge, volume 74 of Schriften zur Informationswissenschaft, Werner Hülsbusch, Glückstadt, 2021, pp. 422–431.
URL: https://epub.uni-regensburg.de/44959/, gerhard Lustig Award Papers. [52] J. W. Pennebaker, R. L. Boyd, K. Jordan, K. Blackburn, The development and psychometricproperties of LIWC2015, Technical Report, University of Texas, 2015.