We provide an overview of task 3 of the CheckThat!lab from the Cross-Language Evaluation Forum (CLEF) 2023, which focuses on predicting the political leaning of English-language news articles and news media outlets. We describe the data collection, the task setup, the evaluation outcomes, and the approaches used by the participating teams. A total of six teams submitted runs for the two subtasks. The top-performing system in Subtask 3A achieved a Mean Absolute Error (MAE) of 0.473, while the best system in Subtask 3B yielded a MAE of 0.549. We make all datasets and evaluation scripts available to the public, aiming to boost further research on this problem.
In the era of widespread digital information, the impact of political bias in news media and
news articles has emerged as a significant concern for democratic societies [
To enhance social awareness and to counteract the spread of false information, the
CheckThat! lab at CLEF [
In this paper, we provide an overview of Task 3, which focuses on detecting the political leaning of news articles and media outlets. We release high-quality, manually annotated data for these two subtask, based on a 3-point ordinal scale for modeling the bias at the article and at the medium level.
The remainder of this paper is organized as follows: In Section 2, we outline the task at hand and describe the dataset we created and released. Section 3 ofers a comprehensive overview of the evaluation settings. We discuss the results and we delve into the details of the submitted systems in Section 4. Section 5 brings to light previous and recent work that aligns with our study. Finally, Section 6 presents our concluding observations and suggests directions for future research.
The goal of the task is to detect the political bias of news reporting at the article and at the media level. This is an ordinal classification task and it is ofered in English. It includes two subtasks, defined below.
Subtask 3A: Political Bias of News Articles Given an article, classify its political leaning as left, center, or right.
Subtask 3B: Political Bias of News Media Given the news article(s) a news outlet (e.g., www.cnn.com), predict the overall political bias of that news outlet as left, center, or right.
2.2.1. Subtask 3A We release a new dataset, which we crawled from AllSides1, a website that gathers news articles from a variety of reputable national and international news sources to ensure a balanced representation across diferent political spectrums. The site ofers meticulous annotations of the bias of news articles, including expert assessments, third-party analysis, independent evaluations, and community input. In addition, AllSides uses annotated articles to support its Balanced Search tool, which displays the news coverage of a specific issue from numerous media providers, each with a diferent political bias from all political perspectives, as depicted in Figure 1.
To ensure that the dataset remains relevant and reflects the current political environment, it includes news articles published from late 2022 till early 2023. Each article has several attributes, as shown in Table 1. In total, we have just over 55k articles in the dataset. We provide statistics about the dataset in Table 2. 2.2.2. Subtask 3B We assess the political bias of English-language media, sourced from Media Bias/Fact Check website.2 On that website, experts conduct an in-depth analysis and annotate the political bias of entire news outlets: examples are shown in Table 3. We further include a certain number of articles, which we crawl from each media source: these are to be used by the participants to analyze that source. The dataset has similar attributes to subtask 3A, plus the source (the name of the medium) as an additional attribute. We have over 8,000 articles (approximately 10 per source) and over 1,000 news sources. Tables 4 and 5 show the label distribution and the number of articles and news media.
Settings The evaluation comprises development and test phases. During the development phase, we provided the participants with the training and the development sets. This enabled them to internally validate their systems and to adjust the parameter values using the development set. During the test phase, the participants submitted their system’s predictions for the provided test set, which did not include reference labels. They were allowed to submit as many runs as they wanted, but only the last submission was considered as the final one. Evaluation This is an ordinal classification task, and thus we used mean absolute error as the oficial measure for both subtasks.
Accenture [24] used machine back-translation to augment the minority classes examples and thus to address the class imbalance. Then, they fine-tuned RoBERTa on this augmented data. TOBB ETU [25] used zero-shot and few-shot classification with ChatGPT exclusively for subtask 3A.
Frank [23] used CatBoost, TF.IDF, oversampling, and an ensemble. They had a file formatting issue, and thus they are not oficially on the leaderboard.
The detection of political bias in news articles and media has been the subject of several studies
[
Historically, bias was primarily understood as coverage inequality, as put forth by Stevenson et al. [31]. However, later definitions expanded to include systematic favoring of particular ideologies or candidates, as illustrated by Waldman and Devitt [32].
These broader interpretations of bias consider factors like visual favorability in news images. Based on a review of numerous studies, [33] proposed three types of media bias: gatekeeping bias, coverage bias, and statement bias. Groeling [34] influenced this classification concept of media bias, focusing on selection bias (what to cover) and presentation bias (how to cover it). Selection bias research typically involves collecting news articles or transcripts, analyzing their content, and identifying systematic biases. Meanwhile, presentation bias is often evaluated through framing, visuals, tone, and sources [35]. Multiple methods have been proposed to quantify news slant, including analyzing the language used by diferent political parties and mapping the distances between media sources based on their mutual followers on social media platforms like Twitter [36, 37, 29].
There have been various approaches for detecting the political bias of news articles. Kulkarni
et al. [38] used an attention-based multi-view model. Baly et al. [
Another related task is hyper-partisan news detection, e.g., [39] proposed a meta-learning approach to model the style similarities between text categories [40]. Systems using averaged word embeddings from pre-trained ELMo models succeeded for this task [41].
Eforts to predict the political ideology of news media used multimodal deep-learningDinkov
et al. [
The CheckThat! lab for CLEF has expanded its task oferings compared to the previous iterations, particularly concentrating on check-worthiness [42], subjectivity [43], bias (this paper), factuality [44], and authority [45]. Notably, only in this sixth edition of the CheckThat! lab a task has aimed at predicting bias at both the article level and the medium level.
Overall, various computational models and datasets have advanced the detection of political bias and factuality and contributed to media literacy eforts [46, 47, 48].
We presented a comprehensive analysis of Task 3 from the CheckThat!lab at CLEF 2023. This lab focused on detecting the political bias in news articles and media outlets. The submissions used transformer-based models (such as RoBERTa and ChatGPT) and gradient boosting on decision trees (like CatBoost), achieving sizable improvements over the baselines.
In future work, we plan to explore more information sources, to add more languages, and to adopt a finer-grained scale.
The work of F. Alam and M. Hasanain is partially supported by NPRP 13S-0206-200281 and NPRP 14C-0916-210015 from the Qatar National Research Fund, a member of Qatar Foundation. The findings herein are solely the responsibility of the authors. (CLEF 2023), 2023. [22] G. Da San Martino, F. Alam, M. Hasanain, R. N. Nandi, D. Azizov, P. Nakov, Overview of the CLEF-2023 CheckThat! lab task 3 on political bias of news articles and news media, in: Working Notes of CLEF 2023–Conference and Labs of the Evaluation Forum, CLEF ’2023, Thessaloniki, Greece, 2023. [23] D. Azizov, S. Liang, P. Nakov, Frank at checkthat! 2023: Detecting the political bias of news articles and news media, in: Working Notes of CLEF 2023–Conference and Labs of the Evaluation Forum, CLEF ’2023, Thessaloniki, Greece, 2023. [24] S. Tran, P. Rodrigues, B. Strauss, E. Williams, Accenture at CheckThat! 2023: Learning to detect Political Bias of News Articles and Sources, in: [49], 2023. [25] M. D. Türkmen, G. Coşgun, M. Kutlu, Tobb etu at checkthat! 2023: Utilizing chatgpt to detect subjective statements and political bias, in: [49], 2023. [26] R. Baly, G. Karadzhov, D. Alexandrov, J. Glass, P. Nakov, Predicting factuality of reporting and bias of news media sources, in: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 2018, pp. 3528–3539. [27] R. Baly, G. Da San Martino, J. Glass, P. Nakov, We can detect your bias: Predicting the political ideology of news articles, in: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2020, pp. 4982–4991. [28] R. Baly, M. Mohtarami, J. Glass, L. Màrquez, A. Moschitti, P. Nakov, Integrating stance detection and fact checking in a unified corpus, in: Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), New Orleans, Louisiana, 2018, pp. 21–27. [29] P. Nakov, H. T. Sencar, J. An, H. Kwak, A survey on predicting the factuality and the bias of news media, arXiv preprint arXiv:2103.12506 (2021). [30] M. Gentzkow, J. M. Shapiro, What drives media slant? Evidence from US daily newspapers,
Econometrica 78 (2010) 35–71. [31] R. L. Stevenson, R. A. Eisinger, B. M. Feinberg, A. B. Kotok, Untwisting the news twisters:
A replication of efron’s study, Journalism Quarterly 50 (1973) 211–219. [32] P. Waldman, J. Devitt, Newspaper photographs and the 1996 presidential election: The question of bias, Journalism & Mass Communication Quarterly 75 (1998) 302–311. [33] D. D’Alessio, M. Allen, Media bias in presidential elections: A meta-analysis, Journal of communication 50 (2000) 133–156. [34] T. Groeling, Media bias by the numbers: Challenges and opportunities in the empirical study of partisan news, Annual Review of Political Science 16 (2013) 129–151. [35] E. C. Tandoc Jr, Journalism is twerking? How web analytics is changing the process of gatekeeping, New media & society 16 (2014) 559–575. [36] K. Darwish, P. Stefanov, M. Aupetit, P. Nakov, Unsupervised user stance detection on Twitter, in: Proceedings of the International AAAI Conference on Web and Social Media, volume 14, 2020, pp. 141–152. [37] P. Stefanov, K. Darwish, A. Atanasov, P. Nakov, Predicting the topical stance and political leaning of media using tweets, in: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 2020, pp. 527–537. [38] V. Kulkarni, J. Ye, S. Skiena, W. Y. Wang, Multi-view models for political ideology detection of news articles, in: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 2018, pp. 3518–3527. [39] J. Kiesel, M. Mestre, R. Shukla, E. Vincent, P. Adineh, D. Corney, B. Stein, M. Potthast, Semeval-2019 task 4: Hyperpartisan news detection, in: Proceedings of the 13th International Workshop on Semantic Evaluation, 2019, pp. 829–839. [40] M. Potthast, M. Hagen, T. Gollub, M. Tippmann, J. Kiesel, P. Rosso, E. Stamatatos, B. Stein, Overview of the 5th international competition on plagiarism detection, in: CLEF Conference on Multilingual and Multimodal Information Access Evaluation, CELCT, 2013, pp. 301–331. [41] Y. Jiang, J. Petrak, X. Song, K. Bontcheva, D. Maynard, Team bertha von suttner at SemEval-2019 task 4: Hyperpartisan news detection using ELMo sentence representation convolutional network, in: Proceedings of the 13th International Workshop on Semantic Evaluation, Minneapolis, Minnesota, USA, 2019, pp. 840–844. [42] F. Alam, A. Barrón-Cedeño, G. S. Cheema, S. Hakimov, M. Hasanain, C. Li, R. Míguez, H. Mubarak, G. K. Shahi, W. Zaghouani, P. Nakov, Overview of the CLEF-2023 CheckThat! lab task 1 on check-worthiness in multimodal and multigenre content, in: Working Notes of CLEF 2023–Conference and Labs of the Evaluation Forum, CLEF ’2023, Thessaloniki, Greece, 2023. [43] A. Galassi, F. Ruggeri, A. B.-C. no, F. Alam, T. Caselli, M. Kutlu, J. M. Struss, F. Antici, M. Hasanain, J. Köhler, K. Korre, F. Leistra, A. Muti, M. Siegel, M. D. Turkmen, M. Wiegand, W. Zaghouani, Overview of the CLEF-2023 CheckThat! lab task 2 on subjectivity in news articles, in: Working Notes of CLEF 2023–Conference and Labs of the Evaluation Forum, CLEF ’2023, Thessaloniki, Greece, 2023. [44] P. Nakov, F. Alam, G. Da San Martino, M. Hasanain, R. N. Nandi, D. Azizov, P. Panayotov, Overview of the CLEF-2023 CheckThat! lab task 4 on factuality of reporting of news media, in: Working Notes of CLEF 2023–Conference and Labs of the Evaluation Forum, CLEF ’2023, Thessaloniki, Greece, 2023. [45] F. Haouari, Z. Sheikh Ali, T. Elsayed, Overview of the CLEF-2023 CheckThat! lab task 5 on authority finding in Twitter, in: Working Notes of CLEF 2023–Conference and Labs of the Evaluation Forum, CLEF ’2023, Thessaloniki, Greece, 2023. [46] P. Nakov, D. Corney, M. Hasanain, F. Alam, T. Elsayed, A. Barrón-Cedeño, P. Papotti, S. Shaar, G. D. S. Martino, Automated fact-checking for assisting human fact-checkers, in: Proceedings of the 30th International Joint Conference on Artificial Intelligence, 2021. [47] F. Alam, S. Shaar, F. Dalvi, H. Sajjad, A. Nikolov, H. Mubarak, G. D. S. Martino, A. Abdelali, N. Durrani, K. Darwish, A. Al-Homaid, W. Zaghouani, T. Caselli, G. Danoe, F. Stolk, B. Bruntink, P. Nakov, Fighting the COVID-19 infodemic: Modeling the perspective of journalists, fact-checkers, social media platforms, policy makers, and the society, in: Findings of EMNLP 2021, 2021, pp. 611–649. [48] P. Nakov, S. Rosenthal, Z. Kozareva, V. Stoyanov, A. Ritter, T. Wilson, SemEval-2013 task 2: Sentiment analysis in Twitter, in: Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013), 2013, pp. 312–320. [49] M. Aliannejadi, G. Faggioli, N. Ferro, Vlachos, Michalis (Eds.), Working Notes of CLEF 2023 - Conference and Labs of the Evaluation Forum, CLEF 2023, Thessaloniki, Greece, 2023.