This paper addresses the challenge of detecting political bias in news articles and media outlets from CheckThat!lab Task 3 [1, 2] by proposing an automated method for classifying these as left, center, or right-leaning. As mass media consumption continues to grow, the capability to identify bias in news reporting is crucial due to the potential societal impact of unaddressed political bias. To tackle this issue, we present a comprehensive approach employing machine learning techniques to detect political leaning in news media and articles. Our model, CatBoost, is evaluated on a diverse dataset comprising over 55,000 news articles sourced from AllSides1 at the article-level. For each model, we aggregate predictions made across news items by a single medium using a majority voting system at medium-level. Our dataset gathered and annotated from over 1,000 popular online platforms as rated by Media Bias/Fact Check2, categorizes political bias into the left, center, or right-wing. We have approximately ten articles from each of these platforms, yielding over 8,000 articles in total. We employ both CatBoost and CatBoost OF3 for media-level classification. These efectively detect political ideology across various media sources, with our CatBoost model demonstrating robustness and efectiveness in handling diverse data. Our findings suggest that utilizing the majority voting technique at the medium level improves model performance. We also highlight the importance of addressing class imbalance and implementing balanced data splits to enhance model performance. Regarding article-level classification using CatBoost, we achieve a Mean Absolute Error (MAE) of 0.270, an F1 score of 0.690, and an accuracy of 0.694. For media-level classification, we achieve a competitive MAE of 0.320, and with the use of the majority voting classifier, our model attains an F1 score of 0.727 and an accuracy of 0.725.
The political leaning of news articles and news media has become an increasingly important
topic in today’s world of information overload. How news is presented and reported can have
a significant impact on people’s perceptions, beliefs, and even voting behaviors [
One of the biggest challenges in detecting the political leaning of news articles and news media
is the lack of a standardized approach [
The CheckThat!lab CLEF 2023 [
Prior research on detecting ideological biases has been emphasized in several studies [
Gentzkow and Shapiro in 2010 [
Recent advances in predicting the political ideology of news media and news articles have
leveraged various aspects such as media stance, factuality, and media profile [
In recent years, Kulkarni et al. [31] used an attention-based multi-view model, leveraging cues from neural inference and natural language processing, to identify news article ideology, achieving higher performance than existing models. However, their method has some potential issues. Horne et al. [32] introduced the News Landscape (NELA) Toolbox. This open-source toolkit allows for investigating news veracity using content-based indicators as a step toward automated news credibility research. Kiesel et al. [33] created a large-scale dataset for hyper-partisan news detection and organized a successful SemEval shared task, with the top team achieving a high accuracy rate. Potthast et al. [34] used a meta-learning approach called unmasking to evaluate style similarity between text categories, revealing significant commonalities and diferences between news types, but found it inadequate for fake news identification. Rashkin et al. [ 35] conducted a study on the language of news media, contrasting the language of legitimate news with satire, hoaxes, and propaganda, and highlighted the potential of stylistic clues in assessing text veracity. Jiang et al. [36] developed a system using averaged word embeddings from a pre-trained ELMo model, achieving first place in a hyper-partisan news detection challenge. Barron et al. developed a model to identify propagandistic material in articles, highlighting the efectiveness of character n-grams and other style criteria over word n-grams while discussing the drawbacks of distant supervision. Martino et al. [37] proposed identifying all fragments containing propaganda techniques in a text and developed a corpus of manually annotated news articles for this purpose, demonstrating the efectiveness of a novel multi-granularity neural network.
Dinkov et al. [39] developed a multimodal deep-learning architecture to predict the political
ideology of news media by studying YouTube channels, which resulted in an extensive
multimodal dataset. Another approach by [
In conclusion, prior and recent research have explored various approaches for predicting
the political ideology of news media, examining the general stance. Studies found that joint
prediction of factuality and political bias proved more advantageous than predicting each
separately [
In the following section, we outline our methodological approach.
TF–IDF. We used the Term Frequency-Inverse Document Frequency (TF-IDF) technique to vectorize our text data [40]. This method calculates the frequency of each term in each article and assigns a weight to each term based on its importance in the article and its frequency in the dataset. This allowed us to feed our textual data into our models and obtain a numerical representation of the articles.
K–means. We used the K-means algorithm to cluster similar articles based on their political bias [41]. This helped us better understand the distribution of political ideologies in our dataset and enabled us to identify any outliers or anomalies in the data.
CatBoost. CatBoost [42] is an open-source machine learning library developed by Yandex, designed explicitly for gradient boosting on decision trees. It ofers a highly eficient and accurate approach to handling categorical features, leveraging a special algorithm to avoid target leakage, and provides numerous options for model interpretation. Its advantages include fast prediction times, robust handling of categorical variables, and a feature that allows it to handle missing data.
Majority voting. We use the majority voting technique as another approach, which is a method employed in ensemble learning [43, 44] by aggregating the predictions made over the news by a single medium at medium level. Figure 1 demonstrates the architecture of our proposed majority voting approach. It determines the final prediction for a given data point by selecting the class or outcome that receives the majority of votes from the ensemble models.
Given a set of classifiers, 1, 2, . . . , , and an input data point , each classifier makes a prediction for the class label: 1, 2, . . . , . The majority voting classifier determines the final output class label by selecting the class with the most votes from the individual classifier predictions.
Mathematically, the majority voting classifier can be represented as: “ modep1, 2, . . . , q (1)
In this Equation 1, the mode function returns the class label that appears most frequently among the individual classifier predictions, where is the final prediction.
For our task, we used hard voting, as we were afraid that the classifiers were not well calibrated, e.g., they can be over-confident in their decisions even when they are wrong; put another way, the classifier might not know when it does not know, and thus their output probability might not be usable directly. Thus, we opted for majority voting. We leave calibration and subsequent soft-voting for future work.
The dataset for this study includes a comprehensive and diverse collection of news articles from diferent news agencies collected from Allsides 1.
Data Attributes For each news article, the following information is available: • ID: A unique identifier for the article. • Title: The headline of the article. • Content: The full text of the article.
• Label: The political leaning of the news article as left, center, or right.
Data Size: The dataset contains a substantial number of articles from each political leaning. In total, we have over 55K articles in a dataset.
In Figure 2, we observe the notable wide range of topic distribution in the dataset. Our dataset has a broad range of topics, including elections, domestic and foreign policy, finance, and others, with “elections” and “other” being the most common topics. Also, Figure 3 shows the class label distribution and the number of articles in each subset. We can see that there are more articles with a right-leaning stance in almost all subsets, with fewer articles in the left-leaning category. Meanwhile, the center class remains consistently in the middle in terms of size.
We use a dataset for assessing the political bias of English-language media, sourced from CheckThat!lab CLEF 2023 which has been crawled from Media Bias/Fact Check2. Table 1 presents examples of news outlets and their corresponding biases.
Data Attributes
This dataset have similar attributes to subtask 3 A, plus the source (name of the medium) as an additional attribute.
Dataset size: We have over 1’000 news media outlets and around 8’000 articles approximately 10 articles per each source.
Figure 4 shows the label distribution for the articles from these media across the three subsets. We can see that the distribution is once again relatively balanced, with similar numbers of instances for each label in each subset. This is important as it ensures that the model trained on the train set will have no biases towards any particular label and will be able to generalize well to the development and the test sets.
Figure 5 shows that over 800 media are used for training set, about 100 for dev, and slightly over 100 for testing. As illustrated in Figure 3, we have a modestly imbalanced data towards the left-leaning news outlets, while the distribution of the other two classes are almost equal.
In this subsection, we provide a detailed description of the experimental setup for our models.
Baseline. The default experimental setup has been used for SVM model with a linear kernel.
CatBoost. For the CatBoost model, we considered the following hyperparameters: learning rate 0.1, depth 6, the maximum number of boosting iterations rate set to 10000, the best model from all iterations is selected as the final model, and the frequency of logging information during a training set to 500, which means that the training progress will be printed every 500 iterations. At medium-level, we modified the number of iterations to 2000, the learning rate to 0.05, and set the progress to log iterations to 100.
CatBoost OF. In the case of the CatBoost model using only the first 300 most important features from the TF-IDF, the experimental setup is slightly modified. The model is trained using only the top 300 most important features derived from the TF-IDF representation of the dataset, focusing on the most relevant information for the classification task. The number of iterations is set to 1000, which means the model trained for 1000 boosting rounds. This parameter determines the maximum number of trees that can be built by the model. The learning rate is set to 0.05, controlling the contribution of each tree to the ensemble model. A lower learning rate generally results in a more robust model, albeit at the expense of longer training times. The frequency of logging the training progress is set to 500 iterations, meaning that the training progress will be logged every 500 iterations, providing less frequent updates compared to the previous setup.
In this subsection, we present the results of our experiments on predicting the political bias of news articles and news outlets. We applied a majority voting ensemble approach to each model by aggregating the predictions made over the news by a single medium. The performance of the individual models and the majority voting ensemble is summarized in Table 2.
Baseline. The task completion’s foundational system was provided by the organizers. A traditional machine learning approach Support Vector Machine (SVM) was selected for this purpose.
CatBoost (Subtask A). This model consistently yielded the best results in our experiments. In terms of MAE model reached 0.270 an F1 score 0.620 and an accuracy 0.621. This can be attributed to its robust gradient-boosting algorithm and optimized implementation for decision tree learning. Moreover, the algorithm incorporates built-in regularization techniques that help prevent overfitting and improve generalization, which is crucial for maintaining high performance on unseen data.
Figure 6 shows a plot illustrating the distribution of article lengths in terms of dataset characters. Our dataset exhibits a long-tailed distribution, with most articles having symbolic lengths peaking at 100–150 and 350–500 characters.
Similarly, Figure 7 shows the distribution of article lengths in terms of words in our dataset, albeit with slightly lower numbers. In our dataset, the majority of the articles have word counts within the ranges of 10–30 and 50–65 words.
The relationship between symbolic count and word count is evident; a higher symbolic length generally corresponds to a higher word count. This observation underscores the interdependence between these two factors in the structure and the content of the articles in our dataset.
Next, we analyzed the feature importance of CatBoost for classifying the political bias in news articles using our dataset. We considered five features: symbolic length, word length, lemmas, content, title, and all text. The results are shown in Figure 8. Our observations indicate that word and symbolic length were not significant features of the model. In our machine learning model, we computed feature importance using the ’SelectPercentile’ method and ℎ2 criterion. We ranked features by ℎ2 scores, indicating their relevance to the target variable.
For our dataset, all text emerged as the most important feature, followed by title, content, and lemmas. Our analysis demonstrates that feature importance varies between them. It is important to note that using all text as a feature when training the model on our data accelerates the learning process.
Next, we computed a confusion matrix, which provides a convenient way to assess our model’s classification performance by presenting the number of correct and incorrect predictions for each class in a tabular format, allowing us to identify patterns and areas where the model may struggle. As shown in Figure 9, the confusion matrix for our model predictions on our dataset reveals distinct diferences in performance across the political bias categories.
In our dataset, the model exhibits a substantial performance in predicting center- and rightleaning articles, whereas it struggles to classify left-leaning ones accurately.
We further investigated the performance of CatBoost a per-class level on our dataset, and the results are shown in Table 3.
Table 3 shows the classification report for CatBoost. The model achieves an overall accuracy of 0.59, with a macro-average F1-score of 0.56 and a weighted F1-score of 0.57. It performs best in classifying center-leaning articles with an F1-score of 0.70, while it has lower F1-scores of 0.49 for both left- and right-leaning articles. Support refers to the number of actual occurrences of the class in the specified dataset. In our case, it means the number of instances for each class (“Center”, “Left”, and “Right”) present in the dataset. For example, there are 2959 instances of “Center”, 2589 instances of “Left”, and 650 instances of “Right”. For “Accuracy”, “Macro Avg”, and “Weighted Avg”, the support is the total number of instances, which is 5198 in this case.
CatBoost (Subtask B). At medium-level CatBoost also consistently achieved superior results throughout our experiments. Moreover, it reached MAE with a score of 0.320 and after applying majority voting we have solid increase in terms of an F1 score and an accuracy which are 0.727 and 0.725, respectively. The confusion matrix in Figure 10 for the test set of the CatBoost model reveals that the model demonstrates strong predictive performance for center-leaning and rightleaning categories. However, its accuracy in predicting left-leaning instances is comparatively lower than that of the other classes.
In Table 4 CatBoost model demonstrates the highest precision for the left class at 0.64, followed closely by the center class at 0.63. The right class exhibits a slightly lower precision of 0.60. In terms of recall, the right class outperforms the others with a value of 0.68. The left and center classes show recall values of 0.60 and 0.58, respectively. The F1-score, which balances precision and recall, indicates that the left class achieves a slightly better score of 0.62 compared to the center and right classes, which have F1-scores of 0.60 and 0.63, respectively. The overall accuracy of the CatBoost model stands at 0.62, while the macro and weighted averages for precision, recall, and F1-score are also equal to 0.62.
CatBoost OF. The CatBoos OF model incorporates the top 300 most important features derived from the Term Frequency-Inverse Document Frequency (TF-IDF) method. Despite a relatively modest Mean Absolute Error (MAE) of 0.375, an F1 score of 0.537, and an accuracy of 0.538, the model exhibited less favorable results compared to other CatBoost models, securing the bottom position. This held true even when a majority voting approach was applied to enhance the performance of our model.
During the initial stage of our analysis, we observed that our dataset exhibited an unbalanced distribution of classes. To investigate the impact of balanced splits on model performance, we created train, test, and development sets with balanced class distributions. Our findings revealed that using balanced splits led to improved results, as shown in Table 5. This suggests that balancing the dataset can contribute to more accurate and reliable predictions. Upon conducting a comparative analysis of the BERT and CatBoost models, we have chosen to implement the CatBoost model in our experiments. This decision was guided by its demonstrably superior performance compared to BERT.
We explored predicting the political bias of news articles and news media using a dataset comprising 55,000 news articles and over 1,000 news media sources with over 8,000 articles. We proposed a new model, CatBoost, which was used to determine the political leaning. Furthermore, we employed the majority voting technique to enhance our model’s performance at the media level.
Our approach efectively classified the political bias, yielding consistent results. The CatBoost model trained on our article-level dataset achieved a classification accuracy of 0.690, an F1 score of 0.694, and a MAE of 0.270. When the model was applied to the medium-level, it reached an MAE of 0.320.
By implementing the majority voting classifier, which aggregates the predictions made over the news by a single medium, we achieved an enhanced F1 score of 0.727 and an accuracy of 0.725. Our comprehensive experiments showed that CatBoost consistently performed efectively. However, the CatBoost OF model delivered the least efective results on the media-level dataset. We noticed that applying majority voting to the news from a single medium improved each model’s performance.
In future work, we aim to explore topic-level bias prediction and move beyond the left-centerright political bias classification. This may require collecting additional labels and breaking away from the current 3-way classification. Furthermore, we intend to carry out cross-language experiments to adapt these methods to languages other than English. Another potential area of investigation is predicting veracity and political leaning simultaneously, necessitating the development of models incorporating both textual and non-textual features, such as source credibility or author reputation. [23] J. Ma, W. Gao, P. Mitra, S. Kwon, B. J. Jansen, K. Wong, M. Cha, Detecting rumors from microblogs with recurrent neural networks, in: S. Kambhampati (Ed.), Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence, IJCAI 2016, New York, NY, USA, 9-15 July 2016, 2016, pp. 3818–3824. [24] A. Zubiaga, M. Liakata, R. Procter, G. W. S. Hoi, P. Tolmie, Analysing how people orient to and spread rumours in social media by looking at conversational threads, PloS one 11 (2016) e0150989. [25] S. Iyengar, K. S. Hahn, Red media, blue media: Evidence of ideological selectivity in media use, Journal of communication 59 (2009) 19–39. [26] S. DellaVigna, E. Kaplan, The fox news efect: Media bias and voting, The Quarterly
Journal of Economics 122 (2007) 1187–1234. [27] A. Simonov, S. K. Sacher, J.-P. H. Dubé, S. Biswas, The persuasive efect of fox news: non-compliance with social distancing during the covid-19 pandemic, Technical Report, National Bureau of Economic Research, 2020. [28] R. Baly, G. Karadzhov, A. Saleh, J. Glass, P. Nakov, Multi-task ordinal regression for jointly predicting the trustworthiness and the leading political ideology of news media, arXiv preprint arXiv:1904.00542 (2019). [29] R. Baly, M. Mohtarami, J. Glass, L. Màrquez, A. Moschitti, P. Nakov, Integrating stance detection and fact checking in a unified corpus, in: Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), New Orleans, Louisiana, 2018, pp. 21–27. [30] K. Popat, S. Mukherjee, J. Strötgen, G. Weikum, Credeye: A credibility lens for analyzing and explaining misinformation, in: Companion Proceedings of the The Web Conference 2018, 2018, pp. 155–158. [31] V. Kulkarni, J. Ye, S. Skiena, W. Y. Wang, Multi-view models for political ideology detection of news articles, arXiv preprint arXiv:1809.03485 (2018). [32] B. D. Horne, W. Dron, S. Khedr, S. Adali, Assessing the news landscape: A multi-module toolkit for evaluating the credibility of news, in: Companion Proceedings of the The Web Conference 2018, 2018, pp. 235–238. [33] J. Kiesel, M. Mestre, R. Shukla, E. Vincent, P. Adineh, D. Corney, B. Stein, M. Potthast, Semeval-2019 task 4: Hyperpartisan news detection, in: Proceedings of the 13th International Workshop on Semantic Evaluation, 2019, pp. 829–839. [34] M. Potthast, M. Hagen, T. Gollub, M. Tippmann, J. Kiesel, P. Rosso, E. Stamatatos, B. Stein, Overview of the 5th international competition on plagiarism detection, in: CLEF Conference on Multilingual and Multimodal Information Access Evaluation, CELCT, 2013, pp. 301–331. [35] H. Rashkin, E. Choi, J. Y. Jang, S. Volkova, Y. Choi, Truth of varying shades: Analyzing language in fake news and political fact-checking, in: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, 2017, pp. 2931–2937. [36] Y. Jiang, J. Petrak, X. Song, K. Bontcheva, D. Maynard, Team bertha von suttner at SemEval-2019 task 4: Hyperpartisan news detection using ELMo sentence representation convolutional network, in: Proceedings of the 13th International Workshop on Semantic Evaluation, Association for Computational Linguistics, Minneapolis, Minnesota, USA, 2019, pp. 840–844. URL: https://aclanthology.org/S19-2146. doi:10.18653/v1/S19-2146. [37] G. D. S. Martino, S. Yu, A. Barrón-Cedeño, R. Petrov, P. Nakov, Fine-grained analysis of propaganda in news articles, arXiv preprint arXiv:1910.02517 (2019). [38] P. Panayotov, U. Shukla, H. T. Sencar, M. Nabeel, P. Nakov, GREENER: Graph neural networks for news media profiling, in: Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics, Abu Dhabi, United Arab Emirates, 2022, pp. 7470–7480. URL: https://aclanthology.org/2022. emnlp-main.506. [39] Y. Dinkov, A. Ali, I. Koychev, P. Nakov, Predicting the leading political ideology of youtube channels using acoustic, textual, and metadata information, arXiv preprint arXiv:1910.08948 (2019). [40] G. Salton, C. Buckley, Term-weighting approaches in automatic text retrieval, Information processing & management 24 (1988) 513–523. [41] M. E. Celebi, H. A. Kingravi, P. A. Vela, A comparative study of eficient initialization methods for the k-means clustering algorithm, Expert systems with applications 40 (2013) 200–210. [42] L. Prokhorenkova, G. Gusev, A. Vorobev, A. V. Dorogush, A. Gulin, Catboost: unbiased boosting with categorical features, Advances in neural information processing systems 31 (2018). [43] T. G. Dietterich, et al., Ensemble learning, The handbook of brain theory and neural networks 2 (2002) 110–125. [44] O. Sagi, L. Rokach, Ensemble learning: A survey, Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery 8 (2018) e1249.