This paper introduces the methodology of Team Accenture for the CLEF CheckThat! shared task on identifying political biases in news articles and news sources. We utilize machine back-translation to augment the minority classes in datasets labeling article and news source bias in three categories-Left, Center, and Right, and used this augmented data to fine-tune RoBERTa transformer models. This was the highest ranking strategy in the shared task for detecting both political bias of a news article (at 0.473 Mean Average Precision) as well as for detecting political bias of a news source (at 0.549 mean average precision).
Political bias in news articles can jeopardize an article’s reliability. Biased articles can employ
loaded language, lack proper context, can frame a story selectively, or can include outright
falsehoods [
Consequently, political bias detection has become an important task. [
CheckThat! 2023 Task 3A provides bias-labeled articles that teams must classify as ‘Left’, ‘Center’, or ‘Right’. Task 3B provides bias-labeled news-sites and a collection of articles scraped from each news site. Given the news source and the articles, the goal is to classify URLs of news outlets as ‘Left’, ‘Center’, or ‘Right’. Teams are evaluated using mean absolute error (MAE) [11] where lower MAE is better. The setup of the task is similar to the article-level fake-news domain detection setup used in [12]. However, Subtask 3B uses domain bias labels rather than domain reliability labels.
In this work, we describe that data augmentation and fine-tuning approach employed by Team Accenture for CheckThat! lab subtasks 3A and 3B. Of the four teams that submitted, the Accenture team achieved the best overall MAE for 3A (0.473). Of the two teams that submitted, the Accenture team achieved the best overall MAE for 3B (0.549).
Modeling Group # of Source # of Articles
Unique Words Train
Test Validation
Train
Test Validation As shown in Figure 1, all of the datasets provided by the CheckThat! organizers had label bias which skewed each dataset towards articles that were labeled class 2 (Right).
Transformer models utilize WordPiece tokenization schemes that are dependant on the model being evaluated. At the time of pre-training, the WordPiece algorithm determines which pieces of words will be retained, and which will be discarded. We present our analysis in Table 2. Unexpectedly, the RoBERTa tokenizers we used did not return UNK tokens on any dataset provided by the CLEF CheckThat! organizers.
In this work, we utilize RoBERTa models. The Bidirectional Encoder Representation Transformer (BERT) is a transformer-based architecture that was introduced in 2018 [13, 14]. BERT has had a substantial impact on the field of NLP, and achieved state of the art results on 11 NLP benchmarks at the time of its release. RoBERTa, introduced by [15], modified various parts of BERTs training process. These modifications include more training data, more pre-training steps with bigger batches over more data, removing BERT’s Next Sentence Prediction, training on longer sequences, and dynamically changing the masking pattern applied to the training data [16]. For this work, we fine-tune roberta-large [17]. The English RoBERTa model contains 50,265 WordPieces.
The organizers provided a training and a development set for each language. We use the provided training set and development set to create internal training and validation sets for experimentation. We use the test set provided by organizers as a hold-out test set.
For each article, training data was augmented using back-translation provided by AWS Translate. We appended back-translated left- and center-labeled articles to the training set. In our 2021 experiment [18], we found that this form of augmentation resulted in a significant increase in recall and F1 score for check-worthy tweets. For both article and news source classification, we used Spanish as the pivot language. Due to significant sample imbalance in the training sets for both tasks, we augmented the 0 (Left)- and 1 (Center)-class until the samples are balanced. Specifically, for classification of article bias, we augmented 5,000 samples of the 0-class and 2,000 of the 1 (Center)-class. For classification of news source bias, we augmented 700 samples of the 0 (Left)-class and 300 of the 1 (Left)-class. Table 3 shows the BLEU score for each back-translation scheme. The lower the score, the more divergent the translation to the original text. In a machine translation workflow we would wish to maximize the BLEU score for the best translation. In a data augmentation workflow, we wish to introduce variation to the training data.
For the Article Bias Classification RoBERTa model, we added an additional mean-pooling layer and dropout layer on top of the model prior to the final three binary classification layers, each of which corresponding to a class (i.e., 0 (Left), 1 (Center), or 2 (Right)). The highest class probability determines the article’s final classification. This approach is sometimes referred to as the one-against-all approach for multi-class problem [19]. Adding these additional layers has been shown to help prevent over-fitting while fine-tuning. We used an Adam optimizer with a learning rate of 2 − 5 and an epsilon of 1.5 − 8. We use a binary cross-entropy loss function, 4 epochs, and a batch size of 32.
For the News Source Bias Classification, the same model architecture above is used to finetune a RoBERTa model and classify all article from each news source. We used the majority class label of all articles of a given news source as the class label to establish source bias.
For comparison, we have fine-tuned an additional RoBERTa model for each task above with a single multi-class classifier instead of three binary classifiers. We kept all parameters above the same with a few exceptions: we used the Softmax activation and the sparse categorical cross-entropy loss function instead.
This paper demonstrates models for the classification of political bias in news articles and news sources. In both models, we utilize a pre-trained RoBERTa Large model fine-tuned to the task. We utilized back translation as a strategy to augment the training data, addressing label bias which is found in many machine learning problems. Of the four teams that participated in this shared task, Team Accenture achieved the best overall MAE for 3A (0.473). Of the two teams that submitted, Team Accenture achieved the best overall MAE for 3B (0.549). In addition, for classification of Bias News Article, the one-against-all and the multi-class classifier did not difer significantly in performance. However, for classification of Bias News Sources, the multi-class classifier outperform the one-against-all approach in accuracy but under-perform in MAE. Overall, Team Accenture’s choice of implementing three binary classification layers instead of a single multi-class classifier is the optimal one in both performance and fine-tuning time. media bias in news websites, in: Companion Proceedings of The 2019 World Wide Web Conference, 2019, pp. 742–745. [8] P. Hajare, S. Kamal, S. Krishnan, A. Bagavathi, A machine learning pipeline to examine political bias with congressional speeches, in: 2021 20th IEEE International Conference on Machine Learning and Applications (ICMLA), IEEE, 2021, pp. 239–243. [9] E. Kim, Y. Lelkes, J. McCrain, Measuring dynamic media bias, Proceedings of the National
Academy of Sciences 119 (2022) e2202197119. [10] Y. Dinkov, A. Ali, I. Koychev, P. Nakov, Predicting the leading political ideology of youtube channels using acoustic, textual, and metadata information, arXiv preprint arXiv:1910.08948 (2019). [11] G. Da San Martino, F. Alam, M. Hasanain, R. N. Nandi, D. Azizov, P. Nakov, Overview of the CLEF-2023 CheckThat! lab task 3 on political bias of news articles and news media, in: Working Notes of CLEF 2023–Conference and Labs of the Evaluation Forum, CLEF ’2023, Thessaloniki, Greece, 2023. [12] S. Castelo, T. Almeida, A. Elghafari, A. Santos, K. Pham, E. Nakamura, J. Freire, A topicagnostic approach for identifying fake news pages, in: Companion proceedings of the 2019 World Wide Web conference, 2019, pp. 975–980. [13] J. Devlin, M.-W. Chang, K. Lee, K. Toutanova, BERT: Pre-training of deep bidirectional transformers for language understanding, arXiv preprint arXiv:1810.04805 (2018). [14] I. Turc, M.-W. Chang, K. Lee, K. Toutanova, Well-read students learn better: On the importance of pre-training compact models, 2019. arXiv:1908.08962. [15] Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, V. Stoyanov, RoBERTa: A robustly optimized BERT pretraining approach, CoRR abs/1907.11692 (2019). URL: http://arxiv.org/abs/1907.11692. arXiv:1907.11692. [16] E. M. Williams, P. Rodrigues, V. Novak, Accenture at CheckThat! 2020: If you say so: Posthoc fact-checking of claims using transformer-based models, in: L. Cappellato, C. Eickhof, N. Ferro, A. Névéol (Eds.), Working Notes of CLEF 2020 - Conference and Labs of the Evaluation Forum, Thessaloniki, Greece, September 22-25, 2020, volume 2696 of CEUR Workshop Proceedings, CEUR-WS.org, 2020. URL: http://ceur-ws.org/Vol-2696/paper_226. pdf. [17] Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, V. Stoyanov, RoBERTa: A robustly optimized BERT pretraining approach, CoRR abs/1907.11692 (2019). URL: http://arxiv.org/abs/1907.11692. arXiv:1907.11692. [18] E. Williams, P. Rodrigues, S. Tran, Accenture at CheckThat! 2021: Interesting claim identification and ranking with contextually sensitive lexical training data augmentation, 2021. arXiv:2107.05684. [19] Y. Liu, Y. F. Zheng, One-against-all multi-class svm classification using reliability measures, in: Proceedings. 2005 IEEE International Joint Conference on Neural Networks, 2005., volume 2, IEEE, 2005, pp. 849–854.