This study introduces an LLM-based technique for detecting subjectivity and objectivity in English and Arabic news articles. Although several transformers, deep learning (DL), and machine learning (ML)- based techniques were exploited for the task, the LLM (Llama-3-8b) outperformed other models, obtaining the highest F1-scores of 72.6% (Arabic) and 50.36% (English). The suggested LLM-based solution provides a rank of 4th (Arabic) and 12th (English) in the task competition. The research emphasizes the potential of advanced LLMs like Llama-3-8b in achieving high subjectivity and objectivity detection accuracy, which is essential for applications in media analysis, sentiment analysis, and automated content moderation. This study contributes to developing robust multilingual text classification systems, paving the way for more sophisticated and accurate linguistic analysis tools.
In the era of technology, the internet is a constant source of textual information, including news articles, social media posts, blogs, and reviews. These texts ofer a diverse range of information, opinions, and narratives. The ability to distinguish between subjective and objective content, especially in news articles, is crucial. Subjective text often includes personal opinions, emotions, and biases, significantly influencing the reader’s perception. Objective text, on the other hand, presents factual and impartial observations. The automatic classification of text sequences into subjective or objective categories has wide-ranging applications, including media analysis, sentiment analysis, and information retrieval. This capability can significantly enhance the quality of information processing and extraction across various domains, leading to more accurate and reliable results.
How news editorials address political issues may influence individuals with difering ideological
beliefs [
The rise of yellow journalism has made it more crucial than ever to determine an article’s objectivity or
subjectivity. NLP can be crucial in identifying subjectivity and objectivity in news articles. Annotation
rules to identify if an article is subjective or objective were provided by Antici et al. [
Frick et al. [
The dataset used in this work includes two classes (SUBJ and OBJ) and features sentences in English and Arabic. Table 1 illustrates the distribution of train, dev, dev-test, and test sets. We trained all models using the training set and evaluated the model’s performance based on the test set.
CLEF 2024 - CheckThat! Lab [
The ML techniques employed include linear regression (LR), support vector machine (SVM), multinomial naive Bayes (MNB), k-nearest neighbors (KNN), and random forest (RF). The DL techniques involved CNN, CNN+LSTM, and CNN+BiLSTM. Finally, two LLMs are fine-tuned for each language to address the given task. Figure 1 illustrates the schematic process of subjectivity detection.
Textual Feature Extraction: Textual feature extraction is a crucial step in natural language processing, involving converting raw text data into numerical formats. This numerical format helps models interpret and process textual information. A Count Vectorizer is employed in the ML models explored in this study. It is a popular method for textual feature extraction that converts text data into a matrix of token counts. In DL models, tokenization and padding transform raw texts into structured numerical data. These numerical formats are fed through an embedding layer, which captures more sophisticated features such as semantic relationships.
ML Models: This study explores several ML models, including LR, SVM, MNB, KNN, and RF. The hyperparameter configurations for these models are detailed in Table 3.
CNN: This study utilizes a CNN model, starting with an embedding layer with an output dimension of 200. The architecture includes two Conv1D layers, containing 64 and 128 filters, respectively, employing a kernel size of 2 and ReLU activation. For downsampling, a GlobalMaxPooling1D layer is used. Following this, a dense layer with 128 units and ReLU activation is added, along with a dropout layer at a rate of 0.5 to mitigate overfitting. The final output layer consists of a single unit with sigmoid activation. The model is trained using the ‘binary_crossentropy’ loss function and the ‘Nadam’ optimizer, with a batch size of 32 over three epochs.
CNN+LSTM: The CNN-LSTM model implemented in this study shares a similar architecture with the CNN model but includes an LSTM layer with 64 units and a 0.2 dropout rate for sequence modeling. Additionally, the dense layer in this design has 64 units and uses the ReLU activation function. The other hyperparameter settings remain the same as those used in the CNN model.
CNN+BiLSTM: This model has an architecture similar to the CNN+LSTM model but replaces LSTM with BiLSTM.
Transformer-based models: In this study, three transformer-based models were fine-tuned using the English dataset, while another three were fine-tuned using the Arabic dataset. The models employed in English were MdeBERTav3 [22], BERT-base-uncased [23], and RoBERTa [24]. Several text preprocessing measures were implemented to reduce noise in the dataset and concentrate on meaningful words, including lowercasing, emoji removal, stop word removal, stemming, contraction expansion, simple spelling correction using Unicode, and HTML tag elimination. For stopword removal, the NLTK stopwords list was employed.
MdeBERTav3 is a multilingual BERT-based architecture specifically designed for multilingual tasks. This model demonstrates superior linguistic comprehension across various languages and performs excellently in this study. The BERT-base-uncased model, another pre-trained transformer architecture, has previously shown exceptional performance in various natural language processing (NLP) tasks and delivered satisfactory results. Finally, RoBERTa, another optimized version of BERT, is used here for the specified task and obtained comparative results.
In Arabic tasks, AraBERTv2 [25] is additionally used besides MdeBERTav3 and RoBERTa. AraBERTv2 leverages its prior training on an Arabic dataset and has demonstrated its usefulness in various natural language processing tasks, such as sentiment analysis, named entity recognition, and question answering. This model has achieved notable results in the task at hand.
Table 4 shows the learning rate (LR), weight decay (WD), warmup steps (WS), and epochs (EP) used for training the large language models.
Mixtral-7b: In this study, the Mixtral-7b [26] was fine-tuned for subjectivity and objectivity detection in news articles in both English and Arabic. Mixtral-7b was chosen due to its advanced capabilities in handling multilingual data. The ability to efectively comprehend and process English and Arabic texts helped this model perform better in the given task. To achieve the desired results, the model was trained on labeled datasets in both English and Arabic, which enabled it to discern between subjective and objective content.
Llama-3-8b: Llama-3-8b [27], another multilingual large language model, is used in this task. Llama3-8b is a versatile and practical model for performing multilingual NLP tasks. It demonstrated its potential to handle complex subjectivity and objectivity detection in English and Arabic.
Table 5 shows the learning rate (LR), weight decay (WD), warmup steps (WS), max-length, Lora-alpha (LA), gradient accumulation steps (GAS), and epochs (EP) used for training the large language models.
On English test data, among ML models, SVM emerged as the top-performing model with a precision of 56.11%, recall of 56.43%, accuracy of 66.12%, macro-F1 score of 56.23%, and SUBJ-F1 score of 35.43%. Among the DL models, CNN+BiLSTM demonstrated the best performance with a precision of 52.22%, recall of 51.56%, accuracy of 68.18%, macro-F1 score of 51.11%, and SUBJ-F1 score of 22.22%. In the transformer category, RoBERTa outperformed others with a precision of 70.01%, recall of 66.61%, an accuracy of 77.69%, macro-F1 score of 67.82%, and SUBJ-F1 score of 85.64%.
In the Arabic test data, Llama-3-8b demonstrated the best performance with a precision of 76.96%, recall of 70.32%, accuracy of 81.61%, macro-F1 score of 72.46%, and SUBJ-F1 score of 56.59%. The large language models dominated other ML, DL, and transformer-based models in English and Arabic. Llama-3-8b is the best-performing LLM in both languages. 5.1. Error Analysis A comprehensive quantitative and qualitative error analysis is conducted to provide detailed insights into the proposed model’s performance.
Figure 2 illustrates the confusion matrix of Llama-3-8b for English and Arabic. Out of 484 English test cases, the model successfully detects the positive class, with 337 True Positives and 25 False Positives. This reflects high precision, meaning the model is accurate when predicting ‘SUBJ.’ Moreover, it correctly identifies many negative instances with 58 True Negatives. However, 64 False Negatives indicate some positive instances are missed. Overall, the model displays a balanced approach with significant proficiency in minimizing incorrect positive predictions, leading to a high F1 score for positive samples.
In contrast, for 748 Arabic test cases, the model exhibits a diferent pattern in its confusion matrix. It accurately identifies instances from both classes with 301 True Positives and 102 True Negatives. However, many False Positives (221) and False Negatives (124) exist. This indicates that while the model can detect positive instances, it tends to misclassify many negative instances as positive, resulting in lower precision. Additionally, the high number of False Negatives suggests better recall, emphasizing the need for improved distinction between the two classes.
It is evident that the models accurately predicted the labels for samples 1, 2, and 4, but made errors with samples 3 and 5. For the third sample, the sentence’s intent is ambiguous, leading to an incorrect prediction by the model. In the case of the fifth sample, although the sentence is subjective, the model mislabels it due to the large language models being trained on insuficient Arabic data, and also, the provided training dataset is small.
This study evaluated several techniques to detect subjectivity in news articles, including ML, DL, transformer, and LLMs. Transformer-based solutions score better than ML and DL-based models. However, Llama-3-8b performed better than all these models, obtaining the highest F1 scores of 72.46% and 50.36% in Arabic and English, respectively. This study demonstrates how efective LLMs are in identifying subjectivity in articles. Even with limited resources, such as Arabic, LLMs outperform other models regarding results. Future improvements can be made using GPT-4, Llama-3-70b, or other LLMs with significant parameters. lotto, Y. He, A. Lipani, G. McDonald, C. Macdonald, I. Ounis (Eds.), Advances in Information Retrieval, Springer Nature Switzerland, Cham, 2024, pp. 449–458. [16] G. Faggioli, N. Ferro, P. Galuščáková, A. García Seco de Herrera (Eds.), Working Notes of CLEF 2024 - Conference and Labs of the Evaluation Forum, CLEF 2024, Grenoble, France, 2024. [17] M. Hasanain, R. Suwaileh, S. Weering, C. Li, T. Caselli, W. Zaghouani, A. Barrón-Cedeño, P. Nakov, F. Alam, Overview of the CLEF-2024 CheckThat! lab task 1 on check-worthiness estimation of multigenre content, in: [16], 2024. [18] J. M. Struß, F. Ruggeri, A. Barrón-Cedeño, F. Alam, D. Dimitrov, A. Galassi, G. Pachov, I. Koychev, P. Nakov, M. Siegel, M. Wiegand, M. Hasanain, R. Suwaileh, W. Zaghouani, Overview of the CLEF-2024 CheckThat! lab task 2 on subjectivity in news articles, in: [16], 2024. [19] J. Piskorski, N. Stefanovitch, F. Alam, R. Campos, D. Dimitrov, A. Jorge, S. Pollak, N. Ribin, Z. Fijavž, M. Hasanain, N. Guimarães, A. F. Pacheco, E. Sartori, P. Silvano, A. V. Zwitter, I. Koychev, N. Yu, P. Nakov, G. Da San Martino, Overview of the CLEF-2024 CheckThat! lab task 3 on persuasion techniques, in: [16], 2024. [20] F. Haouari, T. Elsayed, R. Suwaileh, Overview of the CLEF-2024 CheckThat! Lab Task 5 on Rumor
Verification using Evidence from Authorities, in: [16], 2024. [21] P. Przybyła, B. Wu, A. Shvets, Y. Mu, K. C. Sheang, X. Song, H. Saggion, Overview of the CLEF2024 CheckThat! lab task 6 on robustness of credibility assessment with adversarial examples (incrediblae), in: [16], 2024. [22] P. He, J. Gao, W. Chen, Debertav3: Improving deberta using electra-style pre-training with gradientdisentangled embedding sharing, 2021. arXiv:2111.09543. [23] J. Devlin, M. Chang, K. Lee, K. Toutanova, BERT: pre-training of deep bidirectional transformers for language understanding, CoRR abs/1810.04805 (2018). URL: http://arxiv.org/abs/1810.04805. arXiv:1810.04805. [24] Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, V. Stoyanov, Roberta: A robustly optimized BERT pretraining approach, CoRR abs/1907.11692 (2019). URL: http://arxiv.org/abs/1907.11692. arXiv:1907.11692. [25] W. Antoun, F. Baly, H. Hajj, Arabert: Transformer-based model for arabic language understanding, arXiv preprint arXiv:2003.00104 (2020). [26] A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. de las Casas, F. Bressand, G. Lengyel, G. Lample, L. Saulnier, L. R. Lavaud, M.-A. Lachaux, P. Stock, T. L. Scao, T. Lavril, T. Wang, T. Lacroix, W. E. Sayed, Mistral 7b, 2023. arXiv:2310.06825. [27] H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, et al., Llama: Open and eficient foundation language models, arXiv preprint arXiv:2302.13971 (2023).