This paper addresses a critical gap in applying semantic enrichment for online news text classification using large language models (LLMs) in fast-paced newsroom environments. While LLMs excel in static text classification tasks, they struggle in real-time scenarios where news topics and narratives evolve rapidly. The dynamic nature of news, with frequent introductions of new concepts and events, challenges pre-trained models, which often fail to adapt quickly to changes. Additionally, the potential of ontology-based semantic enrichment to enhance model adaptability in these contexts has been underexplored.
(J. Gama)
CEUR
ceur-ws.org diferent types of information by applying retraining approaches with and without predefined criteria for selecting the data.
This paper is structured into five additional sections. Section 2 reviews key concepts and related works. Section 3 details the proposed methodology. Section 4 explains the system’s setup and implementation. Section 5 presents the evaluation experiments and provides an analysis of the results. Finally, Section 6 concludes the paper and suggests directions for future research.
The developed code can be found at: https://pypi.org/project/online-news-classification/
The dynamic nature of news necessitates classification models that can adapt to ongoing changes. Models
trained on large datasets and capable of generalization without requiring extensive modifications for
each new analysis provide significant advantages. Large Language Models (LLMs) represent a key
innovation in this regard. As described by Min et al.[
In NLP tasks like text classification, LLMs have demonstrated superior performance compared to
traditional models, such as Naïve Bayes and Decision Trees. Unlike these conventional approaches,
which often analyze words in isolation, LLMs process text holistically, capturing contextual
relationships across sentences and documents. Their robust generalization capabilities enable them to handle
a variety of tasks without requiring extensive task-specific fine-tuning. Techniques like few-shot
classification—where models learn to diferentiate classes with minimal examples—and zero-shot
classiifcation—where models classify text without prior exposure to specific classes—further enhance their
versatility[
Prominent LLMs include GPT[
Hybrid models like BART[
Several studies have explored text classification [
Traditional metrics like Precision, Recall, F-measure, and Accuracy are commonly used in batch classification but are unsuitable for online environments where data and conditions change over time. In such cases, prequential error evaluation is more appropriate as it dynamically tracks error or accuracy for each instance, ofering insight into the model’s evolution. Prequential evaluation ofers two approaches: prequential alpha, which incorporates a forgetting factor (α) to give more weight to recent errors, and prequential window, which uses a sliding window of fixed size (w) to calculate error rates based on the most recent subset of data while discarding older information [13].
A review of the state of the art highlights the advancement introduced in this work: the application of online classification techniques to semantically enriched news texts. Unlike traditional approaches that depend on static datasets and pre-trained models, this approach harnesses real-time adaptability, empowering the classification system to process dynamic and ever-evolving news content efectively.
The system adopts an online processing approach, where each document is handled sequentially as it arrives. Each document undergoes enrichment and classification, and the model subsequently learns from this data, adhering to a prequential (predict-then-learn) framework [13].
The system comprises four key tasks (Figure 1): • Pre-processing - this stage involves cleaning and preparing the text by removing stop words and tokenizing the title and abstract for further analysis • Semantic Enrichment - this is the core task of the system, where the document is enriched semantically using an external knowledge base. It involves identifying semantic entities within the text and mapping these entities to their corresponding representations in the knowledge base • Prediction - the system classifies the document by determining its most likely category based on the current model. • Learning - the model learns from the actual class label assigned to the document, improving its performance over time.
Semantic enrichment was performed using the ReFinED [14] library. This library is an entity linker that allows the identification of the entities present in the text and their relationship with the wikidata/wikipedia entities. Furthermore, this library has the ability to disambiguate the entities it identifies (e.g. ’The jaguar is in the zoo’ returns Q35694 (referring to the animal ’jaguar’) rather than Q30055 (which refers to ’Jaguar,’ the automobile manufacturer)). The entities resulting from this process will be used in the classification of documents. In order to facilitate the interpretation of these entities by the LLM, the titles of the Wikipedia pages were used.
Three experiments were conducted to demonstrate the impact of semantic enrichment and the use of a prequential approach for classifying text documents from news datasets: 1. Utilizing the LLM in its original form, following a zero-shot classification approach 2. Fine-tuning the last layers of the model for each specific set of documents 3. Fine-tuning the final layers of the model using already known data from a specific category, selected based on predefined criteria that must be met for retraining.
All these experiments were carried out on a machine with the following characteristics: Intel Core I7-14700K, GeForce RTX 4090, Windows 11 Pro.
The following sections provide details on the datasets, the semantic enrichment process, and the LLMs employed in these experiments.
Various LLMs were employed to conduct the experiments, selected from the HuggingFace platform based on the following criteria: (i) Task - Zero-shot classification and (ii) Language - english.
and Adversarial-NLI (ANLI)
Table 1 lists the ten most-downloaded models as of October 15, 2024, the date of the last consultation.
In these experiments, seven datasets about news were used. Table 2 presents them, as well as their characteristics, with regard to the number of documents and number of categories.
In the next sections, the results and respective interpretation for each experiment are presented.
In Experiment 1, the LLM was utilized in its original form, without any fine-tuning. Table 3 presents the results for each dataset type (non-enriched (¬E) and enriched (E)), broken down by dataset and model. The highest accuracy values in the comparison between non-enriched and enriched datasets are highlighted in bold.
From Table 3, we observe that approximately half (35 out of 70) of the dataset and model combinations yield higher overall accuracy with the non-enriched dataset. Notably, the AG News and News Category datasets perform better with the non-enriched dataset in the majority of models, with 9 out of 10 and 8 out of 10 combinations, respectively. In contrast, The Guardian dataset shows improved results with the enriched dataset in 7 out of 10 model combinations.
In Experiment 2, the final layers of the LLM, which correspond to the classifier, were updated using a prequential approach. This approach enables the LLM classifier to learn incrementally from previously unseen documents. The documents used for retraining the model were selected based on a strategy that considers five criteria, detailed in Table 4.
Table 5 presents the results for each dataset type (non-enriched (¬E) and enriched (E)), dataset and model. The highest accuracy values in the comparison between non-enriched and enriched datasets are highlighted in bold.
Table 5 reveals that only about 44% of cases (31 out of 70) achieve better results with the enriched dataset. However, compared to accuracy values obtained in previous experiments, there is significant improvement in certain instances, with gains exceeding 30% (e.g., the CNN dataset using the MoritzLaurer/mDeBERTa-v3-base-mnli-xnli model). Conversely, some cases show a decline in overall accuracy in this experiment (e.g., the CNN dataset with the cross-encoder/nli-deberta-v3-base model). Interestingly, both outcomes occur with the same dataset, suggesting that the initial training of the model prior to the described process may play a critical role in these discrepancies.
Similar to Experiment 2, a prequential approach is employed here as well. However, unlike Experiment 2, no selection or constraints were applied; instead, all documents were used for learning and retraining. This comprehensive approach ensures that the model leverages the full dataset for incremental updates, enhancing its exposure to diverse patterns. Table 6 presents the results of this experiment for both non-enriched (¬E) and enriched (E) datasets, categorized by dataset and model.
Analyzing Table 6, we find that only about 20% (14 out of 70) of the experiments conducted—encompassing dataset type, dataset, and model—achieve higher overall accuracy with the non-enriched dataset. When compared to the results from Experiment 1 and Experiment 2, there is a significant improvement of approximately 30%/35% in the number of cases where the enriched dataset delivers better accuracy. This highlights the efectiveness of combining a time-adaptive (online) learning approach with semantic enrichment. For instance, in the case of the MiND dataset and the tasksource/deberta-small-long-nli model, the improvement between the baseline model and the enriched learning model reaches 60%.
Figure 2 complements the findings in Table 6 by presenting the prequential analysis results for the most significant diference observed in this experiment. It highlights the contrast between the non-enriched and enriched datasets, focusing on the CNN dataset processed with the MoritzLaurer/DeBERTa-v3base-mnli-fever-anli model.
(a) Prequential (b) Prequential alpha (c) Prequential window
For the CNN dataset using the MoritzLaurer/DeBERTa-v3-base-mnli-fever-anli model, the enriched dataset achieves higher overall accuracy compared to the non-enriched dataset. In Figure 2a, the blue line for non-enriched data (0.8/0.9) is above the orange line for the enriched dataset (0.7/0.8), reflecting a higher error rate for the non-enriched dataset and thus lower accuracy. Figures 2b and 2c provide prequential analyses under diferent conditions: Figure 2b applies a forgetting factor of 0.99 to emphasize recent errors, while Figure 2c uses a sliding window of 500 documents to examine error patterns. In both cases, the non-enriched dataset shows a consistently higher error than the enriched dataset.
Given the constant emergence of new concepts in the news, classification models must continually adapt. To address this challenge, the paper introduces a novel framework that combines LLMs with online retraining for continuous updates, enabling the model to classify documents with finner granularity and precision over time.
Based on the results obtained, it can be concluded that semantic enrichment significantly improves classification performance. Additionally, the prequential approach enabled the models to adapt to the specific data of each dataset, resulting in a substantial improvement compared to the original model. By integrating semantic enrichment, the system enhances contextual understanding, addressing challenges such as the introduction of new entities, topics, and shifting narratives typical of newsroom environments. This approach represents a significant step forward in bridging the gap between static classification models and the fluid nature of real-world news, ensuring greater accuracy and relevance in classification tasks.
Yet, these findings highlight the need for further investigation into the use of specific criteria in data selection, exploring its potential to enhance overall accuracy under certain conditions.
Moreover, the current semantic enrichment process focuses solely on the entities identified in the texts. Future work could explore the semantic relationships between these entities to further enhance the richness of the documents. At the level of the LLM, it is also recommended to investigate explainability mechanisms, enabling a better understanding of the rationale behind the classifications. [10] N. Nazyrova, S. Chahed, T. Chausalet, M. Dwek, Leveraging large language models for medical text classification: a hospital readmission prediction case, 2024. URL: https://www.scopus.com/inward/record.uri?eid=2-s2.0-85206472619&doi=10.1109% 2fICPRS62101.2024.10677826&partnerID=40&md5=57d814e5d4235e5e5f4c820a3601d903. doi:10.1109/ICPRS62101.2024.10677826, cited by: 0. [11] D. Zhang, R. Mi, P. Zhou, D. Jin, M. Zhang, T. Song, Large model-based data augmentation for imbalanced text classification, in: 2024 5th International Seminar on Artificial Intelligence, Networking and Information Technology, AINIT 2024, 2024, p. 1006 – 1010. URL: https://www.scopus.com/inward/record.uri?eid=2-s2.0-85199212390&doi=10.1109% 2fAINIT61980.2024.10581735&partnerID=40&md5=3bb4e1bf49929b06cb1eec09da989df3. doi:10. 1109/AINIT61980.2024.10581735, cited by: 0. [12] A. Edwards, J. Camacho-Collados, Language models for text classification: Is in-context learning enough?, in: 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation, LREC-COLING 2024 - Main Conference Proceedings, 2024, p. 10058 – 10072. URL: https://www.scopus.com/inward/record.uri?eid=2-s2.0-85195940707&partnerID= 40&md5=7696ac972bf51c856dcd1e86d084f407, cited by: 1. [13] J. Gama, R. Sebastiao, P. P. Rodrigues, On evaluating stream learning algorithms, Machine learning 90 (2013) 317–346. [14] J. F. C. C. A. P. Tom Ayoola, Shubhi Tyagi, ReFinED: An eficient zero-shot-capable approach to end-to-end entity linking, in: NAACL, 2022. [15] M. Laurer, Less annotating, more classifying - addressing the data scarcity issue of supervised machine learning with deep transfer learning and BERT-NLI, 2022. URL: https://osf.io/74b8k. [16] cross encoder, cross-encoder/nli-deberta-v3-base · hugging face, 2021. URL: https://huggingface.
co/cross-encoder/nli-deberta-v3-base. [17] Z. Wang, A. Bukharin, O. Delalleau, D. Egert, G. Shen, J. Zeng, O. Kuchaiev, Y. Dong, Helpsteer2preference: Complementing ratings with preferences, 2024. URL: https://arxiv.org/abs/2410.01257. arXiv:2410.01257. [18] D. Sileo, tasksource: A large collection of NLP tasks with a structured dataset preprocessing framework, in: Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), ELRA and ICCL, Torino, Italia, 2024, pp. 15655–15684. URL: https://aclanthology.org/2024.lrec-main.1361. [19] joeddav, joeddav/xlm-roberta-large-xnli · hugging face, 2023. URL: https://huggingface.co/joeddav/ xlm-roberta-large-xnli. [20] M. Laurer, W. van Atteveldt, A. Casas, K. Welbers, Building Eficient Universal Classifiers with Natural Language Inference, 2023. URL: http://arxiv.org/abs/2312.17543. doi:10.48550/arXiv. 2312.17543, arXiv:2312.17543 [cs]. [21] cross encoder, cross-encoder/nli-MiniLM2-l6-h768 · hugging face, 2021. URL: https://huggingface.
co/cross-encoder/nli-MiniLM2-L6-H768. [22] X. Zhang, J. Zhao, Y. LeCun, Character-level convolutional networks for text classification, 2016.
arXiv:1509.01626. [23] Opensnippets, BBC UK news dataset - dataset by opensnippets, 2021. URL: https://data.world/ opensnippets/bbc-uk-news-dataset. [24] H. Unger, CNN Articles - Data Cleaning & Visualization, 2021. URL: https://kaggle.com/code/ hadasu92/cnn-articles-data-cleaning-visualization. [25] F. Wu, Y. Qiao, J.-H. Chen, C. Wu, T. Qi, J. Lian, D. Liu, X. Xie, J. Gao, W. Wu, M. Zhou, MIND: A Large-scale Dataset for News Recommendation, in: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Association for Computational Linguistics, 2020, pp. 3597–3606. doi:10.18653/v1/2020.acl- main.331. [26] R. Misra, News category dataset, 2022. URL: https://www.kaggle.com/datasets/rmisra/ news-category-dataset. [27] T. G. O. Platform, The guardian api, 2024. URL: https://open-platform.theguardian.com. [28] T. N. Y. Times, The new york times api, 2024. URL: https://developer.nytimes.com.