This paper presents a comparative study of diferent approaches for predicting business events from news articles. We evaluate the efectiveness of zero-shot classification models that use Large Language Models (LLM), methods relying on NLI, and a supervised approach using a fine-tuned BERT-based classifier. We also propose a novel ensemble method that combines spaCy, CamemBERT, and FlairNLP models to semantically annotate the news articles in terms of named entities. We discuss the strengths and limitations of each family of approaches that contribute to the development of tools for accurate event prediction from news articles. The public demonstration available at https://jde-predict.tools.eurecom.fr/ enables the user to submit news articles and to visualize the extracted and predicted semantic annotations. In addition, a SPARQL interface is exposed enabling to search through annotations of news articles.
CEUR ceur-ws.org
In this section, we explore three family of methods: Zero-Shot classification using Large supervised classification. We select representative algorithms of these diferent families as they CEUR Workshop Proceedings exhibit diferent advantages and limitations, namely: no need for labeling data for zero-shot classification approaches but with the risk of getting hallucinations for LLM or insuficient background knowledge in the case of NLI; the a priori most accurate classification with a supervised approach but with the limitation of getting labelled data in suficient volume for under-represented classes.
Additionally, we extract named entities and topics using state-of-the-art models. The news articles are written in French. Experts have, beforehand, elicited 11 named business events of interest that are enumerated in Listing 1. This is a multiclass classification problem as one news article can be annotated in terms of multiple business events.
We first exploit the capabilities of pre-trained Large Language Models and we perform prompt engineering to design context-specific prompts that guide the LLMs towards predicting business events accurately. We experiment with four diferent LLMs: bloomz-176b, gpt-3.5-turbo, gpt-4, and claude 1.3. Listing 1: English translation of the prompt used for LLM-based predictions. The original French class names are written in parenthesis
In this prompt, the placeholder ”text” represents the input news article. The LLMs are instructed to analyze the given text and select the relevant category numbers from the provided options. We restrict the models to return up to three category numbers and only if they are explicitly described in the text. By designing this specific prompt, we aim to guide the LLMs to focus on identifying and classifying business events related to the given set of categories. This prompt is the result of several attempts until receiving the best results.
ZeSTE (Zero-Shot Topic Extraction with Common-Sense Knowledge Graph) use ConceptNet’s
common-sense knowledge graph and embeddings to generate predictions without relying
on training data [
We have finally fine-tuned a BERT-based classifier using a supervised approach. To train the classifier, we have created a labeled dataset consisting of 224 news articles paired with their corresponding business event labels. These articles have been selected by experts in order to cover the diferent type of business events we are looking for to extract. We split the dataset into a training set of 179 rows and a test set of 45 rows. During the dataset preparation, we observed a class imbalance issue, where some event classes had a significantly smaller number of instances compared to others. To ensure fair evaluation and prevent the models from being biased towards the majority classes, we performed a stratified split of the dataset.
In addition to predicting business events, we annotate the news articles with mentions of named entities and topics. To accomplish this, we employ multiple Named Entity Recognition (NER) systems, including spaCy, Flair, and a pre-trained CamemBERT model. We ensemble the named entity annotations with various post-processing techniques to refine the results: a majority vote mechanism to determine the final entity annotations, and a harmonization of the entity type from the sentence to the entire document for the same mention.
We annotate news articles in terms of topics using again Large Language Model and casting the problem as a Zero-Shot classification problem where the topics were 8 pre-defined named classes among: ’Merger - Acquisition’, ’CSR (Corporate Social Responsibility)’, ’Human Resources’, ’Employment’, ’International’, ’Portfolio’, ’Investment’, ’Project’.1
The dataset used for evaluation consists of 224 news articles collected from Le Journal Des Entreprises.2 Each article was manually annotated based on a set of 11 distinct business event classes (Section 2.1.1). The articles were also annotated with 8 named topics (Section 2.2). To 1The original topics in French are: ’Fusion - Acquisition’, ’RSE’, ’Ressource Humaine’, ’Emploi’, ’International’, ’Carnet’, ’Investissement’, ’Projet’. 2https://lejournaldesentreprises.com/ evaluate the performance of our models, we employ the Matthews Correlation Coeficient (MCC) as a metric, which assesses the quality of the binary classifications by taking into account true positives, true negatives, false positives, and false negatives as well as with accuracy scores.
Table 1 presents the results of each method using a weighted score which considers the class
distribution of the dataset. Among the evaluated methods, our fine-tuned BERT model displayed
the highest level of performance, achieving a weighted accuracy of 0.9084 and a weighted MCC
of 0.6069. This confirms prior findings where LLMs tend to lag behind fine-tuned transformer
models for doing text classification [
Weighted Accuracy 0.6457 0.6690 0.6734 0.8501 0.7235 0.7636 0.7672 0.9084
Weighted MCC 0.1509 0.2289 0.1754 0.2891 0.3870 0.4419 0.4423 0.6069
We also evaluate the diferent methods on a set of 15 French news articles coming from 3 diferent sources: Les Echos, Le Figaro and Le Monde. We obtained an overall accuracy of 87% for the prediction of the themes and 94.6% for the extraction of the named entities. However, while GPT-4 and Claude enable to predict business events with 86.6% accuracy, the performance of the BERT-based supervised classifier dropped to only 26.6% demonstrating the lack of generalization of the model when applied to news articles coming from another source.3
The demo is available at https://jde-predict.tools.eurecom.fr/. It consists of a backend API, a front-end interface, and it incorporates a caching method for enhanced performance. The backend API serves as the core engine of our system, handling the processing and prediction of business events from news articles. It is made in Python and utilizes the trained models and algorithms discussed in Section 2 to generate predictions based on the URL of an article provided in the request body. The API then returns the predicted business events along with their corresponding probabilities or confidence scores when available.
The front-end interface is created with Next.js and React. It provides a user-friendly web application for interacting with our business event prediction system. Users can input the URL of a news article through a web-based form. The interface displays the text of the news 3The details of the evaluation is available at https://github.com/D2KLab/jde-predict/issues/2 article along its metadata and the predicted events (Figure 1). To optimize performance and reduce computational overhead, we employ a caching method in our system with Redis to store previously processed articles and their corresponding predictions, allowing for faster retrieval of results when encountering similar or identical articles in subsequent requests.
We have developed a tool for predicting business events from news articles by leveraging state-of-the-art machine learning and natural language processing techniques. In the demo, we are visualizing four methods based on GPT 4, Claude 1.3, ZeSTE and BERT supervised. All annotations create a larger knowledge graph that can be used for business applications and queried from https://jde-predict.tools.eurecom.fr/kg/.
This work has been partially funded by Pairing Technology. We thank Le Journal des Entreprises for providing us with access to their news articles.