Financial and economic news is continuously monitored due to their impact on future stock prices. Thus, the polarity extraction from the news is a very relevant task for investment decision-making by traders. In this sense, Sentiment Analysis models can provide accurate methods to extract signals that influence this decision-making. In this work, we describe the contribution to IberLEF 2023 Challenge - FinancES, where we proposed a hybrid approach that addresses the targeted sentiment analysis by creating a pipeline with diferent phases. First, a phase for cleaning texts, followed by an entity recognition phase and, finally, the polarity extraction. The hybrid approach combines diferent models to proceed with each phase. Thus, RoBERTa transformer architecture is employed as a NER, BETO transformer model is employed for polarity analysis, and, finally, a Spanish Spacy model is used for the part-of-speech tagging process. Although the proposed approach has still scope for improvement, since it reached mid-table positions in the leaderboard, it put forwards a diferent method to carry out the proposed classification Financial markets are changeable and sensitive to global events and phenomena [1]. Factors like political news to users' opinions can produce an immediate efect directly on the market, making it grow positively if the news is good or, conversely, pull downwards if it is bad [2]. Therefore, understanding the emotions embedded in these resources can assist financial professionals and economics in predicting stock market fluctuations [ 3].
(S. Montalvo-Herranz)
information by recognising the emotion, opinion or polarity in human language [
The rest of the paper is organised as follows. Section 2 presents related work, where various approaches that address Opinion Mining in the financial domain are analysed. Section 3 details the distribution of the dataset delivered for each task and describes the systems’ architecture proposed. Section 4 analyses the results achieved in the challenge. Finally, Section 5 summarises the findings harvested facing the challenge, and point out various future research line to explore.
Sentiment Analysis is growing and taking an important role in better understanding users’
opinions in several domains [
In this analysis, we have selected a set of proposals that face the same classification problem thrown in the IberLEF challenge. Various architectures have been analysed, from neural networks to transformers architectures. Given the outcomes achieved by the proposals, we addressed the challenge by using transformers, specifically, language models, since they seem to reach more promising results.
This section describes the datasets proposed by the organizers and the model developed to deal with the tasks delivered in the challenge. 3.1. Data The FinancES dataset is constituted of Spanish news headlines harvested from digital newspapers specialized in the targeted domain, such as Expansión, El Economista, Modaes and El Financiero [13]. The resulting dataset was about 14k headlines, manually labelled by three individuals, selecting the target entity and the sentiment polarity on three dimensions: target, companies, and consumers. During this selection, some headlines were discarded cause of their short length and controversy during the labelling process. As a result, the dataset was reduced to 6k-8k. Table 1 shows the distribution of the datasets released for the practising and evaluation phase. target_sentiment
Total companies_sentiment consumers_sentiment
Total Total
label positive neutral negative positive neutral negative positive neutral negative
Practice
As we can see in Table 1, each label is separated into the three common values assigned in sentiment analysing tasks: positive, neutral and negative, to depict a more detailed picture of the dataset. This organization shows some irregularities in the datasets of practice and evaluation, where the total count of the ‘targets_sentiment’ and ‘companies_sentiment’ does not match the ‘consumers_sentiment’, difering in two and three examples. This inconsistency is because these samples were labelled diferently, with a label not included in these values. Apart from this, it is noteworthy that the clear unbalance of neutral cases in ‘target_sentiment’ for practice and evaluation datasets has a highly unequal distribution of examples.
Another characteristic that draws attention is the similar distribution in ‘companies_sentiment’, where the neutral label difers to others in a large number of samples. 3.2. Method The FinancES challenge of the IberLEF 2023 evaluation campaign consisted of two main complex tasks detecting the main economic target and its linked sentiment and characterizing the polarity at the document level for companies and consumers. Figure 1 shows the architecture that was proposed to address both these two main tasks.
As we can see in Figure 1, the proposed system is configured as a pipeline, where the inputs are the news headlines. First, the cleaner module is responsible for unifying the format of the sentences, removing links, emojis and lowercase words. Next, the target extractor aims at identifying potential financial targets. It receives the pre-processed text and employs a pre-trained model to identify named entities specifically, it operates the transformer-based language model RoBERTa. If the model does not identify entities, we came up with a second extractor method, which was based on the premise that the sentence structure of news headlines is not too complex and follows a simple composition like DET+NOUN+VERB. In this sense, we use grammatical tagging to identify potential targets. Concretely, we employ the spaCy pipeline for NER to mark up each word of each sentence to a particular part of speech. Then, we created a set of 14 simple grammar rules to extract potential targets by combining in diferent ways grammatical tags like DET, NOUN, ADJ, and VERB, among others. To decide how to define these grammar rules, we accomplished a statistical analysis of the headlines given to analyse what were the most used grammar structures by the writers. These rules are employed specifically from the start of the sentence until the word tagged as a VERB is located since we assume the target entity will appear in the first part of the sentence. These are the 14 grammar rules designed are shown in Table 2. When a news headline follows this structure, the words that match the rule are extracted systematically. For instance, if we have the following headline: “Las empresas chinas piden menos burocracia para invertir en España”, the set of words that will be extracted is: “Las empresas chinas”. On the Sentiment Analysis task, the sentiment analyser module tackles each task diferently. The polarity of “target_sentiment” in the sentence is obtained by employing the transformer model BETO, which, for each sentence, it returns two values, the label corresponding to the sentiment that could be NEU or POS and NEG referencing the three existing types of emotional tones, and its score. Thus, to assign this tone, we have defined a threshold, which was established based on several experiments conducted. Therefore, depending on the label and score inferred by the model, and if this score defeats the threshold established, the polarity is assigned in this task, otherwise, it is left empty.
For companies and consumers, the sentiment analyzer module needs first that entities are classified into a person or organization. Two techniques are used in this classification: using the labels inferred by the pre-trained Name Entity Recognition model or two dictionaries of words that group nouns related to persons and organizations domains. When the entities are classified, the sentiment analyzer module analyses the context where the entity is placed and extracts the polarity of text equally to the ‘target_sentiment’ task described above.
In this section, we analyse the performance of the architecture proposed in the evaluation dataset delivered. It is worth stressing that, to facilitate the experiments’ reproducibility, we utilised the default parameters in pre-trained transformed models in the experiments carried out. The standardised metrics selected to assess the performance were Precision, Recall and F1-measure. Table 3 collects the results achieved on each challenge’s classification task.
The complete classification report of the target extraction was not included since it would need a large table to add all the terms extracted during the evaluation. However, the accuracy reached by the system was 0.61 (see in Table 4), indicating that the combination of the RoBERTa model as NER with the designed grammatical rules obtained admissible outcomes. However, despite this reasonable result, the remaining subtasks got weak scores such as 0.37, 0.47 and 0.48 in ‘target_sentiment’, ‘companies_sentiment’ and ‘consumer_sentiment’, respectively. We think this low score in extracting the target is due to there are specific expressions in the news headlines that grammar rules do not cover, making the extractor loses some entities during the evaluation. Concerning the polarity detection, the task where the proposed system accomplished the best outcome was in ‘target_sentiment’, harvesting the highest value on precision and recall, Task
target_sentiment companies_sentiment consumer_sentiment
Label negative neutral positive Macro AVG negative neutral positive Macro AVG negative neutral positive Macro AVG Precision
Recall F1-score on positive and neutral labelling, respectively. The worst results coincided with the task and label, obtaining 0.1 and 0.17 on precision and recall. In spite of the overall results displaying balanced values, which vary between 0.5 and 0.4, if we focus on the positive labelling results, it is easy to recognize that they have a negative impact on the system’s performance since recall outcomes drop below 0.3. However, if we look at the dataset’s distribution, the number of samples for the positives across the tasks does not reflect the outcomes obtained. In some tasks, the number of positive examples is higher, but the system’s performance is quite low. For instance, in the “target_sentiment” task, in the positive labelling the system reached a recall of 0.17, but it has assigned the highest amount of samples in the dataset. Consequently, this situation reflects that the Transformer model selected for addressing the sentiment analysis classification problem does not work precisely. We think this behaviour might be reduced by using augmentation techniques, which sampling generation could teach better the model to diferentiate between the three types of classes, positive, negative and neutral. For a more detailed study of these low results, Figure 2 shows the confusion matrix of each classification task.
Confusion matrices allow an analysis deeply of the results. In this case, it is easy quantifiable the mistakes conducted by the system during the classification process. As we can see, the noticeable mistakes were positive to neutral and neutral to negative classifications. For instance, on the task ‘targets_sentiment’, the system misclassified 495 samples, and 259 on the ‘consumers_sentiment’. From here, it can be inferred that the system had grave problems distinguishing positive and neutral polarities since it hesitates and makes significant mistakes. The mistakes in predicting neutral cases are elevated if we examine and compare the three confusion matrices. We believe that this behaviour is due to the value assigned to the threshold for classifying neutral sentences was not precise enough since it seems that a high percentage of the cases are classified as neutral, but they are not. It is worth mentioning that in positive versus (a) Target Sentiment Task.
(b) Companies Sentiment Task.
(c) Consumers Sentiment Task negative classifications in the pictures’ corner, the number of errors is less inflated, reducing misclassified cases. Thus, despite the problems in classification-neutral samples, we think the system knows how to diferentiate acceptably between positive and negative polarity. Despite the several experiments conducted to find the best system’s performance, we can infer from these results that the pre-established value of the threshold does not work correctly for any case. Thus, we think each polarity tone has to be addressed diferently, assigning an individual limit for each type. These analysed issues have made the proposed system only achieve the 7ℎ place in the leaderboard, depicted in Table 4.
Ranking
In this work we describe the contribution to the FinancES challenge, allocated in the IberLEF 2023 shared evaluation campaign of Natural Language Processing systems. The proposal combines several Deep Learning models to address the main subtasks in this challenge, target extraction and sentiment analysis. Thus, the system combined diferent models: BETO for classifying the polarity, RoBERTa for extracting the target, and a Spacy model for grammar labelling.
As we can see in the evaluation section, the gathered results show that there is still a range of improvement. In the study of the resulting confusion matrices, it can be easily observed that our system can “easily” detect negative labels in any of the proposed tasks but works worse in detecting positive labels in any or neutral sentences on the ‘targets_sentiment’ issue. To improve this situation, we suggest the addition of new grammar rules for detecting more structures in sentences. On the other hand, as a future work line, we would like to try cutting-edge Deep Learning strategies that do not require extensive weight training processes, like prompt engineering.
This work has been partially supported by projects DOTT-HEALTH (PID2019-106942RB-C32, MCI/AEI/FEDER, UE), grant “Programa para la Recualificación del Sistema Universitario Español 2021-2023”, and the project M2297 from call 2022 for impulse projects funded by Rey Juan Carlos University. [11] C. Xiang, J. Zhang, F. Li, H. Fei, D. Ji, A semantic and syntactic enhanced neural model for ifnancial sentiment analysis, Information Processing & Management 59 (2022) 102943. [12] E. Shijia, L. Yang, M. Zhang, Y. Xiang, Aspect-based financial sentiment analysis with deep neural networks., in: WWW (Companion Volume), 2018, pp. 1951–1954. [13] P. Ronghao, J. A. García-Díaz, F. García-Sánchez, R. Valencia-García, Evaluation of transformer models for financial targeted sentiment analysis in spanish, PeerJ Computer Science 9 (2023) e1377. URL: https://doi.org/10.7717/peerj-cs.1377. doi:10.7717/peerj-cs.1377.