This work presents the participation of the SINAI team at FinancES@IberLEF2023 shared task, Financial Targeted Sentiment Analysis in Spanish. We have addressed the two proposed tasks, consisting on identifying the main economic target from headlines of financial news for determining their sentiment polarity and identifying the sentiment polarity of each news headline towards both companies and consumers. For target detection, we have explored some popular tools as Stanza and spaCy, and diferent transformers models from Hugging Face and ChatGPT4. For sentiment analysis, we have evaluated some of the most popular transformers models and specific financial transformers. In total, 11 systems have participated (including the baseline provided by the organizers). The best run sent by our team have been placed in position 4th for Task1 and position 2nd for Task 2 with an F1-score of 0.7780 and 0.6349, respectively, being 0.7922 and 0.6423 the best results obtained in the competition for both tasks.
information worldwide, including, of course, financial literacy, more people started to manifest
their interest in economics. Nowadays, it is much easier to find financial information posted
online and, therefore, it is possible to monitor public information, receive early warnings and
perform positive and negative impact analysis. The efects of emotions on financial markets have
been demonstrated in several studies [
Specifically, our team has participated in both of the tasks of this shared competition. For
Task 1, and with the objective of identifying the main economic target from texts, we evaluated
some strategies based on diferent transformers, ChatGPT4 [
Finally, this paper is divided into six diferent sections. Immediately after this introduction, we will talk about everything related with the task description, detailing its purpose and reviewing the dataset. Methodology section will explain what we did for generating our results and in Experimental setup, we discuss about the tools and how they were set up for conducting the experiments. On the other hand, Results and discussion summarizes the results that we obtained in each experiment and reviews them in a comparative way. The last section of this work is Conclusions and future work, and there we discern about our outcomes and, consequently, point to future works that would ofer better results.
This shared task comprises two subtasks. The first one is for target detection and here, participants have to identify the economic target in newspaper headlines. After identifying the main economic target from headlines of financial news, teams have to classify the sentiment polarity (positive, neutral or negative) towards such target in the processed text. For the evaluation, the systems are ranked using the arithmetic mean of the target F1-score and sentiment classification macro-F1. Regarding the second task, participants are expected to conduct a multi-dimensional sentiment classification. As opposed to traditional multi-target tasks, in which multiple targets are identified within the scope of each individual processed text, here each news headline refers to a single target entity, but the stances of other economic agents, companies (com) and consumers (con), are also considered. For Task 2, the systems are ranked using the arithmetic mean of the macro-F1-com and macro-F1-con.
In relation to the dataset [
Total headlines country. The creators of the dataset reviewed all headlines removing the irrelevant ones and manually labelled each headline with the target entity and the sentiment polarity on three dimensions: target, companies, and consumers. Three options are available for sentiment analysis: positive, neutral, o negative. The final dataset comprises of 7,618 news headlines. For the shared task, at a first stage, development-training and development-test sets were made available for participants to develop their systems. Later, training set (including the data of the development set) and test sets were released to participate in the shared task. In Table 1, it is presented the dataset distribution. Finally, it is worth mentioning that the competition was organized through CodaLab and can be accessed at the following link: https: //codalab.lisn.upsaclay.fr/competitions/10052#learn_the_details.
We evaluated several technologies for tasks 1 and 2. For the first part of Task 1, consisting in
identifying the main target of each headline in the dataset, we tested three diferent approaches,
one based on some popular tools, such as Stanza [
In relation to sentiment analysis, we tested diferent transformer-based models with the original Spanish dataset and, for the English models, with an English-translated version of the original Spanish corpus using Google Translator from Python’s deep_translator library [13]. After checking more than ten alternative configurations, based on finance-related and popular transformers models, for the second part of Task 1, related to sentiment analysis on the main target of news headlines, we obtained the best results with mDeBERTa and RoBERTuito and the original, non translated, Spanish dataset. On the other hand, for Task 2, on identifying sentiments for companies and consumers, the best results were obtained using MarIA and the original Spanish dataset. All F1 scores obtained in the development phase can be consulted in Table 3.
Regarding to the software we used to translate the dataset from Spanish to English, it was Google Translator from Python’s deep_translator library [13]. In addition, it is important to note that we did not perform any prior data pre-processing on the dataset to perform the experiments.
With respect to the models, all were downloaded from their public profiles in Hugging Face. During the finetuning process we always used Google Colab for coding under a Pro configuration for being able to use their GPU based hardware options.
Finally, concerning the hyperparameters, Table 4, Table 5 and Table 6 show the configurations that provided the best results for each of the models in the tasks sentiment analysis for main targets, sentiment analysis for companies and sentiment analysis for consumers, respectively. For target detection task, we used default parameters.
This section presents the results obtained in the evaluation phase of the shared task FinancES [
For the first part of Task 1, concerning the identification of the main economic target from ifnancial news headlines, the transformer-based models (Babelscape/wikineural-multilingualner and mrm8488/bert-spanish-cased-finetuned-ner) outperformed ChatGPT4, Stanza and the es_core_news_lg model of spaCy. It is in this task where we appreciate the biggest diferences between our systems and the best results are always for NER specific models that were developed run using some transformer approach.
Continuing with the first task, but now in relation to the second part consisting of determining the sentiment (positive, neutral or negative) towards the main target in the news headlines, we can see that the models using the original Spanish dataset performed better than those using the translated corpus, with MarIA being the best performing model. It should be noted that the English model ROBERTA works slightly better than the Spanish model BETO.
Regarding the second task, on determining the sentiment polarity of each news headline towards both companies and consumers, again the Spanish models overall obtain better results than the English models. However, it is worth noting that the English models ROBERTA and distilrobertafinancial work better than the Spanish models RoBERTuito and BETO. On this occasion, MarIA and mDeBERTa are the transformers that provide the best results for consumers and companies sentiment classification, respectively.
Finally, highlight that the finance-specific transformers have performed worse than the general transformers in the subtasks related to sentiment analysis.
For the competition, we selected run 7, which uses the transformers mrm8488/bert-spanishcased-finetuned-ner and mDeBERTa for target detection and sentiment classification, respectively. With this approach we reached 4th position for Task 1: Financial targeted sentiment analysis and 2nd position for Task 2: Financial Sentiment Analysis at document level for companies and consumers. The oficial leaderboards for both tasks can be consulted in Table 9 and Table 10.
In this paper we have presented the participation of the SINAI team in the shared task FinancES, Financial Targeted Sentiment Analysis in Spanish, at IberLEF 2023. The objective of our experiments, for the target detection task, was to test the performance of the most popular NER tools against transformer-based models. The main conclusion is that transformers-based solutions outperformed others as Stanza or spaCy related approaches. On the other hand, for the sentiment analysis tasks, the aim of our experiments was to test how some of the most popular transformers models behave compared to specific financial transformers models. In abc111 LLI-UAM ABCD Team SINAI AnkitSinghRaikuni UTB-NLP NLP_URJC BASELINE mario.pv UNAM Text Mining fanchuyi this case we conclude that generic transformer models perform better than existing financial transformers models.
In the future, we plan to evaluate why financial transformers perform worse. In addition, we want to continue evaluating external resources to further improve the training phase of the system by analyzing the contribution of each model, testing diferent transfer learning systems as well as models trained on general topics, and using diferent machine translation systems to generate new datasets and/or augment existing ones.
This work has been partially supported by Project CONSENSO (PID2021-122263OB-C21), Project MODERATES (TED2021-130145B-I00) and Project SocialTox (PDC2022-133146-C21) funded by MCIN/AEI/10.13039/501100011033 and by the European Union NextGenerationEU/PRTR, Project PRECOM (SUBV-00016) funded by the Ministry of Consumer Afairs of the Spanish Government, Project FedDAP (PID2020-116118GA-I00) supported by MICINN/AEI/10.13039/501100011033 and WeLee project (1380939, FEDER Andalucía 2014-2020) funded by the Andalusian Regional Government. Salud María Jiménez-Zafra has been partially supported by a grant from Fondo Social Europeo and the Administration of the Junta de Andalucía (DOC_01073). D. Zhu, X. Li, N. Qiang, D. Shen, T. Liu, B. Ge, Summary of ChatGPT/GPT-4 Research and Perspective Towards the Future of Large Language Models, 2023. arXiv:2304.01852. [13] Baccouri, Nidhal, A flexible free and unlimited python tool to translate between diferent languages in a simple way using multiple translators https://deep-translator.readthedocs.io/, 2020.