This paper presents our participation in the IBERLEF 2023 Task - FinancES in Spanish, focusing on two sub-tasks: Financial targeted sentiment analysis and Financial Sentiment Analysis at the document level for companies and consumers. To address these sub-tasks, we propose a unified generative framework that leverages strong pre-trained language models capable of simultaneously extracting all relevant elements from financial news headlines. Additionally, we introduce two simple auxiliary tasks designed to provide the model with additional information to distinguish sentiment classes for diferent perspectives. The experimental results validate the efectiveness of our approach, as our participation system achieved a Top 3 ranking in both sub-tasks. Specifically, our best model achieved a result of 78.2175% and 61.0373% in terms of F1-score for Task 1 - identifying the target term and its corresponding sentiment and Task 2 classifying the sentiments for the companies and consumers, respectively.
The shared-task IBERLEF 2023 Task [
With the power of pre-trained generative language models such as T5 [
The easy accessibility of financial texts has significantly increased with the widespread use of the Internet and the growing need for market transparency. As a result, the field of financial text analysis has emerged. While the concept of utilizing textual analysis in the financial markets is not entirely novel, the influence of sentiment analysis on financial markets has been firmly established.
Recently, Yıldırım et al. [7] explored diferent deep Learning approaches for Sentiment Analysis on the financial dataset by comparing deep learning classifiers to traditional machine learning approaches. Their findings highlighted the superior performance of LSTM models, including bidirectional LSTM and LSTM with dropout, and revealed similar success rates among various optimizers. Another relevant study by Lee et al. (2020)[8] employed the pre-trained BERT-base model and achieved high accuracy in recognizing investor sentiment after finetuning on a labelled sentiment dataset. Mishev et al.[9] investigated the evolution of sentiment analysis methods from lexicons to transformers, emphasizing NLP transformers’ superiority and ability to capture semantic meaning efectively. In recent years, the implementation of the RNN-LSTM network by Kohsasih et al. [10] demonstrated that this approach can show a significant improvement in sentiment analysis task. A study conducted by Ong et al. [ 11] explored the statistical connection between aspect-based sentiment labels and specific stocks. The research revealed a distinct correlation, particularly in relation to aspects such as inflation and the economy, highlighting their significant influence on stock prices.
Unlike previous approaches that rely on classification-based approaches for sentiment analysis
on financial data, in this paper, we convert two challenges in the shared-task [
Decoder Encoder y t u p t u O
s 5Tm leodm x t u n Acuerdos comerciales, sinónimo ..... República p I
headline → target [e]ptarget [e]pcompanies [e]pconsumers [e]aux1 [e]aux2 (1)
In this paper, we utilize the pre-trained generative mT5 model [12] with encoder-decoder
architecture for the following reasons: (1) First, the mT5 is the encoder-decoder architecture
which can perform well with most of text generation tasks such as machine translation [13]
and aspect-based sentiment analysis [
We only use the oficial training set [ 14], which is provided by the organizers, to train our models for two challenges in the shared-task. Table 1 presents the general statistics of the training and testing set, while Table 2 shows the statistical information towards the polarity classes for three sentiments tasks. As depicted in Table 2, it is evident that there exists an imbalance among the sentiment classes, along with distinct distribution variations between the two sets. These imbalances and distribution variations challenge participating teams when addressing sentiment-related problems within the datasets.
We implemented our framework based on the HuggingFace Transformer library [15]. All models were trained with 20 epochs. We use the learning rate of 1e-3 for the mT5-Small, mT5-based and 1e-4 for mT5-large and mT5-XL. The batch size was set to {32,16} based on the size of pre-trained language models. The beam width was set to 5. We did not utilize any development data to tuning models. The AdamW optimizer was selected to optimize our models. The maximum input and output sequence lengths have been configured as 70 and 48 tokens, respectively. For our experiments, we set a fixed random seed of 42 to train the models. As the Spanish language is not familiar to us, we did not apply any pre-processing methods to the entire dataset except eliminating multiple spaces in the text.
The oficial results and the results of the top systems are shown in Table 3. Our best model achieves the Top 3 ranking in both challenges during the final round. Specifically, for Task 1 - identifying the target term and its corresponding sentiment, our model achieved a result of 78.2175% in terms of F1-score, which is lower than the F1 scores of the Top 1 and Top 2 teams, which are -1.0069% and -0.9997%, respectively. Regarding Task 2 - classifying the sentiment for the companies and consumers, our model also demonstrates the competition by achieving an F1 score of 61.0373%, which is lower than the top 1 team’s model by -3.1976%. For the three sentiment analysis tasks, we achieved the F1-score of 78.9838% (Top 3), 58.8635% (Top 2) and 63.2111 (Top 3) for the target sentiment, companies sentiment and consumers sentiment tasks, respectively.
Table 4 shows the overall results of our submission model and other variants on the test set on two challenges and three sentiment analysis tasks. Overall, it can see that the performance of the model is improved when using language models of large size. As mentioned in the previous work [12], larger models are trained with a larger vocabulary in mT5 variants. Consequently, these larger models have the ability to capture and represent a wider range of linguistic patterns and contextual information for the given text input. However, this also poses a challenge regarding computational resources and model storage for large language models. As seen in Table 4, the models trained on both the original tasks and two auxiliary tasks exhibit better performance compared to models solely fine-tuned on the original tasks. This demonstrated our two auxiliary tasks improved the overall performance.
Figure 2 presents the confusion matrix for three sentiment analysis tasks in our final submission. In the sentiment analysis task for target terms, our model performs well in classifying “positive” and “negative” sentiments but struggles with the “neutral” class. The rate of incorrectly predicting neutral labels as positive versus negative labels is relatively high, with a ratio of approximately 30% and 27%, respectively. This result contrasts with the findings from Task 2, which focused on sentiment analysis for companies and consumers. As shown in Figure 2, our approach achieves the highest accuracy on classifying neutral labels, while its performance on positive and negative labels is comparatively lower. A significant percentage of misclassified positive and negative labels, particularly in the “positive” class, are incorrectly predicted as neutral labels in both tasks. One of the reasons for this result could be the imbalance between polarity classes, as shown in Table 2, within the training set.
In this paper, we describe our submission system for the IBERLEF 2023 Task - FinancES: Financial Targeted Sentiment Analysis, which achieved the Top 3 in both sub-tasks in the shared task. Instead of solving two challenges through classification approaches, we present a generative approach based on the contextual pre-trained language model to solve tasks at the same time. Besides, we design two simple auxiliary tasks to provide more information between two tasks simultaneously. Based on our experimental results and analysis, we believe this approach can be widely applied to other domains for targeted sentiment analysis. Due to the limitations in computational resources, we cannot explore large language models such as mT5-XXL [12] or BLOOM [16] models. However, we believe that fine-tuning these models can improve prediction efectiveness. Additionally, applying efective text pre-processing techniques can also enhance performance on the test set.
Dang Van Thin was funded by the Master, PhD Scholarship Programme of Vingroup Innovation Foundation (VINIF), code VINIF.2022.TS120. Natural Language Processing (Volume 1: Long Papers), Association for Computational Linguistics, Online, 2021, pp. 2416–2429. URL: https://aclanthology.org/2021.acl-long.188. doi:10.18653/v1/2021.acl-long.188. [7] S. Yıldırım, D. Jothimani, C. Kavaklioğlu, A. Başar, Deep learning approaches for sentiment analysis on financial microblog dataset, in: 2019 IEEE International Conference on Big Data (Big Data), IEEE, 2019, pp. 5581–5584. [8] C.-C. Lee, Z. Gao, C.-L. Tsai, Bert-based stock market sentiment analysis, in: 2020 IEEE International Conference on Consumer Electronics - Taiwan (ICCE-Taiwan), 2020, pp. 1–2. doi:10.1109/ICCE-Taiwan49838.2020.9258102. [9] K. Mishev, A. Gjorgjevikj, I. Vodenska, L. T. Chitkushev, D. Trajanov, Evaluation of sentiment analysis in finance: From lexicons to transformers, IEEE Access 8 (2020) 131662– 131682. doi:10.1109/ACCESS.2020.3009626. [10] K. L. Kohsasih, B. H. Hayadi, Robet, C. Juliandy, O. Pribadi, Andi, Sentiment analysis for financial news using rnn-lstm network, in: 2022 4th International Conference on Cybernetics and Intelligent System (ICORIS), 2022, pp. 1–6. doi:10.1109/ICORIS56080. 2022.10031595. [11] K. Ong, W. van der Heever, R. Satapathy, G. Mengaldo, E. Cambria, Finxabsa: Explainable ifnance through aspect-based sentiment analysis, arXiv preprint arXiv:2303.02563 (2023). [12] L. Xue, N. Constant, A. Roberts, M. Kale, R. Al-Rfou, A. Siddhant, A. Barua, C. Rafel, mT5: A massively multilingual pre-trained text-to-text transformer, in: Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Association for Computational Linguistics, Online, 2021, pp. 483–498. URL: https://aclanthology.org/2021.naacl-main.41. doi:10.18653/v1/2021.naacl-main.41. [13] O. Agarwal, M. Kale, H. Ge, S. Shakeri, R. Al-Rfou, Machine translation aided bilingual datato-text generation and semantic parsing, in: Proceedings of the 3rd International Workshop on Natural Language Generation from the Semantic Web (WebNLG+), Association for Computational Linguistics, Dublin, Ireland (Virtual), 2020, pp. 125–130. URL: https:// aclanthology.org/2020.webnlg-1.13. [14] P. Ronghao, J. A. García-Díaz, F. García-Sánchez, R. Valencia-García, Evaluation of transformer models for financial targeted sentiment analysis in spanish, PeerJ Computer Science 9 (2023) e1377. URL: https://doi.org/10.7717/peerj-cs.1377. doi:10.7717/peerj-cs. 1377. [15] T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz, J. Davison, S. Shleifer, P. von Platen, C. Ma, Y. Jernite, J. Plu, C. Xu, T. Le Scao, S. Gugger, M. Drame, Q. Lhoest, A. Rush, Transformers: State-of-the-art natural language processing, in: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, Association for Computational Linguistics, Online, 2020, pp. 38–45. URL: https://aclanthology.org/2020.emnlp-demos.6. doi:10.18653/v1/2020.emnlp-demos.6. [16] T. L. Scao, A. Fan, C. Akiki, E. Pavlick, S. Ilić, D. Hesslow, R. Castagné, A. S. Luccioni, F. Yvon, M. Gallé, et al., Bloom: A 176b-parameter open-access multilingual language model, arXiv preprint arXiv:2211.05100 (2022).