As numerals comprise rich semantic information in financial texts, they play crucial roles in financial data analysis and financial decision making. We propose NFinBERT, a number-aware contextualized language model trained on financial disclosures. Although BERT and other contextualized language models work well for many NLP tasks, they are not specialized in finance and thus do not properly manage numerical information in financial texts. Therefore, we propose pre-training the language model on a large collection of “preprocessed” financial disclosures in which the numbers in reports are explicitly replaced with the knowledge and understanding of the financial and accounting functions of reports. Experimental results on two fine-tuning classification tasks show that language models pre-trained on financial specialized texts generally outperform BERT. Furthermore, the proposed numberaware NFinBERT significantly surpasses other models when the task becomes more difficult or number-sensitive.
BERT
For applications in finance and accounting, in
addition to pre-training domain-specific language
models, recent work has focused on fine-tuning the
pre-trained model for downstream tasks, including
sentiment analysis
To this end, we propose NFinBERT,1 a number1The pre-trained model will be publicly available upon aware contextualized language model pre-trained on a large collection of “pre-processed” financial disclosures, for which we explicitly replace numbers in reports with the knowledge and understanding of the financial and accounting functions of reports. We conduct two downstream tasks to evaluate the proposed model: one is binary classification for risk sentence detection and the other is 12-class classification for sentence-level numeral category prediction. The results indicate that that language models pre-trained on financial specialized texts generally outperform BERT. Furthermore, the proposed number-aware NFinBERT significantly surpasses other models on more difficult or numbersensitive tasks. 2
2.1
To pre-train the domain-specific language
models on financial reports, we used the 10-K reports
from 1996 to 2013 collected by
To train NFinBERT, the number-aware language model, we identify the 11 common categories of numbers in financial reports listed in Table 1, with help from several domain experts in finance and accounting. Note that these tokens are usually not pure integers or floats, and may contain commas or parentheses due to the number formats used in accounting, making the preprocessing more complicated than that for normal numbers. For instance, one million is sometimes presented as “1,000,000” in financial reports, and “(1,000)” represents negative one thousand. For such complex preprocessing, we used both regular expressions and named entity recognition (NER)3 to recognize tokens containing publication.
2https://sraf.nd.edu/textual-analysis/ resources/ 3SpaCy was used for NER. numbers and slot them into one of the 11 classes. For example, $1,000,000 in the reports was masked as the token [MONEY], and 95% was masked as [RATIO]. Therefore, in addition to [CLS] and [SEP], BERT’s original masks, we here add 11 masks to train NFinBERT. The distribution of categories is listed in the last column of Table 1. 2.2
BERT
As in
In this section, we describe experiments on two fine-tuned classification tasks to evaluate the effectiveness of pre-trained language models. The first task (denoted as Task 1 hereafter) considers binary classification for identifying risk sentences in financial reports, and the second (Task 2) is multiclass classification regarding the types of numbers mentioned in sentences extracted from the reports. 3.1 3.1.1
We conducted the experiments on 10K-Sentence, a
sentence-level risk classification dataset
Explanation monetary numbers dates phone numbers bond ratings ordinal information quantities addresses numbers related to ratios percentages time unit smaller than a day other numbers
linguistics,4 resulting in 1,536 risky sentences and 896 non-risky sentences for binary classification. 3.1.2
For the second task, we constructed a new dataset containing 25,261,147 sentences in total extracted from the 10-K reports from 1996 to 2013, each of which is labeled with one of the 11 categories listed in Table 1 plus a “[Nothing]” type. Specifically, the dataset is composed of all sentences in the 10-K reports from 1996 to 2013 containing exactly one number or no number, the former of which was labeled with one of the 11 categories and the latter of which was labeled with the [Nothing] type. For the following experiments, we performed 12-class classification for number category prediction on 1,403,397 randomly selected sentences, constituting 5.6% of the original dataset5 with the same category distribution of the original dataset, due to computational resource limitations.
5As the sentences are from the reports of 1996 to 2013 (i.e., 18 years in total), we here simulate a one-year dataset by randomly sampling 1=18 5:6% of the sentences from the original dataset. 3.2
In both tasks, we split the datasets into training, validation, and test sets at an 8:1:1 ratio, respectively. Moreover, to mitigate the label imbalance problem in Task 2, we down-sampled the training data to the median of the numbers of instances in each category,6 resulting in 177,473 sentences in total.7 The resulting category distribution for model training is illustrated in the first row of Figure 1. Note that only the training set was down-sampled; the validation and test sets retained the original category distribution. We used 15 epochs to finetune all BERT-based models, setting the max sentence length to 128 and the batch size to 32, and used the validation set to search learning rates in f10 5; 5 10 5; 10 4g. The best learning rates for BERT, FinBERT, and NFinBERT were 10 5, 10 4, and 10 4, respectively. Note that the results on both validation and test sets are the averaged results over five repetitions.
6Note that we reduced the training instances only in categories for which the number of instances were higher than the median; we kept the rest categories unchanged.
7Note that in our experiments, as we found that using 2%
of the down-sampled sentences achieves satisfactory
performance, we here used only 3,195 sentences for Task 2 model
training due to computational resource limitations.
For both tasks, we compared three BERT-based
models with three baselines—TF-IDF
bag-ofwords (BOW) with logistic regression,
convolution neural network (CNN)
On Task 2, which is more difficult than Task 1, both FinBERT and NFinBERT surpass BERT8 in terms of macro F1 by a significant amount.9 Figure 1 details the performance of each category in terms of F1 score with a heatmap for all three BERT-based models. From the figure, we observe that BERT is outperformed by FinBERT and NFinBERT for the categories with the fewest training instances ([OTHER], [PHONE], and [RATIO]); this is why BERT achieves better accuracy but a lower macro F1 score in Table 3. Moreover, for these three categories, NFinBERT yields more accurate prediction than FinBERT, suggesting that 8The BERT-Base, Uncased pre-trained model was used in the experiments.
9The improvements compared to BERT are statistically significant at p < 0:01 with a paired t-test. the number-aware pre-trained language model is beneficial for Task 2. 4
We introduce NFinBERT, a number-aware language model trained on financial disclosures, in which we identify 11 categories of numeral tokens with the knowledge and understanding of the financial and accounting functions of reports and replace them with additional masks to pre-train the model. The experimental results show that it is crucial to pre-train BERT on a finance-specific corpus for finance-related downstream tasks; moreover, the proposed NFinBERT outperforms other compared models for 12-class classification for sentence-level numeral category prediction.