Extraction of key information such as named entities, key phrases, and numbers is critical for several banking and financial processes. Banks and Financial Institutions resort to the use of automation tools to reduce the human effort required for these processes. Training a system to extract key datapoints reliably and efficiently from text requires large labeled datasets. However, openly available datasets in the financial sector have limited labeled data. In our paper, we address the issues in developing a data extraction system for low resource datasets. We experiment with a Bi-directional long shortterm memory (Bi-LSTM) model which works well on low resource datasets. We introduce a novel domain-specific BiLSTM layer, which allows us to add domain-specific knowledge into the neural architecture. We observed that transfer learning from out-of-domain dataset boosts the accuracy on several extraction tasks. We create three new low resource financial datasets and demonstrate that our model consistently achieves a high degree of accuracy on these datasets. Furthermore, our model outperforms the reported state of the art results on the Financial NER dataset and achieves F1 of 87.48. Our experiments consistently show that transfer learning combined with domain-specific knowledge engineering improves entity recognition in a low resource setting.
Financial Institutions deal with a large number of documents in the form of contracts, reports, application forms etc. These documents are highly unstructured and textual in nature. Processing such documents involve the extraction of key information (entities, contract clauses, key phrases, numbers, etc.). Traditionally, companies have relied on domain experts to capture this information which is timeconsuming. However, recent trends suggest that specialized tools and algorithms are being used to extract key data points from documents to augment and reduce human effort.
Building a system to extract datapoints from unstructured text documents poses several challenges, especially in the financial domain. First, the style of writing varies significantly when compared to news articles, blogs, etc. as “domain specific” lexicons and jargon are used extensively. Secondly, development of any kind of dataset for financial text requires domain experts to label the data. The process of annotation is expensive and cumbersome. Lastly, Financial Institutions are hesitant to share their data as it raises several privacy concerns. Therefore, these constraints curtail the research in the field.
The following sentence is extracted from a financial document
This LOAN AGREEMENT, dated as of November 17, 2014 (this Agreement), is made by and among Auxilium Pharmaceuticals, Inc., a corporation incorporated under the laws of the State of Delaware (U.S. Borrower), Auxilium UK LTD, a private company limited by shares registered in England and Wales (UK Borrower and, collectively with the U.S. Borrower, the Borrowers) and Endo Pharmaceuticals Inc., a corporation incorporated under the laws of the State of Delaware (Lender).1
From this sample, we may want to extract the date
(“November 17, 2014”), type of agreement (“LOAN
AGREEMENT”), names of the borrowers (“Auxilium
Pharmaceuticals, Inc.” and “Auxilium UK LTD”) and the lender
(“Endo Pharmaceuticals Inc.”). In practice, there are few
simple approaches for extracting the data. One of which is
a combination of heuristics and out-of-the-box NER tools.
We can make use of regular expressions to extract the date
and the agreement name. We can use spaCy2 or CoreNLP
Therefore, our motivation is to develop a domain specific datapoint extraction and entity recognition system, even when very little labeled data is available. We treat the problem of extracting the datapoints from unstructured text as a sequence labeling problem and make use of techniques from Named Entity Recognition (NER) and sequence labeling research. Recent efforts in NER research have fo
cused on neural architectures
Studies have shown that transfer learning technique improves the overall performance of the model when there is limited labeled training data. Transfer Learning is a technique where a large dataset (source dataset) is trained with a neural architecture and the learned parameters are used to initialize the weights of the target model.
In our work, we experiment with a Bi-directional Long Short-Term Memory(Bi-LSTM) architecture which works well on low resource datasets. We also develop a novel mechanism to introduce domain-specific knowledge to the neural architecture. Additionally, we show that transfer learning from a pretrained model improves the performance of the models.
Our experiments on 4 financial datasets, including three low-resource datasets - Custodian, Asset Manager, and Leverage Ratio confirm that our architecture works well for low resource conditions.
Key contributions of this paper are Neural Architecture for introducing domain knowledge into the network Study on transfer learning for sequence labeling in a low resource scenario
Our paper is organized as follows. First, we discuss recent works in sequence labeling, low resource deep learning and finance. Second, we describe the datasets and the methodology used for creating the 3 datasets used in our experiments. We then describe the neural architecture used in our experiments. Next, we detail our experiments and results. We perform an ablation study to understand the influence of each of layer in the network with and without transfer learning. Lastly, we conclude the paper with discussion about our work and potential future work.
Traditionally, sequence labeling problems like NER and Part
of Speech Tagging have used Maximum Entropy Models
and hand crafted features
These networks can be trained on a large dataset and then
fine-tuned for a target dataset. Recent efforts in Transfer
Learning have yielded positive results in NLP Tasks
Our work in financial data extraction closely relates to
We use five datasets in our experiments. For training the
out-of-domain model3, we use CoNLL 2003 English dataset
To test our model in the wild, we collected mutual fund prospectus documents which are publicly available on the internet. These documents are fairly large in size (varies from 80 to 300 pages) and have no discernible patterns which can be used by a heuristic system. The documents were collected from the websites of individual fund houses 3This model will be referred as out-of-domain model and pretrained model interchangeably (Ex. BlackRock4) or investment research services (Ex. Morningstar5). From these documents we identify a few key datapoints like Custodian, Asset Manager, Leverage Ratio, etc. which are relevant to organizations dealing with such documents. Our task was to extract the correct entities for each of these datapoints from candidate sentences retrieved from the source document.
In order to create the dataset for Custodian, Asset Manager and Leverage Ratio, we use a proprietary tool to identify parts of the PDF such as table of contents, section headings, keywords, etc. and localize to the approximate region of interest, where the datapoint could be present. Then, the domain experts manually annotate all candidate sentences identifying the correct datapoints.
In Table 1, we describe all the datasets used in our paper.
Our proposed model uses two Bi-LSTM layers - character
and word and a domain specific Bi-LSTM layer. First, we
have the character embedding layer which passes through a
character Bi-LSTM layer. Then, the output of the character
Bi-LSTM layer is concatenated with the word embeddings.
We also concatenate the output of the domain-specific layer
to the word embedding. We use GloVe word embeddings
We observed that the correct named entities are often accompanied by dataset specific keywords. Consider the following example from the Asset Manager dataset
Since January 1, 2002, the Fund is managed by Fideuram Gestions S.A. (the Management Company), a Luxembourg company, controlled by Banca Fideuram S.p.A. (Intesa Sanpaolo Group). 6
From the above sentence, we observe that the correct named entity is ‘Fideuram Gestions S.A.’ and is accompanied by the keyword ‘Management Company’, which is a
known synonym for the Asset Manager. The datapoint Asset Manager has several other keywords such as Investment Advisor, Investment Manager, etc. These keywords are different for Custodian, Leverage Ratio and Financial NER.
In order to introduce this domain knowledge into our neural network, we encode this information as embeddings and pass it to a Bi-LSTM layer. The output of the Bi-LSTM network is concatenated with the word embedding.
Our transfer learning approach is similar to the methods
followed by
Domain Word Character Projection Word + Character Word + Character + Domain Word + Character + Domain + Projection 86.96 89.36 the pretrained model to the target model. We transfer the parameters of the character embeddings and word embeddings. In case we do not perform transfer learning, we randomly initialize the character embeddings and domain-specific embeddings and use GloVe embeddings for the words.
In our study, we experiment by transferring parameters at various layers from an out-of-domain model. The Baseline model is trained only on the in-domain dataset (only Custodian or Asset Manager or Leverage Ratio or Financial NER dataset). We train the model with the same architecture described in 1 without the domain-specific features.
For the pretrained model, we train a Baseline Model
on the CoNLL 2003 English dataset
In our experiments, we transfer the following layers (1) Word Embeddings (Word ) (2) Character Embeddings (Character ) (3) Projection Layer (Projection ). We additionally activate the Domain-Specific Features in our network. (Domain ).
We describe our results on the Custodian, Asset Manager
and Financial NER dataset in Table 2. It can be observed
that the best performing models have transferred
parameters from word and character embeddings and along with
the domain-specific features for the Custodian and Asset
Manager dataset. From Table 2, it is evident that our neural
architecture without transfer learning, outperforms the
reported state of the art results on the Financial NER dataset7.
7
We observe that in all the datasets, the domain-specific
features improve over the baseline F1. However, in the case
of the Financial NER dataset we note that the best
performing system is when word and character embedding layer
is transferred. This observation is consistent with the
findings mentioned in
For our future work, we would like to combine our word
embeddings with ELMo Embeddings
Our work can be extended to clinical texts, where
annotating data is very expensive. Our work closely relates to
MultiTask Learning (MTL). Recent works have shown promise in
Multi-Task Learning for Sequence Labeling Problems in a
low resource scenarios
In conclusion, we demonstrate a Bi-LSTM architecture for low resource datasets. Our experiments consistently show that transfer learning combined with domain-specific knowledge engineering improves entity recognition in a low resource setting.
We would like to thank our anonymous reviewers for their helpful feedback in improving our work. We wish to thank Arjun Rao for internally reviewing the paper. Lastly, we thank the Stride.AI team for their valuable inputs in the research.
Examples In this section, we show a few sample examples from our datasets. Refer to Table 4 5 and 6
The ICAV has appointed RBC Investor Services Bank S.A to act as Depositary for the safekeeping of all the investments, cash and other assets of the ICAV and to ensure that the issue and repurchase of Shares by the ICAV and the calculation of the Net Asset Value and Net Asset Value per Share is carried out and that all income received and investments made are in accordance with the Instrument of Incorporation and the UCITS Regulations.
The custodian is RBC Investor Services Bank S.A which is referred to as Depositary in the sentence. Although ICAV and UCITS are Organizations, they are not the Custodian.
Prior to joining Deutsche Bank, Barbara was a Fund Tax Project Manager at Dexia-BIL, Dexia Fund Services in Luxembourg for two (2) years, and a Senior Fund Manager for DWS Investment S.A. (now the Management Company) in Luxembourg for ten (10) years.
DWS Investment S.A.
DWS Investment S.A. is the management company or the asset manager because of the phrase “now the Management Company”. The reason Deutsche Bank is not the Asset Manager is because the sentence does not mention if it is the Asset Manager.
Under normal market conditions the level of leverage is expected to be between 200% and 800% of the Net Asset Value of the Fund where leverage is calculated using the sum of the absolute value of the notional amounts of the FDI positions in accordance with the “gross method” as set out in the Commission Delegated Regulation.
The example indicates that the expected leverage or the leverage ratio is between 200% and 800%. The system should pick both “200%” and “800%”.