This work presents a novel automatic financial knowledge graph (KG) construction framework by mining massive brokerage research reports without explicit financial expertise guidance and intensive manual rules. We propose a semantic-entity interaction module to construct the interaction feature between the entity and semantic context in the research reports and build a KG from scratch according to a predefined schema based on the obtained interaction features. We train the semantic-entity interaction module using a pre-extracted entity set as a remote supervision-based approach. We further introduce entity augmentation over this entity set from the inference samples of the semantic-entity interaction module to maintain the entity set.
and brokerages. Such reports often cover a wide range of areas and comprehensive data. Therefore, it is reasonable Knowledge graphs (KGs) have emerged as one of the to build a reliable KG based on financial research reports. most popular knowledge representation technologies for However, there are still some challenges in constructmassive information processing tasks. Financial intel- ing KGs in the financial area from research reports, ligence analysis is one of the most important works in among which the most hardest ones are listed below: intelligence analysis, which is facing large volumes of documents and tabular data. KGs have already helped • Entity-relationships are highly coupled to context. ifnancial analysts to process large amounts of data and Entities are not explicitly represented in research cooperate with state-of-the-art trading systems [1, 2] to reports but have a complex interaction with their achieve a high volume return in the market. Such tools text passages. are usually monopolized by large companies and are very • The overall structure of diferent research reports costly to maintain. To democratize such technologies, we are highly complicated. The structures of diferneed a framework that can automatically build a financial ent research reports can contradict each other. As KG from scratch. the research reports accommodate a wide range
In the financial area, research reports contain a wealth of data and knowledge, and much professional of high-quality data collected by professional agencies knowledge, diferent research structures and prothat can be treated as an ideal resource for constructing fessional understandings may express the same a reliable knowledge graph. Financial research reports content slightly diferently. are professional documents with in-depth research on macroeconomics, finance, industries, industry chains, and companies by various financial research institutions
Such features make it dificult to automatically construct a knowledge graph based on research reports from scratch. A solution should involve an in-depth interaction from inter-pipeline interactions to address such a challenge. The high coupling between entities and their context makes the rule-based approach challenging to intervene, and we find that it is more challenging to exploit this part of the features due to the inconsistency of wording in unstructured documents. Therefore, we believe that to deal with such highly coupled features, we need to consider them as a whole. Decoupling entity and contextual information and processing entity features and contextual features to diferent models separately is not ideal.
We use a language model to extract contextual semantic features and bridge the feature connections with a conditional random field [ 3]. Language models like BERT [4] model to extend the entity set. and GPT [5] have proven their performance in many In this work, we develop an automatic knowledge challenging natural language processing tasks [6]. Ap- graph construction pipeline tailored to the financial doplication in Question Answering [7] has proved that lan- main based on research reports. We achieved an 1 score guage models are capable of dealing with complicated 73.5% based on a predefined schema over research resemantic language features. Therefore, BERT is an ideal ports. Our framework is highly scalable, since the overall solution for this semantic feature extraction. Based on the structure is entirely automatic. We designed an entity language model, introducing a downstream specific mod- augmentation to extend the entity set and construct a ule can further improve the semantic features obtained distant supervision over the training process. We also by the language model. In named entity recognition conduct ablation studies to examine the efects of the (NER), there are successful applications combining BERT diferent components of the pipeline. with conditional random fields (CRFs) [ 8, 9]. [10] formulate NER as machine reading comprehension (MRC) task by introducing an MRC module at the end of the BERT 2. Related Works model.
Updating the entity set on the fly can further improve 2.1. Knowledge Graph Construction the reliability of the constructed knowledge graph. The Traditional KG construction is based on a manually specentity set could be easily afected by the noise in the ified ontology and intensive human eforts to learn the raw data. Under such circumstances, we do not want extraction for each relation in the ontology. to put all the eggs into one basket. Filtering raw data is More specifically, supervised methods are learning the first and the most crucial step for building a reliable from sample input and output pairs, like hidden Markov knowledge graph. The most significant budget of con- models (HMMs) [16], maximum entropy-based models, structing a knowledge graph is data cleaning [11]. By such as the MENE system [17] and ME Tagger [18]. Modintroducing a statistical supervision of raw data, such as els based on support vector machines (SVMs) [19] and domain-specific dictionaries and regularization of word CRFs [3] are also common supervised methods. In adfrequencies, human intervention in data cleaning can dition, semi-supervised methods require less training be significantly reduced [ 12]. Therefore, we create an data. For example, a binary AdaBoost classifier [ 20] was automated data cleaning pipeline to preprocess the raw proposed for NER. NELL [21] has introduced a semidata with various filtering methods. Scholars have also supervised bootstrapping approach with a predefined found that using semantic information can also reduce ontology of categories and relations that involve humanhuman efort in data cleaning [ 13, 14, 15]. We thus si- in-the-loop cooperation, fully using human labour, and multaneously use the inference entities of the language existing data. Specifically, Snorkel [ 22] provides a weakly Preprocessing. We follow the standard data cleaning supervised learning model, with handwritten rules, build- in NLP by removing brackets, parentheses, quotes, and ing a generative model based on the overlapping or even other punctuation. Before the pipeline, we filtered the conflicting results of rules. Most recently, unsupervised noisy text spans in sentence-level. We then use the comethods, e.g., KNOWITALL [23], emerged for knowledge reference resolution system (COREF) [27] to the same base construction. entity in the filtered text. We filter out the domainirrelevant entity structure for the output of COREF with 2.2. Named Entity Recognition with a domain-specific predicate dictionary and then tokenize the filtered samples. Sense-disambiguated predicates con
Language Models struct this dictionary from the corpus with the highest By using diferent types of heads, BERT [ 4] can tailor frequency relevant to the financial domain. We extracted for a wide range of natural language processing tasks. entities from the filtered data to obtain entity sets based BERT also has successful applications on named entity on elements covered in the schema. The details of the recognition [24]. [8] proposed to combine CRFs with schema is presented in Figure 2 and discussed in SecBERT on the challenging NER in mining medical docu- tion 4. ments. The same model structure is also applied in NER for Portuguese documents [9]. [25] further introduced an additional BiLSTM in the BERT-CRF structure and further achieved better a performance in Chinese electronic health records NER. Some researchers [26] challenge the BiLSTM in [25], considering it redundant, since BERT and BiLSTM have the same function.
Entity Augmentation. We perform entity augmentation with the inference results of the semantic-entity interaction module, since the extracted entities are collected based on the manually designed schema from analysts’ interest. For scalability concerns, we merge the inference results of the semantic-entity interaction module to augment the entity set.
tracted entities by constructing a distant supervision [28] to the semantic entity interaction module.
Finally, we score the predicate-argument to reflect our confidence in precision and conciseness.
The overall structure of the semantic-entity interaction module is presented in Figure 1. Our proposed semantic entity interaction module is composed of a BERT language model with a CRF [3]. The input sequence is encoded by BERT into an intermediate representation with hidden dimension . A soft attention is then applied to the intermediate representation to learn the interaction better. The output of the soft attention is then fed to the CRF layer. We follow the notation in [29], and have the following scoring function:
(X, y) = ∑︁ ,+1 + ∑︁ , , =0 =1 (1) where A denotes the parameters of the CRF layer, A, represents the score of transitioning from entity to entity , and is the output score of the classification head of the BERT model. We train the semantic-entity interaction module with log-probability loss.
As presented in Figure 1, we perform entity augmentation during the inference phase of the semantic-entity interaction module to extend the entity sets. Practically, we use the pre-trained model, with fixed parameters of the transformer layers and the embedding layer, and only allow the classification head and the CRF to update according to backpropagation.
The original research reports and the annotations are collected by [30], which includes 1, 200 research reports and annotated 5, 131 entities for evaluation. The details of the dataset are shown in Table 1. • Risk indicates the risk warning in the research report. • Article indicates publications cited in the research report. • Industry indicates the industry to which the company belongs. • Brand indicates the brand that the company owns. Some companies may have overlapping brand names, so it is necessary to disambiguate the reference brand and the company name based on the context.
NVIDIA V100 GPU cluster. The batch size is 32 per GPU. We use the BERT-base model as the pre-trained weights of the language model by setting the learning rate as 1− 3 with the Adam optimiser for 10 epochs.
We use HanLP [31] to extract the entities from the ifltered data.
Seeding KG Evaluation KG
to the schema presented in Figure 2. Each element in the schema is explained as follows: improved by 1%. Soft attention can also improve BERT with MRC by 0.68%.
We proposed a novel knowledge graph construction framework based on the brokerage research reports in this work. Our proposed method has achieved 73.5% in 1 score. We expect that our proposed method is also extensible and reliable where we expect the overall performance of our model can be further improved by using a more complicated language model like RoBERTa [33] or GPT-2 [5]. for-ner (2019). [25] Z. Dai, X. Wang, P. Ni, Y. Li, G. Li, X. Bai, Named entity recognition using BERT BiLSTM CRF for Chinese electronic health records, in: 2019 12th International Congress on Image and Signal Processing, Biomedical Engineering and Informatics (CISP-BMEI), 2019, pp. 1–5. [26] Z. Liu, Ner implementation with bert and crf model, 2020. [27] M. Honnibal, I. Montani, spacy 2: Natural language understanding with bloom embeddings, convolutional neural networks and incremental parsing, To appear 7 (2017) 411–420. [28] M. Mintz, S. Bills, R. Snow, D. Jurafsky, Distant supervision for relation extraction without labeled data, in: Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP, 2009, pp. 1003–1011. [29] G. Lample, M. Ballesteros, S. Subramanian, K. Kawakami, C. Dyer, Neural architectures for named entity recognition, arXiv preprint arXiv:1603.01360 (2016). [30] Biendata, Ccks 2020: Evaluation of automated construction of financial knowledge graph based on ontology, 2020. [31] H. He, J. D. Choi, The stem cell hypothesis: Dilemma behind multi-task learning with transformer encoders, in: Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, 2021, pp. 5555–5577. [32] H. Ji, J. Nothman, H. T. Dang, S. I. Hub, Overview of tac-kbp2016 tri-lingual edl and its impact on endto-end cold-start kbp, Proceedings of TAC (2016). [33] Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, V. Stoyanov, Roberta: A robustly optimized bert pretraining approach, arXiv preprint arXiv:1907.11692 (2019).