The securities market carries complex financial interactions, providing challenges to its prediction. To represent this complexity, researchers have utilized multi-source data, such as financial news and macro market indicators, for better performance. However, these efforts often ignore the internal knowledge among these data or suffer from the high cost of acquiring diverse knowledge. Thus, we propose a LLM-driven knowledge enhancement method for securities index prediction. Specifically, we collect the daily data of Shanghai Stock Exchange indexes and their related market indicators and model the internal knowledge among them as triplets. Then we leverage LLM as a knowledge base to acquire diverse knowledge efficiently. Finally, we integrate the knowledge and numeric multi-source data as a heterogeneous graph and apply a GNN model to predict the trend of securities indexes. Experiments demonstrate the effectiveness of our method in prediction and real-world backtest.
Securities prediction has always been a challenging but engaging task. Many studies [
The securities market is a complex environment with various entities and events, often requiring time
for their interactions to be fully reflected in price data. For example, the management change information
in the announcement will affect its stock price, and the crude oil price will affect the stock price of
airline companies. These facts inspire researchers to utilize external information for more accurate price
prediction. Thus, multi-source data, such as sentiment information [
Substantial efforts have been made to leverage this multi-source data. Nevertheless, many treat these
data in isolation, ignoring the relations’ role in promoting prediction performance [
To overcome these challenges, we propose an economical and effective LLM-driven knowledge enhancement for securities index prediction. First, we collect the daily data of Shanghai Stock Exchange indexes (SSE indexes) and their related market indicators, such as the Northbound Capital and the Shenzhen Composite Index, as multi-source data to overcome the hysteresis of price data. Second, we establish the relationships between them by leveraging a large language model (LLM) as a knowledge source, thus obtaining diverse knowledge inexpensively. Third, we integrate the collected numeric data and knowledge into a heterogeneous graph. Finally, we apply a heterogeneous graph-based neural network to learn the representation of SSE indexes and predict their price movement. The experiment results demonstrate the effectiveness of our method in prediction and real-world backtest. In conclusion, our contributions are as follows:
1) We propose a LLM-driven knowledge enhancement method for securities index prediction, leveraging LLM as an automated knowledge base to acquire various knowledge efficiently.
2) We implement a heterogeneous graph-based neural network to incorporate the numeric multi-source data and their knowledge, resulting in a good improvement in the downstream prediction task.
3) We validate the effectiveness of our method in securities index trend prediction, covering 175 securities indexes and 9 years of data. Additionally, we verify the method’s real-world profit capabilities through backtest.
Deep learning has been a promising approach to stock market prediction [19]. Recurrent models, such as
RNN [20] and LSTM [21], are particularly prominent due to their ability to capture temporal information
[19, 22]. In the previous work, the majority enhance their method by incorporating multiple numerical
and textual data, e.g., technical indicators [23], social text [24], and financial news [25]. Nonetheless,
these data often reflect a post-event status of the predicted object in isolation. Since events propagate
between different entities in sequence, it causes a lead-lag effect in these data [
Instead of concatenating the multi-source data as input features, some work also leverages the internal
relationships of the data. For instance, Cheng et al. [
Knowledge enhancement has become increasingly crucial in stock market prediction [
However, the primary methods are confronted with acquisition costs and knowledge diversity. Recently,
large language models have demonstrated their potential for knowledge acquisition, engaging many
researchers in information extraction [
Our objective is to predict the daily trend of SSE indexes, framing it as a binary classification task. Assuming today is day , we use the average closing price over the past days as the baseline. If the closing price on day in the future exceeds the baseline, it is considered an rising trend and labeled as a positive sample. The formula for setting the labels is as follows: = {︃1, 0, + > ∑︀ =−01 − ℎ (1) where represent the closing price on day , and ≥ 1 is a constant that determines the threshold for classifying the trend. Using the average as a baseline can avoid the impact of market fluctuations. represents the prediction horizon.
The data used for trend prediction comes from multiple sources. In addition to SSE indexes trading data , it also includes other data sources 1, 2, · · · , . These data sources have varying dimensions and granularity. In subsequent sections, we will introduce how to organize these data sources into input data for the prediction task.
Generally, the data used for SSE index prediction comes from two sources: SSE index trading data and other market indicators, as shown in the Figure 1. Trading data has a direct relationship with the SSE indexes, but the available historical data often exhibits lagging characteristics. On the other hand, market indicators refer to other economic market indicators related to the SSE indexes. These indicators can reflect more environmental factors and provide guidance for predicting the trends of SSE indexes. Traditional methods typically fuse these two types of data by simply appending market indicators to the daily trading data, thereby expanding the input feature dimensions. However, they overlooks the semantic information and connections behind the market indicators, which are crucial for improving prediction performance.
We propose a knowledge enhancement method based on Large Language Models (LLM), leveraging LLMs to automatically embed knowledge into market indicator data. We argue that the knowledge of market indicators is reflected in the relationships between them. For example, there is a connection between "Southbound Capital (SC)" and the "Hang Seng Index (HSI)". SC reflects the capital flowing from mainland China into Hong Kong, while HSI is an index of the Hong Kong stock market influenced by SC. To characterize the relationships between market indicators, we adopt a approach that establishes relational paths formed by multiple triplet links. In the example above, we take SC and HSI as nodes and form a triplet link based on their relationship: [(SC, index status, Capital from Mainland), (Capital from Mainland, participate in, Hong Kong market), (HSI, index status, Hong Kong market)].
To extract relationships among numerous market indicators, we leverage a LLM to automatically uncover the connections between market indicators. The specific process is illustrated in the following steps:
Step 1: Instruction Construction. We construct instructions with known nodes for input into the LLM. These instructions involve the task description, ontology constraints, output format, examples, and other relevant information. The instructions must enable the LLM to fully understand our task of extracting relationships between market indicators.
Step 2: LLM Interaction. We input the constructed instructions into the LLM and obtain the feedback results. We then validate these results and extract triplets from the successful outcomes.
Step 3: Node Update. After saving the newly generated triplets, we update the known node pool with the newly generated nodes. We repeat Step 1 until the maximum number of iterations is reached.
Step 4: Path Identification. Once the maximum number of iterations is reached, we identify paths from any two market indicator nodes as start and end points using the triplet set. Irrelevant triplets that are not on the identified paths are removed.
The iterative process generates intermediate nodes for market indicators, fully utilizing the knowledge base and reasoning capabilities of the LLM to uncover complex multi-hop and cross paths. After this process, multiple market indicators, including SSE indexes, form a connected heterogeneous graph = (, ).
Furthermore, we align data with knowledge to form the input for the SSE index prediction model. Assuming the prediction model aims to forecast the trend on day , we organize the input data into a graph structure = (, , ). The node set is divided into two categories based on their sources: market indicator nodes , which include SSE indexes among others, and intermediate nodes , which generated by the LLM and reflect the relationships between market indicators.
The market indicator nodes have corresponding data values as predictive support, serving as node feature attributes (, ), where ∈ . For instance, the SSE indexes node uses trading data from the period [ − , ] as its attribute values (, ) = [− , ]. These attribute values reflect the specific state of the nodes at the target time. Differences in these states directly influence the prediction results.
The intermediate nodes do not have specific numerical values but possess clear semantics. Therefore, we number all intermediate nodes and use one-hot encoding as their attributes, i.e., (, ) = ℎ(, ||). These intermediate node encodings reflect the path semantics, guiding the information aggregation between market indicators. The attributes of all nodes form the attribute set . This graph, enriched with knowledge for the prediction , is then input into the model.
Based on the graph data = (, , ) 1, we then construct a GNN model () to predict the trend of the index. The feedforward computation process of this model, as illustrated in Figure 3, consists of three main components: feature mapping, feature fusion, and classification output.
Feature mapping aims to map different types of nodes into a unified vector space. In the input heterogeneous graph, the node set contains types of nodes, and each type of node corresponds to a ifxed set of feature attributions. We prepare a feature mapping function for each type of node. Given a node of type and its associated feature values a, the computation of feature mapping is as follows: 1omitting the superscript x = (a) = a + b , (2) where and b represent the weights and bias, respectively. The vector x denotes the feature vector of node .
Feature fusion is the process of using graph neural networks to aggregate information from nodes and edges. The feature mapping step outputs vector representations for all nodes in the graph, denoted as = {x, x1, · · · , x− 1}. We employ a Heterogeneous Graph Transformer (HGT) model to learn the representations of nodes and the topological structure. The fusion vector representation of node is denoted as ℎ, where indicates the layer number of HGT, and the initial fusion vector is h0 = x. The entire network consists of layers, and the feedforward process for each layer is calculated as: h+1 = ℎ(h+1, {(h , )| ∈ ()}), where () represents the neighbor nodes of , and represents the edge between nodes and . The HGT model, based on the Transformer architecture, can aggregate features from different types of nodes and edges. The vector representation of SSE index nodes includes not only the node’s original features but also additional data features and the semantic knowledge underlying the data.
Classification output predicts the trend of the SSE index based on the high-level representation. Given the fusion feature representation ℎ of the SSE index node, we use a fully connected neural network and the softmax function to calculate the probability of the index rising or falling. This computation is defined as:
= Softmax( h + b).
The model output is a 2-dimensional vector, with each element corresponding to the probabilities of the index rising and falling, respectively. During model training, we utilize the cross-entropy function as the loss function and update all learning parameters through gradient descent.
In this section, we prepare model prediction and market backtesting experiments to validate our method. (3) (4)
Datasets. We collect the price and trading data of 175 SSE indexes from 2013 to 2021, along with 6 technical indicators and 12 market indicators. With the collected data, 364,314 prediction samples are formed. We split the datasets and conduct the experiments by year. In the prediction experiment, we adopt a time-series cross-validation,as referenced in [44]. The samples of each year is split into vfie parts: Jan to Apr, May to Jun, Jul to Aug, Sep to Oct, and Nov to Dec. For the -th fold validation, we take the ifrst parts as a training set and the + 1-th part as a validation set. In the backtesting experiment, we use the samples from Jan to Oct as a training set and those from Nov to Dec as a test set.
Evaluation Metrics. In the prediction experiment, we apply the accuracy (Acc), precision (P), recall (R), and Macro-F1 score (F1) as metrics to evaluate the prediction performance of different models. In the backtesting experiment, we define the daily return ( ) as follows: = || ∈ 1 ∑︁ − − 1 − 1 , where denotes the price of index at time , and denotes the number of indexes in the portfolio held by the model at time . We then use the average daily return (DR) and the Sharpe ratio (SR) to measure the profitability of different models. In the Sharpe ratio, we use the 1-year China Government Bond Yield as the reference for the risk-free rate. To align with the daily return , the annual Bond Yield is divided by 365.
Implementation Details. For the label setting, we take = 5, = 2, and = 1.01 as specified in Eq. 1. The heterogeneous graph generated by the LLM consists of 30 nodes and 56 edges, with the time interval for node features set to = 5. During training, we use the Adam optimizer with a batch size of 512 and a learning rate of 0.001, and we implement early stopping to prevent overfitting. In our GNN model, the embedded vector dimension for feature mapping is set to 60, the hidden vector dimension of the HGT is 90, the number of attention heads is 4, and the number of layers is = 6. For the baseline settings, recurrent models have 4 layers with a hidden vector dimension of 360; graph-based models use an embedded vector dimension of 100, a hidden vector dimension of 120, 3 attention heads, and 4 layers.
We compare our model, Knowledge-enhanced HGT (KHGT), with several baselines, including recurrent models such as LSTM and Bi-LSTM, and graph-based models such as GAT and HGT. Their average performance over 9 years is presented in Table 1. The results indicate that KHGT outperforms all baselines. Furthermore, yearly comparison results are illustrated in Figure 4, showing that KHGT demonstrates strong generalization, achieving the highest Macro-F1 score and accuracy in most years from 2013 to 2021.
By comparing graph-based models and recurrent models, we observe that graph-based models generally do not outperform recurrent models. Specifically, GAT lags behind other baselines in terms of recall and F1 score, and HGT is slightly inferior to Bi-LSTM in terms of accuracy, recall, and F1 score. This indicates 2* denotes that the improvement is significantly in a paired t-test ( < 0.05), same as below. (5) 0.800 0.775 0.750
HGT 2013 2014 2015 2016 2018 2019 2020
2021 2017
Year LSTM
Bi-LSTM
GAT
HGT
KHGT that the structural information in GAT and HGT does not provide a significant predictive enhancement. The superior performance of KHGT is attributed to the contribution of knowledge enhancement.
To assess the real-world profitability of KHGT, we backtest it over a period of 9 years. We adopt a
straightforward and effective trading strategy, as referenced in [
Table 2 presents the average performance of each model, where "Market" refers to always holding all indexes. The results demonstrate that KHGT outperforms all baselines across both metrics. For the baselines, graph-based models generally lag behind recurrent models, consistent with the findings from the prediction experiment. Notably, the profitability of KHGT remains optimal across both metrics.
Intervention experiment. To further verify the effectiveness of knowledge enhancement, we conducted an intervention experiment. We randomly deleted a portion of the knowledge paths between market indicators. A deletion proportion of 1 indicates no knowledge enhancement. For each proportion setting, we randomly selected paths and repeated the experiment three times, reporting the average results.
Table 3 presents the model performance under different proportions. The performance decreases as the number of knowledge paths increases. When eliminating all knowledge, its performance will degrade to the same level as HGT. This result implies that the LLM’s knowledge has a strong enhancement effort. Without domain experts, our method can fully exploit the relationships between market indicators, thereby enhancing the predictive ability of the model in an economical and effective manner. -6.0 -4.0 -2.0 0.0 2.0 4.0 6.0
Price Movement (%)
Visualization of Attention. We train the model with the samples from 2021 and calculate the average attention weight of each edge type. The attention weights in KHGT are visualized in Figure 5. The x-axis represents the price movement of the prediction target, and the y-axis represents the edge type. calculate
deposit get services
from e index status p of y Te influence g participate Ed in
provide services to purchase and
sell regulate 0.30 0.25 0.20 0.15 We observe that "index status of" and "participate in" play relatively significant roles in falling and rising markets, respectively. "Index status of" connects indicators to financial objects, reflecting the impact of market indicators on the objects they measure. "participate in" connects these financial objects to markets, reflecting the convergence and interaction of these objects. The prominence of these two relations suggests that the model tends to focus on changes in specific indicators during falling markets and the interplay of multiple indicators during rising markets.
This work proposes an LLM-driven knowledge enhancement method for the task of securities index prediction. The innovation of this method lies in leveraging the rich knowledge within LLM, significantly reducing the cost and improving the efficiency of acquiring market index-related knowledge. By interacting with LLM through instructions, we establish triplet paths among market indices, thereby forming graph data that embodies implicit knowledge. Utilizing the graph-structured market data, we construct a GNN-based prediction model, which significantly outperforms traditional models.
Limitations and future work. The quality of the knowledge obtained from the LLM depends on itself and the design of the instructions. Knowledge quality is a crucial constraint affecting the model’s predictive capability. Therefore, ensuring the quality of the knowledge is a critical issue that needs to be addressed in future work.
This work is supported by the National Natural Science Foundation of China (No. 72071145). [9] Xin Liang et al. “F-HMTC: Detecting Financial Events for Investment Decisions Based on Neural
Hierarchical Multi-Label Text Classification.” In: IJCAI. 2020, pp. 4490–4496. [16] Shumin Deng et al. “Knowledge-driven stock trend prediction and explanation via temporal convolutional network”. In: Companion proceedings of the 2019 world wide web conference. 2019, pp. 678–685. [17] Shunrong Shen, Haomiao Jiang, and Tongda Zhang. “Stock market forecasting using machine learning algorithms”. In: Department of Electrical Engineering, Stanford University, Stanford, CA (2012), pp. 1–5. [18] Sheng Xiang et al. “Temporal and Heterogeneous Graph Neural Network for Financial Time Series Prediction”. In: Proceedings of the 31st ACM International Conference on Information amp; Knowledge Management. CIKM ’22. ACM, Oct. 2022. DOI: 10.1145/3511808.3557089. URL: http://dx.doi.org/10.1145/3511808.3557089. [19] [20]
Weiwei Jiang. “Applications of deep learning in stock market prediction: recent progress”. In: Expert Systems with Applications 184 (2021), p. 115537.
David E. Rumelhart, Geoffrey E. Hinton, and Ronald J. Williams. “Learning representations by back-propagating errors”. In: Nature 323 (1986), pp. 533–536. URL: https://api.semanticscholar. org/CorpusID:205001834. [21] Sepp Hochreiter and Jürgen Schmidhuber. “Long short-term memory”. In: Neural computation 9.8 (1997), pp. 1735–1780. [22] Luca Di Persio and Oleksandr Honchar. “Recurrent Neural Networks Approach to the Financial
Forecast of Google Assets”. In: 2017. URL: https://api.semanticscholar.org/CorpusID:67797781. [23] Simone Merello et al. “Ensemble Application of Transfer Learning and Sample Weighting for Stock Market Prediction”. In: 2019 International Joint Conference on Neural Networks (IJCNN) (2019), pp. 1–8. URL: https://api.semanticscholar.org/CorpusID:203606095. [24] Gary Ang and Ee-Peng Lim. “Learning Dynamic Multimodal Implicit and Explicit Networks for
Multiple Financial Tasks”. In: 2022 IEEE International Conference on Big Data (Big Data). 2022. [25] Ziniu Hu et al. “Listening to Chaotic Whispers: A Deep Learning Framework for News-oriented Stock Trend Prediction”. In: Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining. 2018, pp. 261–269. [26]
Matthew L O’Connor. “The cross-sectional relationship between trading costs and lead/lag effects in stock & option markets”. In: Financial Review 34.4 (1999), pp. 95–117.
Wei Li et al. “Modeling the stock relation with graph network for overnight stock movement prediction”. In: Proceedings of the twenty-ninth international conference on international joint conferences on artificial intelligence . 2021, pp. 4541–4547.
Wasiat Khan et al. “Stock market prediction using machine learning classifiers and social media, news”. In: Journal of Ambient Intelligence and Humanized Computing (2022), pp. 1–24. Yingmei Chen, Zhongyu Wei, and Xuanjing Huang. “Incorporating Corporation Relationship via Graph Convolutional Neural Networks for Stock Price Prediction”. In: Proceedings of the 27th ACM International Conference on Information and Knowledge Management (2018). URL: https://api.semanticscholar.org/CorpusID:53037746. Kai Zhang, Bernal Jiménez Gutiérrez, and Yu Su. Aligning Instruction Tasks Unlocks Large Language Models as Zero-Shot Relation Extractors. 2023. arXiv: 2305.11159 [cs.CL].