There are various unknown association patterns are involved in the dynamic spatiotemporal dependencies of multivariate time series forecasting, making it a challenging task. However, current mainstream time series prediction models often neglect potential causal relationships. Due to the presence of hidden confounders, these models inadvertently learn spurious relationships. Overlooking potential causal relationships and spurious relationships may lead to limited generalization capabilities when handling out-of-distribution data. To address this challenge, we draw inspiration from the field of causal inference and incorporate a random intervention mechanism. We propose the Causal Intervention Spatiotemporal Graph Neural Network(CISTGNN), which ofers a causal perspective on time series forecasting. Our approach involves randomly sampling and recombining variant patterns across diferent periods to create an intervention distribution, thereby eliminating the misleading efects of hidden confounders. A series of experiments test on a real trafic flow dataset to validate the efectiveness of the proposed method. Compared to the baseline model, our model improves by 1.37 percent on average.
Multivariate time series data, found in various domains like cloud computing, trafic, energy, finance,
and social networks [
However, most time series forecasting methods tend to prioritize spatiotemporal correlations within
sequences while neglecting the underlying physical principles and causal relationships between these
models [
To address this challenge, we conduct a comprehensive causal analysis of the prediction process of the spatiotemporal graph neural network. This analysis gives us a deep understanding of the relationships among various elements, including inputs X, hidden confounders HC, causal features C, shortcut features S, hidden layers H, and predictions Y, as shown in Fig 2. Firstly, hidden confounders are unobserved factors that can afect trafic flow, such as weather, holidays, and unexpected events, as shown in Fig 1. Secondly, shortcut features are features that are correlated with the prediction target but not causally related. These could include weather, holidays, and unforeseen events, which may form hidden confounders and give rise to shortcut features. Lastly, causal features are features that reflect causal relationships in trafic flow, such as trafic signals, trafic rules, trafic events, and trafic demand. These features help machine learning models to understand and predict changes in trafic flow more accurately, thereby enhancing the interpretability and reliability of predictions. However, these shortcut features can unintentionally create a backdoor path, leading to spurious correlations between causal features and predictions. Therefore, our strategy to address this issue focuses on mitigating the efects of hidden confounders. We aim to enhance the model’s generalization capability by efectively leveraging the potential of causal features while filtering out shortcut features.
S C
HC
H1
H2 X
Y
Hidden confounders
Input Shortcut feature Causal feature Hidden layers Output
In response to these challenges, we propose a novel CISTGNN known as the Dynamic Graph Attention Network Based on Causal Intervention. Our approach efectively addresses out-of-distribution by identifying and harnessing stable spatiotemporal patterns with reliable predictive capabilities. To achieve this, we introduce a disentangled spatial attention network that captures both variant and invariant patterns within dynamic graphs. This network empowers each node to focus on its historical neighbors through a disentangled attention information propagation mechanism. Drawing inspiration from the field of causal inference, we incorporate a random intervention mechanism. This innovative approach involves sampling and recombining variant patterns across diferent periods to create an intervention distribution, thereby eliminating the misleading efects of variable patterns. Consequently, our model becomes capable of capturing and leveraging stable spatiotemporal patterns that ofer dependable predictive performance, even when dealing with out-of-distribution. Our contribution can be summarized as follows: • Firstly, we analyze the physical mechanisms behind data generation. This analysis forms the foundation for constructing a causal graph, which explicitly describes the causal relationships among various factors in the data and identifies the impact of hidden confounders. • Secondly, we propose a spatiotemporal graph neural network prediction model based on the principle of causal intervention. In our model, we use the backdoor criterion to efectively mitigate the influence of hidden confounders. • Finally, the reliability and validity of our model are validated by extensive experiments on real-world datasets. Compared with the baseline model, our model improves by 1.37% on average.
Spatiotemporal Graph Neural Networks are a type of graph neural network used for processing
spatiotemporal data, capturing dependencies in both spatial and temporal dimensions [
ASTGNN [
Disentangled Representation Learning is a machine learning approach aimed at acquiring
representations from data that can separate diferent factors or features [
Causal Inference plays a crucial role in shaping the design of machine learning algorithms,
providing essential guidance for their development. As artificial intelligence continues to advance, an
increasing number of researchers are recognizing the pivotal role of causal inference in addressing the
limitations of existing AI methods, particularly in areas like abstraction, reasoning, and interpretability.
In his book The Book of Why, Turing Award winner Judea Pearl categorizes causal inference into three
levels: the first level is “association”; the second level is “intervention”; and the third level is
“counterfactual inference” [
This paper focuses on the prediction of multivariate time series and leverages a graph-based structure to capture the relationships among various variables. In this approach, individual variables in the data are treated as nodes in a graph, and the observations associated with each node are interpreted as either the node’s features or graph signals, which is an intuitive and natural methodology. To represent the connections between these nodes, a graph adjacency matrix is employed.
Definition 1 (Graph
): We represent the relationships among multivariate variables using the notation = (, , ). In this context, the graph consists of a set of vertices (or nodes) ∈ R and a set of edges , where each edge connects two vertices, indicating the presence of a specific relationship matrix has dimensions × , where represents the number of vertices in the graph. between them. The matrix ∈ R× describes the connections between these nodes. The adjacency
The objective of predicting time-varying graph signals is to learn a mapping function that can estimate the future features or properties of nodes within the graph based on their historical attributes. This process entails leveraging the historical attributes of the nodes over the past time steps to make predictions about the features or attributes of the nodes for the future time steps. ︀( X− +1, X− +2, . . . , X)︀ →− (︀ X+1, X+2, . . . , X+ ︀)
In this section, we will begin by presenting the comprehensive flow of our model, as shown in Fig 3. We will commence with an analysis of causal inference as applied to the learning process of graph neural networks. We will involve introducing the relevant Structural Causal Models (SCM) and discussing the concept of backdoor adjustment. Given our assumptions, we think that hidden confounders play a pivotal role in influencing the model’s generalization. Consequently, we propose a novel strategy to mitigate the adverse efects of these hidden confounders. The overarching model comprises four key components: a causal feature learning module, a K-layer spatiotemporal convolution module, a causal intervention module, and a predictive output module. The interaction between each module involves several steps. First, we disentangle the features into causal and non-causal features using the causal feature module. Next, the model learns causal features and non-causal features separately in K-layer spatiotemporal features. Then, we perform random interventions on non-causal features and fuse causal features. Finally, we make the final prediction using the prediction module.
X1 X2 X3 Xn
Spatial Attention
G N N G N N
We first analyze the underlying mechanisms of data generation from the perspective of structural causal models, which can provide better interpretability for the model. We propose a novel framework, CISTGNN, which incorporates causal reasoning into the prediction of future time series in a spatiotemporal environment. We use a structural causal model (SCM) to describe the causal relationship between input X, hidden confounders HC, causal feature C, shortcut features S, spatiotemporal state H, and prediction target Y, as shown in Fig 2, where directed edges denote the causal relationships among nodes. We assume that causal feature C and shortcut features S can be disentangled from the spatio-temporal data X and integrate them to form the spatio-temporal state H, thus describing the dynamic spatio-temporal patterns in the data.
The contextual condition HC is defined as the common cause of both X and H. This situation can
lead to H being biased towards the general state while potentially disregarding specific environmental
factors due to dataset limitations, resulting in an unfair bias of H. It is evident from Fig 2 that certain
backdoor paths exist, including HC → X → C and HC → X → S. Breaking the link HC → X → C
enables X to fairly incorporate each contextual condition C into the spatiotemporal state H. To achieve
graph representation learning, we ought to eliminate these backdoor paths. Fortunately, causal theory
[
( | ()) = ( | ) = ∑︁ ( | , )( | ) ∈ℋ ∈ℋ ∈ℋ = ∑︁ ( | , )() = ∑︁ ( | , ) (),
From the analysis of Formula (1), it is known that we can mitigate the bias brought by hidden confounders through the backdoor criterion. In our paper, our approach is to perform random interventions on non-causal features and fuse causal features.
To better learn causal features, we disentangle spatial features into invariant patterns and variant
patterns. In the spatial dimension, diferent nodes interact with each other, and this interaction is highly
dynamic. To capture this dynamism, this paper employs an attention mechanism that can adaptively
capture spatial causal relationships. We use the following attention mechanism[
(− 1) represents the input data of the -th layer, which is a three-dimensional tensor characterized by dimensions × − 1 × − 1. Here, signifies the number of nodes, − 1 represents the number of channels, and − 1 denotes the time length. stands for the parameter matrix for spatial attention, which is a two-dimensional matrix with dimensions × . The bias for the spatial attention mechanism denoted as , is a one-dimensional vector with dimensions . Parameter 1 is a Si,j of this matrix represents the relevance strength between node and node . one-dimensional vector with dimensions − 1. Additionally, 2 represents the parameter matrix for spatial attention with dimensions − 1 × − 1, and 3 is a one-dimensional vector with dimensions − 1. To enhance the expressive power of the model, we apply the activation function , which is a non-linear function. The attention matrix , is a two-dimensional matrix sized × . Each element
PI = (Si,j) , Pv = (− Si,j)
In the context of our model, we denote the masks for invariant and variant patterns as PI and Pv
respectively [
In the temporal dimension, trafic conditions exhibit correlations between diferent time steps, and the nature of this correlation can change in response to varying situations. To efectively address this dynamic nature of temporal data, we employ an attention mechanism that dynamically assigns varying in the data. levels of importance to diferent time steps. This approach enables the model to adapt its focus based on specific situations, thereby enhancing its ability to capture the evolving correlations and variations Q = B · ︂( ︁( ℎ
Q′, = (Qi,j) matrix, consisting of node degrees, D = ∑︀
A . where A is the adjacent matrix, I is a unit matrix, and the degree matrix D ∈ R× is a diagonal
Secondly, we can represent the graph Fourier transform and inverse graph Fourier transform as an equation:
Graph convolution is a method used to extract spatial features and dependencies in signals on graphs. The core idea is to define a kernel function using the Laplacian matrix and its eigenvalues, and then apply this kernel function to filter the signal on the graph. The graph convolution operation can be represented by the following formula:
* = (L) = (︀ UΛU )︀ = U (Λ)U in graph convolution, * signifies the operation transforming signals on a graph. Initially, and undergo graph Fourier transforms to the frequency domain. These are then multiplied, and the convolution result is obtained via inverse graph Fourier transform. The matrix facilitates the forward transform, while U enables the inverse, translating signals between spatial and frequency domains. within our model, we employ learnable parameters denoted as B, b ∈ R− 1× − 1 , M1 ∈ R , M2 ∈ R− 1× , and M3 ∈ R− 1 . These parameters are essential for computing the elements of the spatiotemporal attention matrix, denoted as . This matrix, with dimensions − 1 × − 1, serves as a representation of the correlations between diferent time steps in the input data. The value of each element Q, in this matrix signifies the strength of dependency between the -th and -th time steps.
The spatiotemporal convolution module utilizes graph convolution to model spatial structures and standard convolution to simulate temporal dynamics. The synergy of these modules equips the neural network to handle spatiotemporal data with precision, significantly improving its modeling capabilities for complex tasks.
Graph convolution extends convolution operations to graph structures, capturing spatial features in graph data, useful for tasks like trafic flow prediction. It’s implemented through spectral and spatial methods. Spectral methods use Laplacian matrices but have high computational complexity. Spatial methods aggregate features with weighted sums, proving more eficient and adaptable to dynamic graphs. The Laplacian matrix, derived from adjacency and degree matrices, is crucial for structural characterization. Eigenvalue decomposition of it yields matrices Λ and , providing spectral properties insights. Graph Fourier transform, facilitated by , shifts signals from spatial to frequency domains, enabling filtering. Graph convolution employs a kernel function and , involving a sequence of graph Fourier transform, filtering, and inverse transform, forming a comprehensive framework for graph signal processing.
Firstly, we can represent the Laplacian matrix and its eigenvalue decomposition as an equation: L = D −
A = UΛU ˆ = U , = Uˆ (4) (5) (6) (7) (8) procedural flow.
Combined, these steps form the graph convolution process, elucidating its foundational concepts and
Nevertheless, in the context of large-scale graphs, a direct eigenvalue decomposition of the Laplacian
matrix can be exceedingly time-consuming. To mitigate this challenge, our paper adopts a more eficient
approach based on the utilization of Chebyshev polynomials to approximate the solution[
(L˜ ) is the maximum eigenvalue of the Laplacian matrix.This definition is as follows: () = 2− 1() − − 2(), with initial values 0() = 1, 1() = . This recursive definition is pivotal in our method as it efectively extracts information from up to 1 neighboring nodes centered around each focal node and applies the convolution kernel . To finalize the graph convolution process, we employ the
K − Rectified Linear Unit (ReLU) as the activation function, denoted as ReLU ( * ). This combination of steps allows eficient and scalable processing of large-scale graphs, efectively approximating the convolution operation of the Laplacian matrix using Chebyshev polynomials.
To dynamically adapt and fine-tune the correlations between nodes, our approach introduces a novel element. For each term of the Chebyshev polynomial, we perform a multiplication operation by taking (L̃︀) and multiplying it by spatial attention matrix PI ∈ R× and PV ∈ R× . This operation is symbolized as (L˜ ) ⊙
PI and (L˜ ) ⊙
PV, where the symbol ⊙ represents the Hadamard product. Consequently, the graph convolution formula mentioned earlier can be succinctly represented as follows: * = (L) = ∑︁ * = (L) = ∑︁
(9)
(10)
(11)
(12)
This definition can be extended to accommodate graph signals with multiple channels. For instance,
in recent developments, the input is represented as ℎ
− 1 channels. At each time step , filters are applied to the graph
each node is updated by leveraging information from its 0 to K − 1 order neighbors.
(Θ 1, Θ 2, . . . , Θ ) ∈ R× − 1× represents the convolutional kernel parameters[
After conducting a graph convolution operation to capture the neighborhood information of each node in the spatial dimension, then we introduce a standard convolutional layer in the temporal dimension. The primary purpose of this convolutional layer is to update the node signals by efectively , yielding * , wher Θ = (− 1), where each node’s features encompass integrating information from adjacent time slices.
ℎ () = Re (︁ Φ * ︁(
Re (︁
* ˆ (ℎ− 1))︁ (− 1) denotes the output of the -th layer within the recent component. In this formula, the parameter ℎ represents the parameters of the time-dimension convolution kernel, signifies the graph convolution unit activation function. This formula can be broken down into the following sequential steps: kernel function, * denotes the graph convolution operation, and Re stands for the rectified linear
We introduce a spatial-temporal convolution module that relies on spatial-temporal convolution operations and spatial-temporal attention mechanisms. The purpose is to extract both spatial and temporal features from trafic data. To achieve this, we stack multiple spatial-temporal convolution modules and spatial-temporal attention modules within a spatial-temporal block. This strategy allows us to comprehensively capture a broad spectrum of dynamic spatial-temporal correlations. To ensure that the output of each component aligns with the prediction target, we have incorporated a fully connected layer after each component. Additionally, we utilize the rectified linear unit (ReLU) as the activation function.
We propose a novel spatiotemporal graph neural network called CISTGNN, which excels in handling spatiotemporal data. We evaluate its performance through a multi-step prediction task, aiming to predict data for multiple future time steps. We conduct Experiments on four publicly available trafic datasets that already include graph structures. On datasets with graph structures, we compare CISTGNN with other spatiotemporal graph neural networks that use predefined graphs.
We evaluate the performance of our proposed CISTGNN model using four trafic datasets, PeMSD3, PeMSD4, PeMSD7, and PeMSD8, both of which come equipped with graph structures. We divide the dataset into training, validation, and test sets in a 6:2:2 ratio, maintaining the chronological order. In the multi-step prediction task, we leverage data from the preceding 12 time steps to forecast data for the subsequent 12-time steps, efectively predicting trafic flow for the next hour based on the previous hour’s data. Our CISTGNN model is implemented using the PyTorch framework, and the experiments are conducted on a machine featuring an Intel(R) Xeon(R) Silver 4310 2.10GHz 12-core CPU and an NVIDIA GeForce GTX 4090 with 24 GB of GPU memory. The source code for our CISTGNN can be found at https://github.com/xinxinluo123/CISTGNN. During model training, we employ the Adam optimizer with a learning rate of 0.001. The model’s multi-step prediction task consists of Stacking three spatio-temporal layers. The first layer of the output module consists of 512 output channels, while the second layer has 12 output channels. Our training process set 80 epochs, with a node embedding dimension set at 10. The batch size remains configured at 32. Additionally, we leverage Chebyshev polynomials of order 2, utilizing a total of 64 Chebyshev filters and 64 temporal filters.
To evaluate the performance of the multi-step prediction task, we employ three evaluation metrics: Mean Absolute Error (MAE), Mean Absolute Percentage Error (MAPE), and Root Mean Square Error (RMSE). We denote the real signal as and the predicted signal as ˆ . Both are represented as matrices with dimensions × , where signifies the number of nodes, signifies the number of time steps, and indicates the test set. These metrics provide valuable insights into the accuracy and robustness of our predictions.
We assess the performance of our model through a multi-step prediction task and conduct a
comparative analysis against nine baseline models: FC-LSTM [
ASTGCN 18.67 ± 0.42 30.71 ± 1.02 19.85 ± 1.06 22.90 ± 0.20 35.59 ± 0.35 16.75 ± 0.59 28.13 ± 0.70 43.67 ± 1.33 13.31 ± 0.55 18.72 ± 0.16 28.99 ± 0.11 12.53 ± 0.48
Graph WaveNet 19.85 ± 0.03 32.94 ± 0.18 19.31 ± 0.49 25.54 ± 0.03 39.70 ± 0.04 17.29 ± 0.24 26.85 ± 0.05 42.78 ± 0.07 12.12 ± 0.41 19.13 ± 0.08 31.05 ± 0.07 12.68 ± 0.57
We utilize the CISTGNN model for the multi-step prediction task. In the context of multi-step prediction, Table 2 provides an overall performance comparison between our CISTGNN model and 9 representative comparison methods. This comparison is based on average MAE, RMSE, and MAPE across 12 prediction time steps, leading to the following observations:
Table 2 shows the comparison of the prediction performance of CISTGNN with the 9 benchmark methods. We observe that (1) time series prediction models, including traditional methods (i.e., VAR), and machine learning-based methods (i.e., SVR) because they only consider temporal features but not spatial correlation, which is equally important for spatio-temporal trafic prediction. Therefore, they have the worst prediction performance (2) Spatiotemporal graph neural networks generally perform better because they use graph neural networks to further model spatial correlation. DCRNN is a typical RNN-based method for spatiotemporal graph data prediction, and STGCN, ASTGCN, Graph WaveNet, and STSGCN are four typical CNN-based methods that only focus on the correlation of spatio-temporal data and ignore the causality of spatio-temporal data.
We conduct experiments on the PEMS03 dataset. The results show that our method improves the
MAE, RMSE, and MAPE(%) by 0.82, 0.89, and 1.16 respectively, compared to the baseline method [
Our CISTGNN model comprises several key components, all of which contribute to the model’s performance. To validate their efectiveness, we conducted ablation experiments on the PeMSD4 and PeMSD8 datasets by removing the following components: • CISTGNN w/o causal intervention: Causal intervention has been removed, and spatiotemporal attention is now directly transferred to the graph convolutional neural network, focusing solely on the dynamic attention mechanism without causal intervention. • CISTGNN w/o causal attention: Eliminate the causal attention module and replace it with direct causal intervention on non-causal data. • CISTGNN w/o causal intervention + causal attention: The causal intervention and causal attention models in CISTGNN have been removed, allowing the model to concentrate solely on relevance learning.
We evaluate the test results using three metrics: RMSE, MAPE, and MAE. The specific results are shown in Fig 4 and Fig 5. From these results, we can make the following observations: • The CISTGNN model outperforms all datasets, confirming the efectiveness of each component. • Removing causal intervention leads to a performance decrease, particularly in terms of RMSE on the PeMSD4 dataset and MAE on the PeMSD8 dataset. • Removing causal attention leads to performance degradation, especially in PeMSD8, which emphasizes the importance of establishing causal attention when establishing dynamic dependencies.
In our study, we rethink the utilization of spatio-temporal graph neural networks for multivariate time-series prediction, with a specific emphasis on the causal perspective. We’ve discovered that existing spatio-temporal graph neural network learning approaches frequently depend on shortcut features to support their predictions. However, these shortcut features might inadvertently introduce confounders and create backdoor paths, leading to erroneous correlations in spatiotemporal graph neural network learning. To address this confounding efect, we introduce a causality-based attention-learning mechanism and a causal intervention mechanism guided by causality theory. We propose the Causal Intervention Spatiotemporal Graph Neural Network (CISTGNN), which ofers a causal perspective on time series forecasting. CISTGNN is composed of two crucial components: the Causal Spatiotemporal Attention Module and the Causal Intervention Module. By distinguishing between causal relationships and spurious ones, we reduce the model’s reliance on shortcut features and efectively leverage causal features. Experimental results validate the eficacy of this approach.