Spatio-temporal predictions have been extensively studied due to their impact on real-world applications [1,2,3,4,5]. For example, heavy rainfall events can cause significant damage to infrastructure and pose serious threats to human safety. Predicting these events with greater accuracy allows better preparation and response [6], ultimately saving lives and reducing the economic impact of such events.
Deep learning methods, such as deep spatio-temporal prediction models [7,8], have improved the performance of rainfall forecasting over the years. However, the role of external data in enhancing the prediction accuracy is still controversial. Some argue that external data can provide more useful information for the prediction model, while others claim that external data can introduce more noise and complexity to the learning process. In this study, we propose to improve spatio-temporal predictions by combin-ing spatially-linked external real-world data along with a TGCN to learn the spatio-temporal dependencies from the combined data. As it has been proven that utilizing more multi-source real-world data is more likely to lead to higher accuracy [9], our study aims to introduce a fresh perspective on integrating external real-world data into the proposed framework. We use heavy rainfall prediction as a case study for our proposed method, and overall we aim to provide accurate spatio-temporal predictions by leveraging as much information as possible, enabling better decision-making for a broad range of spatio-temporal applications and at the same time offering a novel angle and a comprehensive evaluation to demonstrate the feasibility of integrating additional external real-world data without the necessity of customizing transformer attention mechanisms. Our approach is experimentally validated by predicting heavy rainfall events and real estate hotspots.
The traditional method for predicting heavy rainfall involves manually engineering features from weather data, including temperature, pressure, humidity, etc. Meteorologists rely on their expertise to interpret this data and forecast future weather patterns. This process entails observing and analyzing atmospheric factors to predict weather patterns. However, this traditional approach is time-consuming, labor-intensive, and susceptible to human error, especially when dealing with large datasets. As data grows, it becomes increasingly challenging to analyze large amounts of information by hand.
Previous research has investigated using deep learning for precipitation prediction [10,11] with promising results. However, some limitations can be significantly improved to enhance deep model performance. One area with room for enhancement is leveraging spatial dependencies. To tackle this challenge, we propose a model that integrates both Graph Convolution Networks (GCNs) and a Transformer. This model enables combining external spatially-linked data for spatio-temporal predictions.
Specifically, we employ a GCN to analyze the adjacency matrix on a grid level and generate correlations between each grid element. The GCN captures the spatial relationships and dependencies among neighboring grid points, allowing for a comprehensive understanding of the data's spatial dynamics. We then utilize a Transformer model to encode the temporal precipitation data and combine it with the spatial correlations obtained from the GCNs. By combining the GCNs and the Transformer within the proposed TGCN model, we create a framework that harnesses both the spatial and temporal dimensions of the data.
Graph Convolutional Networks (GCNs) are a type of deep learning model designed to process data represented in a graph structure, such as social or sensor networks [12]. GCNs have demonstrated their effectiveness in various applications, including node classification, link prediction, and recommendation systems [13,14,15,16]. The concept of Graph Neural Networks (GNNs) was initially introduced in [17] and further expanded upon in subsequent research by [18]. GNNs, a type of recurrent neural network (RNN), iteratively propagate information from neighboring nodes until reaching a stable fixed point. This iterative process has traditionally been computationally expensive, but recent studies, such as [19], have made significant improvements in this area. Inspired by the success of Convolutional Neural Networks (CNNs) in computer vision, which extract highlevel features from images using convolution and pooling layers, current models aim to adapt these layers to directly process graph inputs. GCNs can be categorized into two types of graph convolution layers: spectral graph convolution and localized graph convolution, as discussed in [20]. Early research primarily focused on spectral graph convolutions, pioneered by [21]. The current state-of-the-art model, GCN, further simplified the graph convolution operation by employing a localized first-order approximation. However, spectral methods require operations on the entire graph Laplacian during training, which can be computationally expensive. Several subsequent works, such as FastGCN [22] have aimed to alleviate this issue. Recently, researchers have explored the application of GCNs in time series prediction. For example, spatio-temporal GCNbased approaches have been proposed for traffic flow prediction [23], and the integration of time-aware topological information into GCNs using the mathematical framework of zigzag persistence [24].
In this section, we discuss various existing temporal and spatial-temporal forecasting methods. For example, Recurrent Neural Networks (RNNs), especially long-short-term memory (LSTM) [25], have gained popularity in time series forecasting [26]. Convolutional Neural Networks (CNN) and its variant Temporal Convolutional Neural Networks (TCN) are another option for sequence prediction [27], offering parallel computations compared to RNNs [28]. In recent years, researchers have explored Transformers and its variants in time series forecasting, achieving state-ofthe-art performance in tasks like energy consumption and stock market [29,30,31]. Designing a model capable of comprehensively capturing both spatial and temporal patterns represents another emerging trend in spatial-temporal prediction tasks [32,33]. For example, [33] introduced a spatial-temporal graph neural network for predicting traffic flow.
In this section, we detail our model architecture and the benefits of our design.
The architecture we propose, illustrated in Figure 2, incorporates a combination of techniques to enhance the prediction model. We begin by utilizing a transformer encoder to effectively encode the time series precipitation data, and then integrate local climate features into the model, enabling a comprehensive understanding of the factors influencing heavy rainfall.
To address spatial dependencies and relationships among grid points, a GCN is introduced. This GCN learns the spatial dependencies within the dataset, considering the interconnectedness of grids based on their spatial locations. By leveraging the GCN, the model becomes capable of capturing and integrating spatial information, thereby enhancing prediction accuracy.
The latent code, which combines the encoded time series precipitation data and the spatially connected local climate features learned through the GCN, is fed into a multi-layer perceptron (MLP) for prediction. This integrated architecture allows the MLP model to leverage the fused information, including temporal precipitation data, other climate features, and spatial factors, to effectively learn and infer future heavy rainfall areas.
Our proposed TGCN model consists of Encoder, GCNs and Multi-layer Perceptron (MLP) layers. The major component in the transformer is the Multi-head self-attention.
Where the K and V are matrices that store the keys and values. Q is the query that will map against a set of keys.
We have developed a predictive model using the Transformer architecture, tailored for heavy rainfall forecasts. Unlike traditional methods that only use past rainfall data, our model factors in numerous external variables to boost accuracy. We examine local features, including geography, atmospheric conditions (pressure, temperature, wind), humidity, and topography, all of which influence heavy rainfall likelihood in a specific area. Therefore, we have developed a transformer-based prediction model [34] that incorporates GCNs to process the spatial features. By doing so, our model can capture the spatial relationships among various features in a graph structure, such as the dependencies between grid point locations and their corresponding climate data. The integration of the GCNs enhances our model's ability to capture both temporal and spatial information. Our model design starts with a transformer encoder capturing temporal precipitation patterns, followed by embedding this data and merging it with local climate data like moisture and humidity. We enhance prediction accuracy with this added context. As illustrated in Figure 3, GCNs involve learning a linear transformation of the feature vectors of each node in a graph, which is then used to update the node features by aggregating information from the node's neighbors. Mathematically, this can be expressed as:
In the equation, β
represents the feature vector of node π£π at layer π + 1, π (π+1) denotes the learnable weight matrix for layer π + 1, π© (π£π) represents the set of neighbors of node π£π, and πππ is a normalization constant that ensures proper scaling of the aggregated information. The function π denotes a non-linear activation function, which introduces non-linearity into the model. In our specific case, we utilize the ReLU activation function. This equation can be interpreted as calculating a weighted sum of the feature vectors of the neighbors of node π£π at layer π + 1, where the weights are determined by the learned weight matrix π (π+1) . Then, a non-linear activation function is applied to obtain the updated feature vector β
for node π at layer π + 1. This process is repeated across multiple layers to learn expressive representations of the graph data.
For the final prediction, we utilize a four-layer MLP model that combines time series data with other features, effectively leveraging both temporal and spatial information captured by our model for more accurate predictions.
By leveraging the transformer architecture, incorporating GCNs, and utilizing a four-layer MLP model, our approach enables the effective integration of temporal and spatial information for improved prediction accuracy.
As illustrated in Figure 2, we propose to map temporal data and non-temporal data into the same latent space and merge the latent vectors for the subsequent prediction task.
To encode the local climate features and capture the spatial dependencies among the grid points for data π₯π, we employ a GCN to learn the relationships and dependencies within the spatial domain. The output hidden features at a specific layer πΏ can be denoted as β (πΏ) π . Equation 2 is applied in this context. Assuming we use πΏ π layers in total, and we use the final layer to summarize climate information, which is defined as:
where β 2. We encode temporal precipitation data using a transformer encoder [34],
. Since π₯π‘ and π₯π are encoded as ht and hc, we define the merged hidden state as hm hm = πΆππ πΆπ΄π (ht, hc)
To further process the merged information, we use another multi-layer perceptron specifically trained for the prediction task. Similarly, we define the π-th layer network as (assuming πΏ π layers in total)
where β We use the output from the last layer for prediction
Loss is measured with the Binary Cross-Entropy loss (BCE)
The binary cross entropy (BCE) loss can be formulated as follows:
) where: π is the total number of samples, π¦π is the true label for sample π, ππ is the predicted probability π, log denotes the natural logarithm.
Our data and code are publicly available 1 . In our dataset, the train and test split ratio is 7:3.
1 https://github.com/jiang28/Deep-Spatio-Temporal-Encoding
Our precipitation dataset is sourced from the NOAA HRRR dataset2 , offering real-time climate data at a 3 km spatial resolution and 1-hour temporal resolution. This dataset [35] encompasses total precipitation, precipitation rate, and nine additional climate variables, including humidity (%), moisture availability (%), pressure (Pa), wind speed (m/s), and total cloud cover (%). Simulated brightness temperature data is acquired from the GOES 11 satellite3 . The precipitation data consist of the following three types:
β’ Temporal precipitation data, denoted as π₯π‘, as shown in Table 1 and Figure 5. It captures the historical patterns and fluctuations in precipitation over time. Specifically, we define the temporal precipitation rate and total accumulated precipitation over the past 6 hours as π₯ β , which consists of π timestamps: dataset represents a specific location within the study area, such as a region or a cell. To represent the relationships between these grid points, we used an adjacency matrix. In the adjacency matrix, a value of 0 indicates that two grid points are not neighbors, while a value of 1 denotes their neighboring relationship.
The real estate dataset captures the dynamics of the U.S. real estate market by collecting spatially correlated data from multiple sources. It consists of 7,436 neighborhoods, 567 cities, 304 counties, 225 metros, and 50 states across the U.S. The data are connected through spatial locations, forming a multi-level spatial hierarchy. The dataset consists of three main components: census data, pricing history, and school district information. Here are some statistics about the real estate dataset:
β’ Spatial Hierarchy Levels: The dataset includes a multi-level spatial hierarchy, including information at the state, metro, county, city, and neighborhood levels. β’ Census Data: The census data consists of 16 variables related to various aspects of housing prices, personal income, demographics, and spatial information. β’ Pricing History: The dataset includes temporal housing price history for each neighborhood, spanning from 1996 to 2019. β’ School District Information: The dataset incorporates school district information. It provides details on the number of school districts present in each county within the studied area. Additionally, the dataset includes information on the top school district(s) within the region. Temporal data format. It has data on the grid id, longitude, latitude, grid points, grid spacing, vertical level, timestamps, total precipitation, and precipitation rate.
To facilitate the task of predicting real estate hotspots, the dataset is classified into two classes based on the house price increase rate for each neighorhood: 1 for hotspots and 0 for non-hotspots. The detailed settings of the Real-estate Dataset can be found in [36].
We evaluate the performance of a classification system using various metrics, including Accuracy, Recall, Precision, F1-score, and ROC. These metrics are calculated based on the number of true positives (π‘π), false positives (ππ), false negatives (ππ), and true negatives (π‘π). Accuracy measures the proportion of observations, both positive and negative, that were correctly classified by the system, and can be computed using the formula:
Recall measures the proportion of true positives that were correctly identified by the system, and can be computed using the formula:
Precision measures the proportion of identified positives that were actually true positives, and can be computed using the formula:
F1-score is a weighted average of precision and recall, and provides a single measure of the system's accuracy on the dataset, and can be computed using the formula:
ROC (Receiver Operating Characteristic) curve is a graphical plot that illustrates the performance of a binary classifier system. It is created by plotting the True Positive Rate (TPR) against the False Positive Rate (FPR), which can be computed using the formulas:
Overall, these metrics provide a comprehensive evaluation of a classification system's performance and can help identify areas for improvement. Our study identifies heavy rainfall risk areas based on precipitation rate. Following the United States Geological Survey (USGS) standard 4 , we define the heavy rainfall risk as follows:
Recognizing the significance of precipitation rate as a critical factor, our objective is to pinpoint areas that are susceptible to encountering heavy rainfall within the next hour. The classification into two classes simplifies the problem and provides a clear distinction between areas with different levels of heavy rainfall risk. Using a 4 mm/hour
When comparing model performance on the Precipitation dataset, the proposed model has achieved an accuracy of 86.6%.
threshold, we classify areas as either low-risk (labeled as 0) or high-risk (labeled as 1). For example, out of 10,000 grid points in the study area, 4,798 have a potential for heavy rain risk, while 5,202 do not. This classification simplifies decision-making and resource allocation.
We use the following baseline methods:
β’ Random Forest (RF) [37] β’ Support Vector Machine (SVM) [38] β’ Decision Tree (DT) [39] β’ Linear Regression (LR) [40] β’ Multilayer Perceptron (MLP) [41] β’ Long Short Term Memory (LSTM) [25] β’ Transformer [34]
Based on the results presented in Table 2 and Table 3, we can analyze the performance of different models on the Real Estate dataset and the Precipitation dataset, respectively. In Table 2, the proposed model outperforms all the baseline models with an accuracy of 95.6%. The proposed model also exhibits the highest precision for both classes (0 and 1), achieving 0.93 and 0.97, respectively. It demonstrates high recall values for both classes as well. The F1 scores are also higher for the proposed model compared to the baseline models, indicating a better balance between precision and recall. The TGCN model's performance is further reflected in the ROC score of 0.954, which indicates its ability to discriminate between the two classes effectively.
Table 3 shows that the proposed model again achieves the highest accuracy of 86.6%. Similar to the Real Estate dataset, the TGCN model demonstrates superior precision and recall values for both classes compared to the baseline models. It achieves precision scores of 0.9 and 0.83 for classes 0 and 1, respectively, along with recall scores of 0.82 for class 0 and 0.85 for class 1. The F1 scores also indicate the TGCN model's overall better performance. The ROC score for the TGCN model is 0.867.
These results demonstrate that the proposed TGCN model consistently outperforms the other models on both datasets in terms of accuracy, precision, recall, F1 score, and ROC score. The TGCN model's ability to capture temporal, nontemporal, and spatial information through its integration of the transformer layer and the graph convolutional network contributes to its good performance in identifying and predicting hotspots and heavy rainfall areas.
In conclusion, the accurate prediction of heavy rainfall events is crucial for effective urban water usage, disaster response, and mitigation efforts. This paper proposed a prediction model that leverages spatially connected features and real-world climate data to predict heavy rainfall risks across a broad range. Through extensive experimentation, it was observed that the TGCN model outperformed the other machine learning methods in forecasting both heavy rainfall events and real estate trends.
While this study successfully demonstrated the effectiveness of the proposed TGCN model in predicting heavy rainfall risks, there are several avenues for future research and improvement.
We plan to incorporate more diverse and comprehensive datasets, including additional meteorological and geographical features. This expansion has the potential to enhance the accuracy and generalizability of the TGCN model. Furthermore, we are considering the integration of real-time data streams and the utilization of advanced data fusion techniques to further enhance the model's forecasting capabilities.
This work was partially supported by the National Science Foundation (NSF) under Grant No. 2318641. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not reflect the views of the National Science Foundation.