Forecasting of air pollutant concentration, which is in uenced by air pollution accumulation, tra c ow and industrial emissions, has attracted extensive attention for decades. In this paper, we propose a spatio-temporal attention convolutional long short term memory neural networks (Attention-CNN-LSTM) for air pollutant concentration forecasting. Firstly, we analyze the Granger causalities between di erent stations and establish a hyperparametric Gaussian vector weight function to determine spatial autocorrelation variables, which is used as part of the input feature. Secondly, convolutional neural networks (CNN) is employed to extract the temporal dependence and spatial correlation of the input, while feature maps and channels are weighted by attention mechanism, so as to improve the e ectiveness of the features. Finally, a depth long short term memory (LSTM) based time series predictor is established for learning the long-term and short-term dependence of pollutant concentration. In order to reduce the e ect of diverse complex factors on LSTM, inherent features are extracted from historical air pollutant concentration data meteorological data and timestamp information are incorporated into the proposed model. Extensive experiments were performed using the Attention-CNNLSTM, autoregressive integrated moving average (ARIMA), support vector regression (SVR), traditional LSTM and CNN, respectively. The results demonstrated that the feasibility and practicability of Attention-CNN-LSTM on estimating CO and NO concentration.
In recent years, air pollution, especially the large-scale haze caused by ultra ne particles
and volatile organic compounds (VOCs) of mobile pollution sources, has attracted worldwide
attention [
The approaches for forecasting air pollutant concentrations mainly include deterministic and
statistical models. Deterministic models simulate the atmospheric physic and chemistry in
the processes of emission, di usion and transformation of air pollutions, cannot explain the
non-linearity and heterogeneity of some factors on the formation of pollutants [
Recently, deep neural networks (DNNs) can automatically learn salient feature mappings
from high-dimensional input data, and avoid the complicated process of arti cial design and
extraction of features to solve a wide variety prediction of complex problems, such as further
perfect the prediction performance of air pollutant concentrations. Inherently considering
spatiotemporal correlations of historical air pollutant data, meteorological data and timestamp data Li
et al. proposed a novel LSTM model to forecast air pollutant concentration [
In general, the air polluting process usually involves a variety of interacting pollutants, which are a ected by local reactions, spatio-temporal evolution properties of air pollutant concentration and confounding factors, such as the direction of wind and humidity. Therefore, the research on the prediction of air pollutant concentration still faces the following two challenges: (i) LSTM is hard to deal with the time series with long-term dependency and complex task, (ii) air pollution causal pathways are complex among di erent locations in nature, since they may be in uenced by geography, atmospheric phenomena and other complex factors.
To handle both challenges outlined above, we propose a deep spatio-temporal hybrid model to estimate air pollutant concentrations. The main contributions of the paper are given as follows: (i) Granger causality is used to model the spatial correlation between di erent stations in adjacent regions, and consider the spatial dependence of air pollutant concentrations between the propagation of air pollution under di erent wind directions in each sub-region by constructing a hyperparametric Gauss vector weight function. (ii) By constructing the Attention-CNN hybrid model, we can e ectively extract the intrinsic features from historical air pollutant concentrations, meteorological and timestamp data by learning over a long time span, and then use LSTM layer to extract temporal information from these feature mappings. (iii) We use air pollutant concentration data and meteorological monitoring data from northern Taiwan in 2015 for research and analysis to evaluate our methods. Abundant experiments prove that the model is superior to traditional machine learning methods.
The rest of the paper is organized as follows: Section 2 mainly introduce the data description, Attention-CNN-LSTM model, spatio-temporal correlation using Granger causality analysis, extraction of spatial and temporal features, attention mechanism in feature map and channel and prediction for air pollution concentration of multiple monitoring stations. Section 3 shows the experimental results. Finally, some concluding remarks and suggestions for future work are in Section 4.
2.1. Data Preprocessing The experimental data in this paper is from the Environmental Protection Administration, Executive Yuan, R.O.C. In the current experiments, air pollutant concentration data and meteorological monitoring data is collected every hour for 25 monitoring sites (shown in Figure 1) from Jan/01/2015 to Dec/31/2016. The meteorological data and air quality data used for CO and N O concentration prediction are shown in Table 1.To mitigate the negative impact of missing values on data analysis performance, we delete the timestamp to eliminate missing values, because lling in missing values requires accurate prediction of spatial and temporal correlation between di erent time series. If lling in missing time series data is not representative, temporal autocorrelation and spatial correlation may not be strong. In all the experiments, the data is divided into the training set (80%) and testing set (20%).
The framework of the proposed spatial-temporal prediction model for multi-scale pollutant concentrations is shown in Figure 2. The main inputs (historical CO/N O concentration data) are included in brown box, and auxiliary inputs (meteorological data, related pollutant concentration and the time of day) are included in light blue box; r represents the number of time steps used, and the numbers in the parentheses represent the dimensions of each type of feature.
Considering the spatio-temporal correlation between 25 stations and their historical information using GC (Granger causality) analysis, as an index to measure the interaction between time series, has been favored in recent decades. For complex spatial factors, we use GC to analyze the correlation between the air concentration time series. We de ne the time series of air pollutants at two monitoring sites as Yi and Xi respectively. The formula of GC and the null hypothesis are given as follows:
n Yi(t) = X j=1 i(j)Yi(t
n Yi(t) = X j=1
n j) + X
j=1 i(j)Yi(t i(j)Xi(t
j) + t; if i 2 Nd j) + t; if i 2= Nd (1) (2) where Nd is the neighborhood set of the spatial clustering (The K-Means algorithm is adopted to gather them); t is a white noise Gaussian random vector; n is the number of time stamps; vector i is the correspondent weights for Yi; vector i represents the spatial weight between spatial locations Yi and Xi.
In the section of Spatial-temporal feature extraction, two or more CNN layers are selected
to extract the intrinsic features from historical air pollutant data for long-term span learning,
and then the one-hot encoding is used. The method encodes the hourly data and combines the
extracted features with current meteorological data and related pollutant data to improve the
predictive performance. In addition, we add batch normalization (BN) after the second and
third convolutional layers of the model. Considering that the scaled exponential linear units
(SELU) function has better convergence performance and can e ectively avoid the gradient
disappearance problem, it is taken as activation function in this paper [
In the section of Feature map attention, We adopted an attention mechanism to weigh the hidden features to enhance their validity. In the attention mechanism, F = ff (1); f (2); :::; f (j)g is the output hidden feature maps of the convolutional layer, where j 2 R is the number of the convolution kernel. The weighted feature maps F 0 are computed by softmax : (3) (4) (5) (6) (7) (8) (9) F 0 = W
F f 0(i) = !i f (i) !i = softmax(F ) =
exp(f (i)) Pit=1 exp(f (t)) here W = f!1; !2; :::; !j g is a weight matrix and its size is the same as that of the feature maps. To generate W , the attention mechanism consists of 3 convolution layers with the stride 1. The rst convolution layer has k s lters with convolution kernel size 5 5, the second and third layers have k lters with convolution kernel size 3 3, and the number of lters is 100 for each convolutional layer.
By stacking several layers of LSTM, the features in spatially correlated contaminant data with long-term dependence can be automatically extracted layer by layer, and the fused features can be used to compute multiscale time series prediction of air pollutants. For the LSTM layer, one input is the temporal information of X = (x1; x2; :::; xt), and another input is the hidden unit ht 1 from the last time step. The forward training process of Attention-CNN-LSTM can be expressed by the following equations: ft = (Wf [ht 1; xt] + bf ) it = (Wi [ht 1; xt] + bi) Ct = ft
Ct 1 + it tanh(WC [ht 1; xt] + bC ) ot = (Wo [ht 1; xt] + bo)
ht = ot tanh(Ct) where it, ot, and ft denote the activation of the input gate, output gate and forget gate, respectively; Ct and ht denote the activation vector for each cell and memory block, respectively; and W and b denote the weight matrix and bias vector. The output from the last step of the LSTM is then fed to the fully connected layer for spatio-temporal air pollution prediction.
3.1. Performance metric and settings
Each monitoring station collects air quality data once an hour, and the dataset contains more
than 400000 instances, each with concentrations of CO and N O. To prove the e ectiveness of the
proposed Attention-CNN-LSTM, several models are used for comparison, including ARIMA [
Obviously, the current state has di erent e ects on di erent time intervals in the future [
As it is shown in the Tables 2,3,4,5, the accuracy of all the models decreases as the prediction time extends. However, for CO concentration, the RMSE standard deviation of Attention-CNN-LSTM, which varies from 0:4602 to 0:5803, is much lower than that for the other models, indicating that Attention-CNN-LSTM achieves higher accuracy and stability in long-term prediction. This is due to the combination of 3D-CNN, attention mechanism and LSTM, which extract advanced spatio-temporal features while maintaining the transmission of state information, rather than using LSTM alone or CNN. Prediction results show that Attention-CNN-LSTM is valid for air pollutant concentration forecasting in total data set.
N O CO
ARIMA SVR CNN LSTM CNN-LSTM Attention-CNN-LSTM ARIMA SVR CNN LSTM CNN-LSTM Attention-CNN-LSTM
In this paper, an attention-based CNN-LSTM model has been proposed to forecast air pollutant concentration. Granger causality analysis is utilized to explore the spatial correlation among di erent monitoring sites and spatial features are added into the prediction model. The model combines ordinary convolution units and attention mechanism to extract the spatial and temporal feature maps. Finally, the air pollutant concentration of multiple monitoring stations is predicted by LSTM, which can learn temporal dependencies on time series of pollutant concentrations. Experimental results have demonstrated that the proposed Attention-CNNLSTM outperforms the other state-of-the-art algorithms in terms of RMSE, MAE, MAPE and R values. In order to further improving the performance of the proposed method, several aspects remain to be investigated in the future work: (1) Exploring the spatio-temporal clustering method based on weather patterns because the air polluting process may be a ected by multiple weather patterns; (2) Exploring multi-faceted causality analysis and environmental factors.
N O CO N O CO
ARIMA SVR CNN LSTM CNN-LSTM Attention-CNN-LSTM ARIMA SVR CNN LSTM CNN-LSTM Attention-CNN-LSTM ARIMA SVR CNN LSTM CNN-LSTM Attention-CNN-LSTM ARIMA SVR CNN LSTM CNN-LSTM Attention-CNN-LSTM
This work was supported in part by the Leading Talents of Science and Technology Innovation in Zhejiang Province 10 Thousands Plan under Grant 2018R52040, in part by the National Key Research and Development Program of China under Grant 2016YFC0201400, in part by the Provincial Key Research and Development Program of Zhejiang Province under Grant 2017C03019, and in part by the International Science and Technology Cooperation Program of Zhejiang Province for Joint Research in High-tech Industry under Grant 2016C54007. N O CO