Spatial-temporal Transformer Network with Self-supervised Learning for Trafic Flow Prediction

Abstract

Trafic flow prediction plays a critical role in improving the quality, security, and eficiency of Intelligent Transportation Systems (ITS). Accurate prediction requires modeling spatial and temporal characteristics simultaneously. Existing works usually extract the spatial features by CNN-based modules and temporal features by RNN-based modules. However, the CNNbased modules are locally biased, performing poorly in global spatial dependencies; and the RNN-based modules concentrate on learning the high-level temporal dynamics (e.g., periodicity), and fail to consider the numerical closeness between future data and historical observations as a strong prior knowledge for the prediction. To alleviate these limitations, we propose a Spatial-temporal Transformer Network with Self-supervised Learning (ST-TSNet). ST-TSNet uses a Pre-Conv Block and vision transformer to learn the spatial dependencies in both local and global contexts. Furthermore, a skip connection from the input of historical records to the output prediction is introduced to utilize similar patterns to improve the prediction results. Finally, a self-supervised strategy called stochastic augmentation is proposed to explore spatial-temporal representations from massive trafic data to benefit the prediction task. Experiments on two datasets, TaxiBJ and TaxiNYC, demonstrate the efectiveness of ST-TSNet. The codes is available at https://github.com/pengzhangzhi/spatial-temporal-transformer.

1. Introduction

few abrupt changes, showing many similarities in adjacent frames. As depicted in the time series of Fig.1, the Trafic flow prediction is a build block in Intelligent Trans- ratio of current trafic flow to the previous one (shown portation Systems (ITS), which is essential for providing in blue line) floats up and down within a fixed ratio of high-quality trafic service. An accurate prediction of 1 as the trafic flow (shown in orange line) periodically future trafic flow data depends on modeling the spatial- evolves. This means that adjacent trafic flow snapshots temporal information from the previous observations. have a close value and exhibit similar distribution. Thus, This problem can be considered from the spatial and tem- an intuitive idea is to use historical observations as the poral perspectives. From the spatial perspective, learn- base prediction for future data. Such motivation provides ing the local spatial correlations is essential since trafic a prior knowledge that forces the model to predict the fuvolume is most influenced by its nearest neighbors. How- ture data partially based on the original historical records ever, in real-world scenarios, two distant regions may be instead of completely depending on the extracted tempostrongly correlated in their tracfi distributions as they ral patterns. However, such similarity is overlooked in feature the similar functionality (e.g., transportation hub). existing methods [ 5, 2 ], as they process the historical data Most of existing works [ 1, 2, 3 ] adopt the convolutional for high-order temporal characteristics (e.g., periodicity), layers as their backbone to extract the spatial features, distorting the numerical similarity. which may introduce short-range bias due to their small With the rapid growth of trafic sensors deployed, a receptive field. These methods perform well in extract- massive amount of trafic flow data is collected but not ing local context while hindering in global dependencies. fully utilized. Similarly, in the field of natural language Recently, Vision transformer (ViT) [ 4 ] has shown impres- processing (NLP), TB-level unlabel corpus are collected sive performance in computer vision, due to its innate but relatively fewer label data is available for various power at extracting non-local features. We are motivated language tasks. The gap , however, in NLP is successfully to apply ViT to learn the long-range spatial dependencies. alleviated by self-supervised learning [ 6 ], where unlabel

From the temporal perspective, many works have been data are utilized to learn language representations and proposed to extract complex temporal patterns, e.g., daily then transferred to facilitate downstream tasks. While and weekly periodicity [ 1, 2 ]. However, we argue that in the field of trafic flow prediction, current training a simple temporal characteristic: temporal similarity is algorithms are supervised learning, where the historioverlooked. Trafic flow data are generally smooth with cal records are regarded as input and the trafic data in STRL’22: First International Workshop on Spatio-Temporal Reasoning the next timestamp is served as label. No efective unand Learning, July 24, 2022, Vienna, Austria supervised learning algorithms are proposed to learn * Corresponding author. spatial-temporal representations to facilitate the trafic $ [e-mail] (Z. Peng); [e-mail] lfow prediction task. (X. Huang©)2022 Copyright for this paper by its authors. Use permitted under Creative Commons License Driven by these analyses, we propose a novel frameCPWrEooUrckReshdoinpgs IhStpN:/c1e6u1r3-w-0s.o7r3g ACttEribUutRion W4.0oInrtekrnsahtioonpal (PCCroBYce4.0e).dings (CEUR-WS.org) work called Spatial-temporal Transformer Network with Skip Connection

Fusion

ViT Pre-Conv Block

−2

ViT Pre-Conv

Block concatenate

−

−6 … −1 −3

Past Pre-training strategy

Present

Conv3 Conv1

MSEloss

ViT

Traffic flow map Future

Time

2. Related Work

Self-supervised Learning (ST-TSNet). ST-TSNet consists

of a Pre-Conv Block and ViT for learning spatial correlations in both local and global contexts. In addition, Trafic Flow Prediction . There are two types of flow we directly connect the historical data to the output to data in the trafic flow prediction task: grid-like raster make full use of the historical data as the base predictions. data and graph data and thus two distinct paradigms are Lastly, a self-supervised task named stochastic augmen- derived for the two types of data [ 7 ]. In our work, we tation is proposed to pre-train our ST-TSNet to learn focus on raster data. Existing mainstream trafic predicspatial-temporal representations and fine-tune them to tion methods for raster data fall into one of the following benefit the prediction task. classes: statistical methods or deep learning methods.

The contributions of this work are summarized as fol- Statistical methods include auto-regressive integrated lows. moving average (ARIMA) [ 8 ], Kalman filtering [ 9 ] and historical average. These methods often require strong • We propose a novel framework Spatial-temporal and trivial theoretical assumptions, which may violate Transformer Network with Self-supervised Learn- the nonlinearity of trafic flow data, thus having poor ing (ST-TSNet) to capture spatial-temporal fea- performance in the real world. Recent advances have tures. witnessed the impressive capacity of deep learning to • We employ a simple yet efective skip connection extract nonlinear features from big data [ 10 ]. Many restrategy, plugged into ST-TSNet, to make full use searchers are inspired to apply deep learning to handle of the temporal similarities in trafic flow data. trafic flow prediction task. Existing deep learning meth• We introduce self-supervised learning to our ods are based on convolutional neural networks (CNNs) framework and design a pre-training task and recurrent neural networks (RNNs) [ 11 ]. ST-ResNet called stochastic augmentation to explore spatial- [ 1 ] first employs the CNNs with residual connections to temporal features to boost trafic flow prediction learn the spatial dependencies and construct historical task. data into diferent branches according to the temporal • We conduct extensive experiments on two bench- semantics to learn temporal features. Similar ideas are marks (TaxiBJ and TaxiNYC) to evaluate the ef- adopted by subsequent works [ 2, 12 ] in which 3D convofectiveness of our methods and the results show lution is used to learn the spatial-temporal dependencies. that our ST-TSNet outperforms state-of-the-art Moreover, RNN-based models [ 13, 14 ] are inspired to methods. use convolutional layer to capture spatial features and sequential hierarchy (e.g., LSTM and GRU) to extract outflow, and H and W denote the number of rows and temporal patterns. However, these methods are time- columns of the grid map, respectively. The purpose of consuming as they make predictions step by step and trafic flow prediction is to predict given historical may sufer gradient vanishing or explosion when captur- trafic flow records ℎ = {x | = 0, . . . , − 1}. As ing long-range sequences [ 15 ]. To alleviate the problems, shown in Fig.1, the historical data is summarized into [ 15, 16 ] discard the recurrent chain structure and employ two categories in the time axis: Closeness sequence Multiplicative Cascade Unit (CMU) with autoencoders = {− 1, − 2, · · · , − (− 1), − } ∈ while preserving the convolutional layers for learning R2× × × is a concatenation of recent hisspatial features. The methods used by existing works can torical data where is the length of closebe considered from spatial and sequential perspectives. ness sequence. Trend sequence = Fmraoimnstthreeasmpa,tiinacllpuedrisnpge2cDtivaen,dco3nDvoclountivoonlualtiloany.erFsroamre tthhee {R2×−××,is− a 2c·oncate, n·· · atio,no−f pe·riodic}historica∈l sequential perspective, there are many choices, including data from the past few weeks, where is the length of RNN, GRU, LSTM and CMU. Most existing works are trend sequence, is the number of intervals within a combination of these methods. In summary, existing a week. methods that based on CNNs sufer from short-range bias as the small receptive field limits their capacity to 3.2. Spatial-temporal Transformer extract global dependencies.

Self-supervised Learning. Self-supervised learning Network is a great way to extract training signals from massive Overall, we employ a symmetric structure for handling amounts of unlabelled data and to learn general represen- the trend data and and the closeness data : tation to facilitate downstream tasks which the labelled a Pre-Conv Block followed by a ViT with two shortcuts data are limited. To generate supervision information (i.e., two blue lines shown in Fig.1) from the input to the from data, a general strategy is to define pre-training fusion layer. In the end, fusion layer adaptively merges tasks for models [ 17, 18 ] to learn semantic representa- four components (two residual components ˆ and ˆ, tions, and then transfer them to downstream tasks to two outputs ˆ and ˆ) to generate prediction improve performance and robustness. Many works in ˆ. computer vision have defined various tasks based on Pre-Conv Block. The trafic flow in a region is highly heuristic methods[ 19, 20 ]. For example, [ 21 ] learns vi- relevant to its nearby regions. We design a Pre-Conv sual representations by predicting the image rotations. Block for capturing such short-range dependencies. As In natural language processing (NLP), masked language illustrated in Fig.1, Conv1 and Conv2 are the main conmodeling, e.g., Bert [ 6 ] have shown to be excellent for volutional layers to capture short-range dependencies. pre-training language models. These methods mask a Thus, we employ a small kernel size ( i.e., 3 × 3) which portion of the input sequence and train models to predict leads to the receptive field of 5. Such design ensures the the missing content with the rest. Such methods are efec- Pre-Conv Block only captures the local dependencies at tive for learning semantic correlations of elements within most in 5 × 5 regions. The short-range dependencies a sequence, e.g., sentence. The trafic flow data can also are well-captured by the Pre-Conv Block while leaving be viewed as a sequence temporally, while the efective- the long-range features to the vision transformer. Inness of self-supervised learning remains unexplored in serting CNNs before ViT has shown to be efective in trafic flow prediction task. strengthening the capacity of ViT [ 22 ]. Conv3 is the residual shortcut, employing 64 kernels with size 1 × 1, 3. Methods which adds up to the main branch as a residual component. Generally, we will use much more kernels (e.g., 64) than that in Conv4. By enlarging and then reducing the 3.1. Problem Formulation number of channels, Pre-Conv Block can learn various We partition a city into an image-like grid map according spatial-temporal dependencies and then refine them into the longitude and latitude, as shown in the trafic flow a compact feature map. map of Fig.1, where each grid denotes a region. The Vision transformer. We apply vision transformer (ViT) value of a grid denotes the trafic flow (inflow or outflow). [ 4 ] after the Pre-Conv Block to capture the global depenThe device deployed at a region will periodically record dencies, as shown in the right of Fig.1. ViT is comprised of the number of people arriving at and departing from two main components: “Patchify” stem and transformer the location to collect the inflow and outflow. The encoder. “Patchify” stem spatially splits the input feature trafic flow map of the entire city at time is noted map into non-overlap × patches and linearly projects as xt ∈ R2× × , where 2 refers to the inflow and patches into tokens. Each token contains the information of a patch of regions. Then the tokens are fused with learnable positional encoding to preserve the 2D positional information and fed into transformer encoder. The encoder utilizes a multi-head self-attention mechanism to model the long-range dependencies followed by a layer normalization and residual connection (Add & Norm) to the next sub-layer, where a Feed Forward Network (FFN) and another Add & Norm are employed to further process the tokens. Lastly, the tokens are averaged and then linearly transformed to generating output:ˆ and ˆ.

Skip Connection. Skip Connection are employed to pre-trained data (Ω , ) transfer similar patterns from the historical observations predictions ˆ ← (Ω) to the output as the base prediction. To preserve the origi- loss ← (ˆ, ) nal similar patterns in historical data, we directly connect () input and to the fusion layer, as shown in the blue update line of Fig.1. Before connecting, we aggregate histori- until stop criteria is met; cal input data in the time dimension to match the shape.

For two historical sequences ∈ R2× × × and ∈ R2× × × , we compute:

Algorithm 1 The pre-training procedure with stochastic

augmentation.

Input: MASA model: , closeness data: , trend

data: , and predicted future data: .

Output: pre-trained MASA model. repeat ← ∪ ∪ target ← ()

Remaining snapshots Ω ← − the rest to predict the target. Such scheme can be expanded to three cases: (1) if the last frame is selected as the target, then this is similar to supervised training, the ˆ = () ∈ R2× 1× × , (1) historical records are used to predict future data; (2) if the ˆ = () ∈ R2× 1× × , (2) earliest frame is the target, then future observations are used to predict the past frame, as shown in the green axis where ˆ and ˆ are the two residual compo- of Fig.1; (3) if any intermediate frame is selected as the nents. (· ) is an aggregation function R2× × × → target, then the historical data and future observations R2× 1× × , where denotes the length of historical are used to predict present, as shown in the red axis of data sequence. Here we use a summation function. Fi- Fig.1. Diferent from the downstream prediction task, nally, the two residual components will be fused in the where input historical records and future data are paired fusion layer. to be the training samples, our stochastic augmentation Fusion Layer. The degree of influence of the four compo- produces several times more samples for pretraining by nents (i.e., two outputs ˆ, ˆ and two residual randomly constructing input-target pairs. For example, components ˆ, ˆ) is diferent, and the influence given a group of five frames, the supervised learning only in diferent regions also varies. Therefore, to dynami- gives one training sample as stated in case (1). While our cally calibrate their contributions, we follow [ 23 ] to use stochastic augmentation paradigm yields five pretraining a parametric-matrix-based fusion method, where the pa- samples (every frame in the group is selected to be the rameter matrices are learned from historical data. For- target once), five times more samples than supervised mally, training. With the large amount of pretraining samples, our models can explore useful spatial-temporal represenˆ = · ˆ + · ˆ+ tations for the downstream prediction task. Specifically (3) for the trafic flow prediction task, we define the group · ˆ + · ˆ, as the union of closeness data, trend data, and predicted where · denotes element-wise multiplication, is the learnable parameter that measures the influence of each component.

3.3. Self-supervised Learning with

Stochastic Augmentation

Our stochastic augmentation aims to pretrain our model to learn general spatial-temporal features to facilitate the prediction task. The pretraining strategy is conceptually simple: we select a group of continuous trafic frames, randomly sample a frame as the predicted target and use ground truth: = ∩ ∩ . Then we randomly sample one snapshot as the target and the rest data Ω = − as the input, constructing pre-training data (Ω , ) to pre-train our model. The algorithm is depicted in Alg.1.

4. Experiments

4.1. Dataset and Evaluation

Dataset. Our experiments are based on two trafic flow

datasets: TaxiBJ and TaxiNYC. Additional external data

• TaxiBJ [ 23 ]: TaxiBJ is a citywide crowd flow dataset collected every half hour in Beijing. Based on the geographic area of Beijing, we partition the Beijing city into 32 × 32 regions. • TaxiNYC [ 16 ]: TaxiNYC is the taxi trip record dataset collected every one hour in New York City. New York City is divided into 16 × 8 regions based on the longitude and latitude1. predicted target back to the original value. We split the last 28 days as the test set for both datasets, and the remaining are regarded as training data. During training, we select 90% of the training data for training models and the remaining 10% is the validation set to early-stop our training algorithm. Our model is implemented and trained by PyTorch. We use Adam [ 25 ] as the optimizer with a learning rate of 0.001 for TaxiBJ and 0.005 for TaxiNYC. Cosine learning rate decay is employed to adjust the learning rate at each iteration. The batch size is 128 for both TaxiBJ and TaxiNYC. We run our model for 600 epochs on TaxiBJ and 800 epochs on TaxiNYC. Our ViT has two blocks, and the patch size is set to (8, 8); the token dimension is set to 128; the number of attention heads is 2; the size of FFN is 512.

1The raw records are available at the NYC government website. A

processed version for experiments is available at github

Evaluation Metric. Three metrics: Rooted Mean

Square Error (RMSE), Mean Absolute Percentage Error 4.3. Quantitative Comparison (MAPE), and Absolute Percentage Error (APE) are used to evaluate our proposed method. We follow previous Table 1 shows the comparing results against the stateworks that compute the metrics on trafic flow value of-the-art methods. We compare our ST-TSNet with the that is larger than 10 to ensure a fair comparison. We following baselines: HA, ST-ResNet [ 23 ], MST3D [ 12 ], conducted experiments ten times for reliable results and ST-3DNet [ 2 ], 3D-CLoST [ 14 ], STAR [ 24 ], and PredCNN presented the means and standard variances of the re- [ 15 ]. The results of the baselines are from [ 16 ]. sults. On TaxiBJ, our method exceeds the SOTA STREEDNet in terms of MAPE and APE and achieves comparable 4.2. Implementation Details results in RMSE. While on TaxiNYC, our method significantly outperforms the SOTA ST-ResNet across all Min-Max normalization is applied in our experiments metrics by a fair margin (1.53 RMSE, 4.61 MAPE, and to scale the data to range [ − 1, 1 ] and denormalize the 1.35E+05 APE improvement).

ST-TSNet has a more significant performance improvement on TaxiNYC than TaxiBJ. The possible reason of (a) Predictions on consecutive time (b) A prediction sample for each method (e) Absolute error of the above predictions (d) Residual

weight (c) Self-attention map the improvement is that the amount of data of TaxiNYC the predictions of each method at diferent time intervals. is twice that of TaxiBJ (41,856 vs. 20,016), which sig- The magnified subplot reveals that our method has better nificantly facilitates the pre-training. This result prove accuracy in predicting the peak. Fig.2 (b) spatially visuthe efectiveness of the self-supervised learning module alizes a prediction sample of each method, and Fig.2 (e) proposed in our method. STREED-Net and STAR have displays the absolute errors of these predictions, demonimpressive performance on TaxiBJ against other base- strating that our ST-TSNet has lower prediction errors lines due to the simple single-branch design. However, than baselines. Fig.2 (c) shows the self-attention map for such simple architecture performs worse than ours in four reference patches. The visualizations are produced a larger dataset TaxiNYC (1.88 RMSE higher than our by attention scores computed via query-key product in ST-TSNet) as there are rich spatial-temporal information the ViT. We use the pentagram-marked regions as query, that a single-branch structure can not extract efectively. and show which patch (region) they attend to. The four Although STREED-Net and PredCNN both introduce cas- corner patches usually attend to remote regions (brighter cading hierarchical structure in their backbone, STREED- color meaning higher attention scores) while caring less Net has better performance than PredCNN. The reason about their neighbors. The reason is that the short-range is that STREED-Net additionally introduces channel and features are perfectly captured and encoded into tokens spatial attention mechanisms to dynamically refine the by Pre-Conv Block, resulting in the ViT focusing more on learned features to generate predictions. Nevertheless, the long-range features. Fig.2 (d) visualizes the weights of the cascading hierarchical structure still sufers from inflow and outflow of two residual components. Combinshort-range bias as it only allows distant snapshots to ing the ground truth in Fig.2 (c), we observe that although interact at higher layers. ST-ResNet, STAR, and Pred- the weights vary in diferent regions and difer from inCNN introduce a 2D convolutional layer, and MST3D, flow to outflow, they tend to concentrate on the regions ST-3DNet, and 3D-CloST employ 3D convolution. The with higher trafic flow. The reason is that these regions 3D convolutional layer is better than the 2D counterparts show a more regular time series, having more similar as it can additionally capture temporal features, while patterns in residual components. 2D convolutions are restricted to only capture spatial features. However, they all sufer from short-range bias 4.5. Ablation Study due to the small receptive field of convolution. Moreover, they do not introduce the skip connection and any additional pre-training strategies, resulting in inferior performance.

To verify the efectiveness of proposed methods, we design a list of variants by appending modules step by step and comparing them on TaxiBJ and TaxiNYC. The basic variant is Vision transformer (ViT). We separately append skip connection (SC), Pre-Conv Block (PC), and stochastic argumentation pre-training (SA) to ViT to construct other variants. We further consider the external factors on our ST-TSNet (ST-TSNet (w Ext)). We use an external

4.4. Qualitative Analysis

We ofer four intuitive visualizations of proposed meth

ods to explain their behaviors in Fig.2. Fig.2 (a) compares module (two-layer multilayer perceptron) to model the external features according to [ 5 ]. The external data is transformed and added together with the main output to yield prediction.

The results in Table 2 show that: 1) the full version of the our methods (i.e., ST-TSNet (w Ext)) achieves the best performance. 2) Adding each module step by step will progressively improve the performance. It suggests that each module is an indispensable component for our ST-TSNet.

We additionally study the strategy of the skip connection by introducing a new residual component: the Pre-Conv Block output . We investigate two connection strategies: additionally and solely connect to the fusion layer. Results show that the two strategies degrade performance (1.66 and 1.35 RMSE degradation), suggesting that the is harmful for prediction. The performance degradation may be caused by the convolutional operations in Pre-Conv Block disrupt the semantic information in historical data (e.g., trafic distributions), resulting in the and predicted target have diferent distributions. In contrast, the historical records ( and ) and the predicted target are collected from the same distribution and temporally correlated. Thus the historical records share similar patterns with the predicted target that can directly contribute to the prediction, while the confuses the model.

5. Conclusion

In this paper, we present a novel trafic prediction frame

work, spatial-temporal Transformer Network with Selfsupervised Learning (ST-TSNet) for learning spatialtemporal features. ST-TSNet is equipped with Pre-Conv Block and ViT to capture local and spatial dependencies. In addition, we observe the similarity in trafic flow data, which enables us to take advantage of the historical data as the base prediction for the future. Finally, we propose a pretext task named stochastic argumentation to enable models to further explore spatial-temporal representations under limited data. Experiments on two datasets demonstrate the superiority of our proposed methods.

Acknowledgments

This research was funded by the National Natural Sci

ence Foundation of China under Grant No.62062033, and the Natural Science Foundation of Jiangxi Province under Grant No.20212BAB202008. Zhangzhi, in particular, would like to thank his father Jianhua Peng and mother Changmei Zhang for countless love and support during the work was developed. I love you all.

References

[1] J. Zhang , Y. Zheng , D. Qi , Deep spatio-temporal residual networks for citywide crowd flows prediction , in: Proc. of AAAI , 2017 .

[2] S. Guo , Y. Lin , S. Li , Z. Chen , H. Wan , Deep spatial-temporal 3d convolutional neural networks for trafic data forecasting , IEEE Transactions on Intelligent Transportation Systems ( 2019 ).

[3] S. Fang , Q. Zhang , G. Meng, S. Xiang , C. Pan , Gstnet: Global spatial-temporal network for trafic flow prediction , in: Proc. of IJCAI , 2019 .

[4] A. Dosovitskiy , L. Beyer , A. Kolesnikov , D. Weissenborn , X. Zhai , T. Unterthiner , M. Dehghani , M. Minderer , G. Heigold, S. Gelly , J. Uszkoreit , N. Houlsby , An image is worth 16x16 words: Transformers for image recognition at scale , in: Proc. of ICLR , 2021 .

[5] J. Zhang , Y. Zheng , D. Qi , R. Li , X. Yi , Dnn-based prediction model for spatio-temporal data , Proceedings of the 24th ACM SIGSPATIAL International Conference on Advances in Geographic Information Systems ( 2016 ).

[6] J. Devlin , M.- W. Chang , K. Lee , K. Toutanova , BERT: Pre-training of deep bidirectional transformers for language understanding , in: Proc. of ACL , 2019 .

[7] X. Yin , G. Wu, J. Wei , Y. Shen , H. Qi , B. Yin , Deep learning on trafic prediction: Methods, analysis and future directions , IEEE Transactions on Intelligent Transportation Systems ( 2021 ).

[8] B. M. Williams , L. A. Hoel , Modeling and forecasting vehicular trafic flow as a seasonal arima process: Theoretical basis and empirical results , Journal of Transportation Engineering-asce ( 2003 ).

[9] J. hua Guo , W. Huang , B. M. Williams , Adaptive kalman filter approach for stochastic short-term trafic flow rate prediction and uncertainty quantification , Transportation Research Part C-emerging Technologies ( 2014 ).

[10] R. Salakhutdinov , Deep learning , in: Proc. of KDD , 2014 .

[11] Y. Lv , Y. Duan , W. Kang , Z. Li , F. Wang , Trafic flow prediction with big data: A deep learning approach , IEEE Transactions on Intelligent Transportation Systems ( 2015 ).

[12] C. Chen , K. Li , S. Teo , G. Chen , X. Zou , X. Yang , R. Vijay , J. Feng , Z. Zeng , Exploiting spatiotemporal correlations with multiple 3d convolutional neural networks for citywide vehicle flow prediction , 2018 IEEE International Conference on Data Mining (ICDM) ( 2018 ).

[13] Z. Zhao , W. Chen , X. Wu , P. C. Y. Chen , J. Liu, Lstm network: a deep learning approach for short-term trafic forecast , Iet Intelligent Transport Systems ( 2017 ).

[14] S. Fiorini , G. Pilotti, M. Ciavotta , A. Maurino , 3dclost: A cnn-lstm approach for mobility dynamics prediction in smart cities , 2020 IEEE International Conference on Big Data (Big Data) ( 2020 ).

[15] Z. Xu , Y. Wang , M. Long , J. Wang , Predcnn: Predictive learning with cascade convolutions , in: Proc. of IJCAI , 2018 .

[16] S. Fiorini , M. Ciavotta , A. Maurino , Listening to the city, attentively: A spatio-temporal attention boosted autoencoder for the short-term flow prediction problem, ArXiv preprint ( 2021 ).

[17] R. Zhang , P. Isola , A. A. Efros , Colorful image colorization , in: Proc. of ECCV , 2016 .

[18] D. Pathak , P. Krähenbühl , J. Donahue , T. Darrell , A. A. Efros , Context encoders: Feature learning by inpainting , in: Proc. of CVPR , 2016 .

[19] I. Misra , L. van der Maaten , Self-supervised learning of pretext-invariant representations , in: Proc. of CVPR , 2020 .

[20] M. Noroozi , P. Favaro , Unsupervised learning of visual representations by solving jigsaw puzzles , in: Proc. of ECCV , 2016 .

[21] S. Gidaris , P. Singh , N. Komodakis , Unsupervised representation learning by predicting image rotations , in: Proc. of ICLR , 2018 .

[22] A. Hassani , S. Walton , N. Shah , A. Abuduweili , J. Li , H. Shi , Escaping the big data paradigm with compact transformers , ArXiv abs/2104 .05704 ( 2021 ).

[23] J. Zhang , Y. Zheng , D. Qi , R. Li , X. Yi , T. Li , Predicting citywide crowd flows using deep spatiotemporal residual networks , Artif. Intell . ( 2018 ).

[24] H. Wang , H. Su , Star: A concise deep learning framework for citywide human mobility prediction , 2019 20th IEEE International Conference on Mobile Data Management (MDM) ( 2019 ).

[25] D. P. Kingma , J. Ba , Adam: A method for stochastic optimization , in: Proc. of ICLR , 2015 .