Spatiotemporal Weather Data Predictions with Shortcut Recurrent-Convolutional Networks: A Solution for the Weather4cast Challenge

Abstract

This paper presents the neural network model that was used by the author in the Weather4cast 2021 Challenge Stage 1, where the objective was to predict the time evolution of satellite-based weather data images. The network is based on an encoder-forecaster architecture making use of gated recurrent units (GRU), residual blocks and a contracting/expanding architecture with shortcuts similar to U-Net. A GRU variant utilizing residual blocks in place of convolutions is also introduced. Example predictions and evaluation metrics for the model are presented. These demonstrate that the model can retain sharp features of the input for the first predictions, while the later predictions become more blurred to reflect the increasing uncertainty.

Keywords

• eol>weather
• satellite data
• neural networks
• gated recurrent units

1. Introduction

ences have been discussed by, e.g., [ 2, 3, 4, 5 ]. Thus, there is considerable interest in further developing ML model Weather forecasting requires the prediction of spatiotem- architectures for these applications. poral changes in atmospheric fields. Most commonly, this To support the development of ML/AI for weather preprediction is carried out using numerical weather predic- diction, the Weather4cast 2021 Challenge had the objection (NWP, e.g. [ 1 ]), which models the evolution of the tive to predict the time evolution of atmospheric satellite atmosphere by computationally evaluating approximate data products using machine-learning methods. This physical laws, such as forward-integrating the partial paper presents the ML solution submitted by the author. diferential equations of fluid dynamics.

Although the NWP approach to forecasting is firmly grounded in first principles, NWP models are compu- 2. Challenge Data and Objectives tationally expensive: state-of-the-art models are run on powerful supercomputers and yet take a significant amount The Weather4cast 2021 Challenge Stage 1 presented the of time to complete a model run, which limits their use- competitors with a data problem of predicting the time fulness in short-term predictions. These issues have evolution of satellite-based measurements of four difermotivated considerable research on using artificial in- ent atmospheric fields: temperature, representing either telligence (AI) and machine learning (ML) methods for the surface temperature (where no cloud is present) or the weather prediction, particularly after the prominent progresscloud top temperature (in cloudy areas); convective rainin convolutional neural networks (ConvNets) in the last fall rate (crr_intensity); probability of tropopause folding decade. ConvNets can learn to recognize complex spatial (asii_turb_trop_prob); and cloud mask (cma). These are structures, which make them well-suited for modeling data products created by the Nowcasting Satellite Appliatmospheric fields. They have been used as components cation Facility (NWC SAF) processing from satellite data of weather models, for example, to speed up computa- from the EUMETSAT-operated MeteoSat satellites. The tionally expensive modules of NWP codes. Recurrent products were given for six regions (numbered R1–R6) neural networks have also been used to model the time in Europe, Northern Africa and the Middle East, each evolution of atmospheric fields. Numerous applications 256 × 256 pixels of approximately 4 km × 4 km each. A and potential uses of ML/AI in weather in climate sci- few other variables also were available such as cloud type, as well as static data with the elevation of the terrain, the latitude and the longitude.

The goal of the challenge was to predict, in each area, the next 32 frames of a sequence given the previous 4 frames, with frames separated by 15 min. Training and validation datasets, with complete 4 + 32-frame samples, as well as a test dataset with only the 4-frame inputs, h t p e D

Input

Output

Downsampling

Upsampling

Recurrent

were provided for regions R1–R3, constituting the “Core” described below. competition. Meanwhile, R4–R6 only had test data avail- The encoder section consists of four recurrent downable, meaning that they had to be evaluated using models sampling stages. Each stage first passes the sequence trained on R1–R3; this was called the “Transfer Learn- through a residual block [ 10 ], with each frame processed ing” competition. Furthermore, all regions had a set of using the same convolutional filters. A strided convo“held-out” data which were made available only during lution in the residual block is used to downsample the the final week of the competition; the final results were input by a factor of 2. Then, the sequence is processed by based on the performance with these data. a gated recurrent unit (GRU) layer [ 11 ]; a tensor of zeros

The performance of the models was evaluated using is passed as the initial state of the GRU. The number of the mean-square error (MSE) for each variable. How- channels in the convolutions is increased with increasing ever, some adjustments were made to the MSE to ac- depth in the encoder. count for the particular needs of each variable, except for The forecaster section is approximately a mirror image crr_intensity. First, the loss for temperature was modified of the encoder section. Each stage consists of a GRU layer to account for varying amounts of missing data in each which is followed by bilinear upsampling and a residual region. Second, asii_turb_trop_prob is a probabilistic vari- block. A shortcut similar to U-Net [ 12 ] is utilized: The able and the output of the model was passed through ifnal state of each GRU in the encoder is passed through a truncated and normalized logit transform before the a convolution and then used as the initial state of the evaluation of the MSE. Third, although cma is techni- GRU of corresponding depth in the forecaster. This alcally evaluated using the MSE, the variable in the output lows the high-resolution features of the recent frames to data file is required to be quantized such that the value be passed through, preventing the first predictions from is either 0 or 1; therefore, model output values < 0.5 are being blurry. A final projection and a sigmoid activarounded to 0 and outputs ≥ 0.5 are rounded to 1 before tion produce the output as a single variable constrained evaluation. The details of the metrics can be found in [ 6 ]. between 0 and 1.

The main diference of the architecture presented here to that of [ 7 ] is that the use of Trajectory GRU (Traj3. Solution GRU) is rejected as TrajGRU was found to cause training instability. Two variants are considered instead. The 3.1. Models ifrst utilizes the Convolutional GRU (ConvGRU) layer adopted by e.g. [ 9, 13, 14 ]. In the second variant, the convolution in the ConvGRU is replaced by a residual block modified to be used for this purpose. The use of the residual block increases the depth of the operations in the GRU and is thus expected to allow it to better process nonlinear transformations and also to increase the The model presented here is a neural network combining recurrent-convolutional layers and shortcut connections in an encoder-forecaster architecture. The architecture is presented in Fig. 1. It is based on that developed in [ 7 ] for precipitation nowcasting and adopted by [ 8 ], as well as similar to that of [ 9 ], with some diferences that are distance at which pixels can influence each other at each modest number of weights in the models by standards of step of the ConvGRU. The latter efect may recover some modern ConvNets, approximately 12.1 million weights of the advantages of TrajGRU over ConvGRU that [ 7 ] in the ConvGRU variant and 18.6 million in the ResGRU found. The author is unaware of previously published variant. The loss over the validation set was used as the instances of a residual layer being used in place of the metric for each variable except cma for which a rounded convolution in GRU. In this paper, this variant is called MSE that takes the 0, 1 quantization into account was “ResGRU”, although the same abbreviation was used for a used. diferent combination of GRUs and residual connections A parallel setup of eight Nvidia Tesla V100 GPUs was in [ 15 ]. used to train the models. Training for one epoch took

The models were implemented using TensorFlow/Keras approximately 20 minutes with this hardware. The eight [ 16 ] version 2.4. The source code and the pre-trained parallel GPUs only provided a speedup of a factor of models can be found through the links in Appendix A. approximately 3 compared to training on a single GPU, suggesting that single-GPU training of the models should 3.2. Training be feasible, although the batch size would likely have to be reduced as the models require rather large amounts of GPU memory.

In practice, condition 3 was never activated as the model continued to achieve marginal gains on the validation data at least every few epochs until the maximum train- The model presented here reached the top of the final ing time of 12 h or 24 h (depending on the training run) leaderboards in both the Core and the Transfer Learning was reached. This suggests that the model did not sufer categories of the Weather4cast 2021 Challenge Stage 1. significantly from overfitting, which typically causes the It is a versatile solution to the problem of predicting the validation loss to start increasing even as the training loss evolution of atmospheric fields, producing sharp predickeeps decreasing. This is perhaps due to the relatively tions for the near term and increasing the uncertainty for

5. Conclusions

Since the scores for the target variables were evaluated independently from each other, a separate instance of the model was trained for each target variable, but using all variables as inputs for each model. The models were trained on the training dataset of R1–R3 such that every available gapless sequence of 36 frames was used for training, resulting in 72192 diferent sequences (albeit with considerable overlap). The training was performed with combined data from all regions R1–R3 in order to increase the training dataset size and improve the ability of the model to generalize; specializing the model to single regions was not attempted. The static data (latitude, longitude and elevation) were also used for training. Data augmentation by random rotation in 90° increments as well as random top-down and left-right mirroring was used to further increase the efective number of training samples. The model for asii_turb_trop_prob was trained using a custom logit loss corresponding to the metric specified in [ 6 ], while the other variables were trained using the standard MSE loss. The Adam optimizer [ 17 ] was used to train the models with a batch size of 32.

The progress of the training was evaluated using the provided validation dataset for R1–R3. After each training epoch, the evaluation metric was computed on the validation set and then: 1. If the metric improved upon the best evaluation result, the model weights were saved. 2. If the metric had not improved in 3 epochs, the learning rate was reduced by a factor of 5. 3. If the metric had not improved in 10 epochs, the training was stopped early.

4. Results

Both the ConvGRU and ResGRU variants of the model were trained for each target variable. The evaluation results for the validation dataset are shown in Table 1. Comparisons to TrajGRU were found impractical as the models using TrajGRU would not converge properly due to the training instability mentioned in Sect. 3.1.

Based on the evaluation results, three submissions were made to the final leaderboards of Weather4cast Stage 1: one using the ConvGRU variant for all variables (codenamed V4c), another using ResGRU (V4rc), and a third using the best model for each variable based on the validation metrics (V4pc). It was indeed this last model that produced the best results also on the leaderboards for both the Core and Transfer Learning competitions, as shown in Table 2.

Figures 2–5 show examples of the predictions using the validation dataset. These are all shown for the same scene except for Fig. 3, where a diferent scene was chosen because the one used for the others did not contain precipitation. It is clear that the predictions start relatively sharp and get blurrier over time, reflecting the increasing uncertainty. The blurriness is likely exacerbated by the use of the MSE metric, specified in the data challenge, which is prone to regression to the mean. Especially in Fig. 4, one can also see that the model can predict the motion of features in the images. T=-2

T=0 0.0 0.2 0.4 0.6 0.8 1.0 longer lead times. The architecture can be easily adapted to other tasks such as probabilistic predictions or outputs that are diferent from the inputs. Further research is needed to handle, for instance, diferent spatial and temporal resolutions of inputs and data available for future time steps.

Acknowledgments

This project benefited from parallel development in the fellowship “Seamless Artificially Intelligent Thunderstorm Nowcasts” from the European Organisation for the Exploitation of Meteorological Satellites (EUMETSAT). The hosting institution of this fellowship is MeteoSwiss in Switzerland. The author thanks U. Hamann and A. Rigazzi for discussions regarding the model and training.

A. Online Resources

The source code with instructions to replicate the results presented in this paper can be found at https://github. com/jleinonen/weather4cast-stage1. The model weights used in the challenge submissions can be downloaded at [ 18 ].

References

[1] P. Bauer , A. Thorpe , G. Brunet, The quiet revolution of numerical weather prediction , Nature 525 ( 2015 ) 47 - 55 . doi:doi.org/10.1038/nature14956.

[2] A. McGovern , K. L. Elmore , D. J. Gagne , II, S. E. Haupt , C. D. Karstens , R. Lagerquist , T. Smith , J. K. Williams , Using artificial intelligence to improve real-time decision-making for high-impact weather , Bull. Amer. Meteor. Soc . 98 ( 2017 ) 2073 - 2090 . doi: 10 .1175/ BAMS-D- 16- 0123 .1.

[3] M. Reichstein , G. Camps-Valls , B. Stevens , M. Jung , J. Denzler , N. Carvalhais , Prabhat, Deep learning and process understanding for data-driven earth system science , Nature 566 ( 2019 ) 195 - 204 . doi: 10 . 1038/s41586-019-0912-1.

[4] C. Huntingford , E. S. Jefers , M. B. Bonsall , H. M. Christensen , T. Lees , H. Yang , Machine learning and artificial intelligence to aid climate change research and preparedness , Environmental Research Letters 14 ( 2019 ) 124007 . doi: 10 .1088/ 1748 -9326/ ab4e55.

[5] S. E. Haupt , W. Chapman , S. V. Adams , C. Kirkwood , J. S. Hosking , N. H. Robinson , S. Lerch , A. C. Subramanian , Towards implementing artificial intelligence post-processing in weather and climate: proposed actions from the oxford 2019 workshop , Philos. Trans. R. Soc. London, Ser. A 379 ( 2021 ) 20200091 . doi: 10 .1098/rsta. 2020 . 0091 .

[6] IARAI , Weather4cast 2021 : Competition metrics , 2021 . URL: https://www.iarai.ac.at/weather4cast/ wp-content/uploads/sites/3/2021/04/w4c.pdf .

[7] X. Shi , Z. Gao , L. Lausen , H. Wang , D.- Y. Yeung , W.-k. Wong, W.-c. WOO, Deep learning for precipitation nowcasting: A benchmark and a new model , in: I. Guyon, U. V. Luxburg , S. Bengio , H. Wallach , R. Fergus , S. Vishwanathan , R. Garnett (Eds.), Advances in Neural Information Processing Systems , volume 30 , Curran Associates , Inc., 2017 . URL: https://proceedings.neurips.cc/paper/2017/file/ a6db4ed04f1621a119799fd3d7545d3d-Paper.pdf .

[8] G. Franch , D. Nerini , M. Pendesini , L. Coviello , G. Jurman, C. Furlanello , Precipitation nowcasting with orographic enhanced stacked generalization: Improving deep learning predictions on extreme events , Atmosphere 11 ( 2020 ). doi: 10 .3390/ atmos11030267.

[9] S. Ravuri , K. Lenc , M. Willson , D. Kangin , R. Lam , P. Mirowski , M. Fitzsimons , M. Athanassiadou , S. Kashem , S. Madge , R. Prudden , A. Mandhane , A. Clark , A. Brock , K. Simonyan , R. Hadsell , N. Robinson , E. Clancy , A. Arribas , S. Mohamed , Skillful precipitation nowcasting using deep generative models of radar , 2021 . arXiv: 2104 . 00954 .

[10] K. He , X. Zhang , S. Ren, J. Sun , Deep residual learning for image recognition , in: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2016 . doi: 10 .1109/CVPR. 2016 . 90 .

[11] K. Cho , B. van Merrienboer , D. Bahdanau , Y. Bengio , On the properties of neural machine translation: Encoder-decoder approaches , in: Proceedings of SSST-8 , Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation , 2014 , pp. 103 - 111 .

[12] O. Ronneberger , P. Fischer , T. Brox , U-net: Convolutional networks for biomedical image segmentation , in: N. Navab , J. Hornegger , W. M. Wells , A. F. Frangi (Eds.), Medical Image Computing and ComputerAssisted Intervention - MICCAI 2015 , 2015 , pp. 234 - 241 . doi: 10 .1007/978-3- 319 -24574-4_ 28 .

[13] L. Tian , X. Li , Y. Ye , P. Xie , Y. Li , A generative adversarial gated recurrent unit model for precipitation nowcasting 17 ( 2020 ) 601 - 605 . doi: 10 .1109/LGRS. 2019 . 2926776 .

[14] J. Leinonen , D. Nerini , A. Berne , Stochastic superresolution for downscaling time-evolving atmospheric fields with a generative adversarial network , IEEE Trans. Geosci. Remote Sens . ( 2020 ). doi: 10 .1109/TGRS. 2020 . 3032790 .

[15] W. Gao , R.-J. Wai , A novel fault identification method for photovoltaic array via convolutional neural network and residual gated recurrent unit , IEEE Access 8 ( 2020 ) 159493 - 159510 . doi: 10 .1109/ ACCESS. 2020 . 3020296 .

[16] F. Chollet , et al., Keras , https://keras.io, 2015 .

[17] D. P. Kingma , J. Ba , Adam: A method for stochastic optimization , in: 3rd International Conference for Learning Representations , San Diego, California, USA, 2014 . URL: https://arxiv.org/abs/1412.6980.

[18] J. Leinonen , Model weights for a Weather4cast 2021 Challenge Stage 1 solution, 2021 . doi: 10 .5281/ zenodo.5101213.