Comparison of AI-Based Approaches for Statistical Downscaling of Surface Wind Fields in the North Atlantic Vadim Rezvova,b, Mikhail Krinitskiya , Alexander Gavrikova and Sergey Guleva a Shirshov Institute of Oceanology, Russian Academy of Sciences, Moscow, Russia b Moscow Institute of Physics and Technology (MIPT), Dolgoprudny, Russia Abstract Surface wind is one of the most important physical fields in climate research. Accurate prediction of high-resolution near-surface winds has a wide variety of applications. Statistical downscaling methods obtain high-resolution information about the physical quantity distribution using available low-resolution data. They avoid high-resolution hydrodynamic simulations that are computationally expensive. Deep learning methods are one of the typical examples of the machine learning approaches to complex nonlinear functions approximating. In this work, we consider statistical downscaling of near-surface wind in the North Atlantic. For this, cubic interpolation, various architectures of convolutional networks, and generative adversarial network are applied. Based on the results obtained, the quality of these statistical downscaling methods is compared, and their advantages and disadvantages are identified. Keywords 1 Statistical downscaling, neural networks, North Atlantic, near-surface wind 1. Introduction Climate change is one of the most serious problems of the modern world. It leads to an increase in temperature and changes in local patterns in different seasons caused by local changes in wind speed. General circulation models are used to study the climate system and its changes on a global scale. The results of general circulation models have low resolution and large spatial scale of the computational cells. General circulation models are extremely computationally expensive even for low resolution outputs. It restricts the ability to predict high-resolution climate variables [1]. The low resolution of climate models results and the systematic errors lead to implausible predictions of future climate scenarios, especially for extreme events [2]. The results of general circulation models can be corrected to increase their resolution, using downscaling. Downscaling allows obtaining high-resolution information about physical quantities based on low-resolution modeling data. Such methods can be divided into two large groups: dynamic and statistical ones [3]. In dynamic downscaling, high-resolution numerical models are applied in sub- domains of the area of interest, and the results of the coarser model are used as boundary conditions for high-resolution modeling in the sub-domains [4-5]. This approach significantly reduces computational costs, since high-resolution modeling is not carried out simultaneously in the entire workspace. Statistical downscaling avoids high-resolution numerical simulations altogether. In this group of methods, the values of physical quantities obtained as a result of a low-resolution numerical simulation are inputs of a certain function. The outputs are local values of the same variables. The functional relationship between low and high resolution data is approximated by training statistical models on a set of data pairs. VI International Conference Information Technologies and High-Performance Computing (ITHPC-2021), September 14–16, 2021, Khabarovsk, Russia EMAIL: rezvov.vyu@phystech.edu ORCID: 0000-0003-1470-647X (A. 1); 0000-0001-5943-0695 (A. 2); 0000-0002-4198-4400 (A. 3) - ©️ 2021 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). CEUR Workshop Proceedings (CEUR-WS.org) 129 Subsynoptic and mesoscale atmospheric dynamics over the North Atlantic is considerably interesting for understanding the mechanisms of highly localized precipitation, heat and moisture transport. Analysis of intensity and characteristic trajectories of hurricanes in the North Atlantic is an integral part of the quantitative assessment of the impact of highly variable atmospheric processes on cyclonic activity and fluxes during ocean-atmosphere interaction [6]. Extremely strong turbulent surface heat and momentum fluxes associated with the above atmospheric processes are highly localized in space and time and require high temporal and spatial resolution for their adequate study [7]. One of the most important questions is whether modern deep learning methods are better for solving the problem of statistical downscaling compared to traditional statistical methods. Recent advances in deep neural networks have led to a significant increase in statistical downscaling techniques. At the moment, the prospect of moving away from numerical downscaling methods based on physical equations towards statistical methods is unclear. In addition, emerging artificial neural network architectures, including convolutional neural networks and GANs capable of solving the downscaling problem, are trained first on standard publicly available sets of photographs and images. Therefore, a certain gap arises in understanding how applicable the models trained on such data to real fields of climate variables. This paper discusses the statistical downscaling of the surface wind speed and direction in the North Atlantic. Model exploration analyzes whether the quality of downscaling improves with increasing depth of convolutional models or when an adversarial learning process is used. In this work, two-dimensional cubic interpolation of the wind from a low-resolution grid to a high-resolution grid is chosen as a reference solution for comparing various methods. Thus, the purpose of this work is to compare the capabilities of existing approaches of statistical downscaling, which are different degrees of complexity and depth of artificial neural networks, in relation to the fields of the surface wind and, in general, to identify the advantages and disadvantages of neural network methods in solving this problem. 2. Materials and methods 2.1. Initial data Long-term atmospheric reanalysis performed using high-resolution model configurations for the North Atlantic is provided by a retrospective dynamic model NAAD [8]. The result is atmospheric fields with a high resolution (14 km) for the North Atlantic region. Area of modelling covers the North Atlantic region from 10⁰N up to 80⁰N and from 90⁰W up to 5⁰E. The center of the area is located at the point with coordinates 45⁰N, 45⁰W. The main NAAD experiment in which high- resolution calculations were performed is HiRes. In this experiment, the work area is a regular grid of 110 X 110 nodes. The spacing between the nodes of the regular grid in HiRes is approximately 14 km. The lower near-surface level is 10–12 m above the ocean surface. In addition to the HiRes experiment, the NAAD model also ran a moderately low resolution experiment LoRes. The calculations in this experiment are carried out on a regular grid of 550 X 550 nodes, the distance between which is 77 km. All experiments of the NAAD model were conducted over a 38-year period from January 1979 to December 2016 with three-hour time resolution. Both the low-resolution input variables and the high-resolution target variables are composed of two orthogonal horizontal components of wind speed at 10-12 m above the surface and sea-level atmospheric pressure 2.2. Methods The first downscaling method used in this work is cubic interpolation from a low-resolution to a high-resolution grid. The results of all other approaches associated with the use of artificial neural networks will be compared with the result of this method, chosen as the reference solution. The recent publications on the use of artificial neural networks for statistical downscaling of climate variables described the use of convolutional neural networks. Convolutional neural networks 130 are quite effective for tasks with spatially distributed data, given in the form of arrays on regular grids, which is often used in climatology and meteorology. The simplest artificial neural network studied in this work is a linear convolutional neural network. The idea for this model was taken from the LinearCNN architecture proposed in [9]. Linear convolutional network allows to associate low-resolution input with high-resolution data using only convolutional layers without the use of non-linear activation functions. Increasing the depth of the convolutional neural network improves the quality of prediction. However, such an increase in the number of convolutional layers can lead to instability in the training of the model as a result of problems that arise during optimization by the backpropagation algorithm. Batch normalization has been proposed to stabilize the training of networks. Another effective way to solve the problem of learning instability is to include connections allowing the output of earlier network layers to be passed to a later stage directly, bypassing the intermediate layers of the model. One example of such connections is residual connections. Thus, we propose to combine the advantages of using deeper neural networks, batch normalization and residual connections. As a result, in this paper we investigate a residual convolutional neural network based on the EnhanceNet model [9]. Another example of a solution to the problem of learning instability in deep neural networks is skip connections. That’s why we also investigate U-Net-based convolutional neural network with skip connections. Unlike all the neural network models described above, which are separate convolutional neural networks, training adversarial networks is an adversarial process in which two models are simultaneously trained. In adversarial network architecture, the generative model, or generator, is opposed to its adversary, the discriminative model, or discriminator, which learns to determine whether a sample is taken from the distribution generated by the generator or from the true distribution of the data. In particular, if the generative and discriminative models are artificial neural networks, then the network as a whole is called a generative adversarial network. In this case, both models are trained using an error backpropagation algorithm. In this paper, we investigate a network model based on SRGAN [10], which is a generative adversarial one. 2.3. Quality metrics To measure the error between scaled and true values of climate variables at high resolution, various loss functions are considered to highlight some aspect of the accuracy of the scaling. For optimization purposes, in this work, spatially averaged loss functions are used, and to assess the quality, metrics are used that take into account both average and local deviations of the model output from the true values. To compare the various models studied in this paper in terms of the quality of scaling, various quality metrics are considered that assess the degree of deviation between the target values of the variables and the result of the models. One of the most important indicators of the scaling quality is the accuracy of determining the absolute value of the wind speed by the model. Therefore, as the simplest metric of the scaling quality in this work, we use the root mean squared error (RMSE) of determining the wind speed. The introduction of the RMSE-95 metric makes it possible to evaluate the scaling quality of extremely high wind speed values. The peak signal-to-noise ratio (PSNR) tends to infinity as the mean square error MSE (“noise”) approaches zero. Since the goal of training the neural network is to minimize MSE, a higher PSNR value may indicate a higher image quality. As the maximum value of the variables (“signal”) increases, which indicates that higher values appear in the scaled data, the peak signal-to-noise ratio also increases. An increase in the “signal”, and, consequently, an increase in PSNR, may mean a weakening of data smoothing, which is necessary for a better visual perception of the scaled fields of climatic variables. 3. Results For all neural network models studied in this work, the same training and validation sets are used. The data describing the fields of atmospheric variables are quite strongly correlated on time scales of 131 the order of several days. In addition, seasonal recurrence of atmospheric processes is observed, leading to correlations on scales of an integer number of years [9]. To correctly take into account such a feature of a given time series, both the training and validation samples consist of a certain number of full consecutive years. The presence of an integer number of seasonal cycles in the training sample also makes it possible to average seasonal variability, which can affect the learning outcome. All models, including cubic interpolation, convolutional neural networks, and the generative adversarial network, were trained and evaluated using the capabilities of the PyTorch machine learning framework for the Python language. A general overview of the performance of the studied models is presented in the Table 1. The comparison is carried out according to the number of trained parameters of the model and memory consumption. All calculations were performed on GPU with 32 GB memory. Table 1 Model performance comparison Model Number of parameters Memory consumption, Mb Linear CNN 11,328 0,01 Residual CNN 1,334,475 5,1 CNN with skip connections 72,127,620 275,2 GAN 1,361,994 (generator) 5,2 (generator) 5,215,425 (discriminator) 19,9 (discriminator) Deeper nonlinear models consist of significant number of convolutional layers and therefore require more memory to store the trained parameters. Thus, the overall memory consumption increases as the model becomes more complex. The smallest number of parameters is trained in a linear convolutional network, which obviously follows from its simplest two-layer architecture. Since the architectures of the residual convolutional network and the generator of the generating adversarial network are similar in depth, the number of trained parameters is practically the same. The generative network discriminator contains 5 times more parameters than the generator. Despite the fact that the total number of parameters of the most complex model - a convolutional network with skip connections - is almost 11 times more than the total number of parameters in the generator and discriminator of the generating network, the generating adversarial network shows visually much better results. Comparison of the values of quality metrics shown by all investigated methods in this work on the validation set is given in the Table 2. Table 2 Downscaling quality comparison Method RMSE, m/s RMSE-95, m/s PSNR Cubic interpolation 1.44 1.90 35.16 Linear CNN 2.85 5.32 27.68 Residual CNN 1.42 2.21 32.87 CNN with skip connections 1.32 1.97 34.46 GAN 1.88 3.3 33.99 Convolutional network with skip connections outperforms residual convolutional network in all quality metrics. This is due to better downscaling over continents, which is not in line with the purpose of this study. Despite the fact that the generative adversarial network is inferior to methods based on convolutional neural networks in terms of quality metrics, this architecture can be considered as the most promising of all the investigated. The generating network model is the only one that detects the small-scale structure of wind fields. The best values of the RMSE-95 and PSNR quality metrics are shown by the cubic interpolation method. Convolutional network with added skip connections is the best method according to the RMSE metric. 132 Despite the fact that the generative adversarial network does not outperform the other methods, including the reference solution, in any of the selected quality metrics, it can be considered as showing the most encouraging results. Simultaneous training of the generator and the discriminator changes the training direction of the generator so that it learns to repeat the small-scale wind pattern over the ocean without overfitting over the continent (Figure 1). (a) (b) (c) (d) (e) Figure 1: Difference between downscaled and true wind speed (m/s), 00:00, 1 January 2010: (a) Cubic interpolation; (b) Linear CNN; (c) Residual CNN; (d) CNN with skip connections; (e) Generative adversarial network. 4. Discussion This study does not imply that neural network downscaling methods will be used directly for operational prediction based on data on a coarse grid. The results have shown that in terms of the spatial resolution of the downscaling models, the resulting data are not competitive in comparison with the existing dynamic methods. Nevertheless, some use of the generative adversarial network in the downscaling of climatic variables can be considered a promising basis for the further development of statistical scaling methods. One of the problems that needs to be solved is the increase in the number of predictors for training. Perhaps datasets with more climate variables and static predictors will allow the model to train more efficiently. There are many applications for statistical downscaling techniques that require more accurate local wind speed predictions. These include renewable energy, local distribution of air pollutants, water transport, sailing, etc. In cases where average speed predictions are important, as for renewable energy sources, computationally cheap neural network scaling methods can be most widely used. For other needs requiring accurate extreme wind speed values, additional research will be necessary. For example, the training time of the models may be insufficient, and the scaled predictions, as the results of this study show, may be too smooth. In conclusion, it should be noted that the emergence of new methods for solving the problem of scaling the wind speed increases the scope of the forecasts obtained in this way, which is especially important for regions with complex topography. In turn, further research of neural network methods will improve their quality, expanding their application in addition to numerical weather forecasting models. 133 5. Acknowledgements This work was undertaken with financial support by the Russian Science Foundation grant № 17- 77-20112-P. The studies were carried out using the resources of the Center for Shared Use of Scientific Equipment "Center for Processing and Storage of Scientific Data of the Far Eastern Branch of the Russian Academy of Sciences", funded by the Russian Federation represented by the Ministry of Science and Higher Education of the Russian Federation under project No. 075-15-2021-663. 6. References [1] Hersbach, Hans, et al. "The ERA5 global reanalysis." Quarterly Journal of the Royal Meteorological Society 146.730 (2020): 1999-2049. [2] Maraun, Douglas, et al. "Towards process-informed bias correction of climate change simulations." Nature Climate Change 7.11 (2017): 764-773. [3] Hewitson, Bruce C., and Robert George Crane. "Climate downscaling: techniques and application." Climate Research 7.2 (1996): 85-95. [4] Rummukainen, Markku. "State‐of‐the‐art with regional climate models." Wiley Interdisciplinary Reviews: Climate Change 1.1 (2010): 82-96. [5] Xue, Yongkang, et al. "A review on regional dynamical downscaling in intraseasonal to seasonal simulation/prediction and major factors that affect downscaling ability." Atmospheric research 147 (2014): 68-85. [6] Tilinina, Natalia, Alexander Gavrikov, and Sergey K. Gulev. "Association of the North Atlantic surface turbulent heat fluxes with midlatitude cyclones." Monthly Weather Review 146.11 (2018): 3691-3715. [7] Gulev, Sergey K., and Konstantin Belyaev. "Probability distribution characteristics for surface air–sea turbulent heat fluxes over the global ocean." Journal of Climate 25.1 (2012): 184-206. [8] Gavrikov, Alexander, et al. "RAS-NAAD: 40-yr High-Resolution North Atlantic atmospheric hindcast for multipurpose applications (new dataset for the regional mesoscale studies in the atmosphere and the ocean)." Journal of Applied Meteorology and Climatology 59.5 (2020): 793- 817. [9] Höhlein, Kevin, et al. "A comparative study of convolutional neural network models for wind field downscaling." Meteorological Applications 27.6 (2020): e1961. [10] Ledig, Christian, et al. "Photo-realistic single image super-resolution using a generative adversarial network." Proceedings of the IEEE conference on computer vision and pattern recognition. 2017. 134