Recent years have witnessed great progress in emulators based on neural network (NN). Current state-of-the-art emulators methods often apply shallow NN to attain high performance in physics system, which brings a faster speed processing on resource-constrained environments. Although several works have focused on improving accuracy in physics emulators, an efective and eficient method for tackling time consuming problem of existing system on high-resolution remains lacking. In this paper, we propose a optimum NN emulator of a microphysics (MPS) parameterization scheme to efectively solve the problem in numerical weather prediction (NWP) model, Korea Integrated Model (KIM) in particular. Specifically, we adopt a shallow NN to build an intelligent emulator, which can learn the feature map and estimate the vertical MPS forcing increment profiles. This study mainly relies on two technical contributions: (1) Optimization: reviewing and improving models for simulating non-linear parameters; (2) Feasibility: ecfiient computation with minimal loss of physical information. We validate the proposed model with four seasons (10-day forecast at 200 seconds interval) of KIM. Results indicate that the proposed single-layer network shows best performance for emulating MPS in KIM. Our analyses will provide a guideline for optimal physical parameterizations modeling.
In recent decades, there have been considerable improvements in terms of
predictability in numerical weather prediction (NWP) models. Advancement in
computer hardware technology is considered one of the big drivers
accounting for improvement in this area. However, increase in spatial-temporal
resolution for NWP models, which is critical for better predictability, requires
exponential increase of computational power. Thus, utilizing even more
computational resource and/or achieving better model code optimization is continually
needed. Parameterization of subgrid-scale physical processes has been an active
research area ever since the birth of NWP. There are extensive research
activities on physics parameterization [
Previous studies of parameterization based on machine learning can be
categorized into developing emulation-based methods for accelerated calculation
[
In this study, we explore various types of NN structure that represent physics of the atmosphere. Specifically, we compare a series of NN models to test its efectiveness with the combination of the following options: (i) numbers of each layer, (ii) neuron structure, and (iii) activation function.
This section describes several studies based on emulation similar to our proposed model. We classify these studies into two groups: (i) single-physics and (ii) unified-physics emulation.
The state-of-the-art models for physics emulator include a step that optimizes
hidden layer based on machine learning, or approximates these weights based
on various activation functions. Many previous studies demonstrated that the
NN emulator approach can be applied successfully to speed up the computation
of a single physics module. Machine learning is first used to emulate longwave
radiation for the European Centre for Medium-Range Weather Forecasts models
[
Aside from these, multi-perceptrons have been proposed for predicting
nonlinear phenomena in the physics of the atmosphere influenced by a number of
factors. To accelerate expensive radiative transfer computations, deep NN has
been applied to predict vertical profiles of longwave and shortwave radiative
lfuxes in weather and climate models [
Traditional emulation-based methods were parameterized based on NN by
dividing the physical process separately. Some errors arising from these models had
a significant efect on the accuracy of the overall NWP model. As all physical
processes closely interacts with each another, one error source in a particular
physics module can cause larger errors in other sub-physics. Belochitski
proposed a shallow NN based emulator of a complete suite of atmospheric physics
parameterizations [
This paper studies the problem of physics from the perspective of a NN to propose an emulation-based algorithm.
Physics emulator is to learn input features on a network to map the variables into next input. Let X(t) and Y (t) represent the input and output at time t, respectively. Denote X = (x0, x1, · · · , xn)T ∈ Rnas the input information related to physics observed on the network and Y ∈ Rm as the output information, where n and m are the number of input and output dimension, respectively. The purpose of training is to find θ that maximizes the conditional probability of input-output pairs in the training sets. On each t step, the decoder receives the features of the previous input. If the model is trained to be predicting the output with constantly updated input presented in the previous phase, we are training function Φ successively as: Φ = hx(0t), · · · , x(nt); θ i → hx(0t+1), · · · , x(mt+1); θ i , where t denotes time step. Inputs and outputs consist of the same attributes for each step and each output attributes are connected in a recursive manner. For any xi, the inputs are calculated according to the following formula: where wi,j is the weights between input xi and output yj of physics. 3.2
Since the network is a matrix, for any value wi,j in the matrix, its output yi is defined as the following: yi =
M X x′iwi,j + bj .
i=1 Using the definition of layers, we can obtain the output matrix of the size n × m. Assume that there are L network layers, where the 0th layer is the input layer and the lth(1 ≤ l ≤ L) layers are fully connected layers. For any fully connected x′ = 1, x − max − 0, min min
if x > max , if min ≤ x < max otherwise The min and max range of each attribute are set manually as physically allowable values in KIM. We define the emulator network as a weight vector W representing the state of the entire attributes and concatenate each value associated variables as W . W ∈ Rn× m represents the weighted matrix: w1,1 w1,2 · · · w2,1 w2,2 · · · W = ... ... . . .
wn,1 wn,2 · · · w1,n w2,m . . . wn,m n× m (1) (2) (3) (4) layer l ∈ [1, · · · , L], the output of the lth(1 ≤ l ≤ calculated using the following equation: L) fully connected layer is where sl− 1 denotes the number of parameters in the (l − 1)th layer; Hs(l) ∈ Rsl− 1 is the sth physics hidden parameter in the lth layer and σ (· ) is the activation function. The neuron structure can be divided into four following structures according to the change in the number of each neuron:
sl− 1 Hr(l) = σ (X fθ l Hs(l− 1)),
s=1 α : sL < {s2, · · · , sl− 1} ≤ s1 β : sL < s1 < {s2, · · ·
, sl− 1} γ : {s2, · · · , sl− 1} < sL < s1 δ : sL ≤ { s2, · · · , sl− 1} < s1, where {S2, · · · , Sl− 1} denotes the number set of hidden neurons connected to input layer Hs(1), and sl is a number of output neurons. Activation functions Hyperbolic tangent tested in this study are described in Table 1. The sl− 1, number of l, and σ (· ), defined in this section, are chosen as optimized formula by experiments in the next section. Finally, the errors between actual scores and the predicted scores are minimized by L1-norm. 4
In this section, we evaluate the NN model for emulating physics among the models proposed in section 3. (5) (6)
To verify the eficacy of the proposed methods, we conducted experiments with
generated MPS dataset for every 200 seconds using ERA5 reanalysis by latest
KIM version 3.6. MPS has been selected for this first study because the
nonlinearity of MPS is hard to predict and it is one of the most time-consuming
part of the atmospheric physical processes. The input and output attributes
are shown in Table 2. The dataset consists of 4 sets with each season to
consider the temporal features. Dynamical core of the NWP is a spectral element
cubed-sphere nonhydrostatic model with a horizontally quasi-uniform resolution
of 100km, with 91 vertical layers (˜0.01hpa) on a hybrid sigma-pressure vertical
coordinate. Data is extracted randomly in each KIM forecasting data, and
composed of 6,998,688 training sets. We use the other 1,749,672 sets for evaluation.
Our methods are implemented using PyTorch [
We evaluate our emulation both quantitatively and qualitatively for estimating the optimal emulator. The results of this experiment are evaluated with three metrics shown in Eq. (7): Root Mean Square Error (RMSE), mean absolute error (MAE), and Peak Signal-to-noise ratio (PSNR). Given NN output to ei and KIM output to e¯i, the evaluation function can be written as:
n MAE = 1 X n i=1
|ei − e¯i| v
n RMSE = tuu n1 X(ei − e¯i)2
i=1 PSNR = 10 · log( max2 n
X(ei − e¯i)2), n i=1 (7) 4.2
For comparison, we set a benchmark case that consists of 2-layers NN based on a hyperbolic tangent and the neuron structure of γ . Table. 3 depicts the average accuracy of emulations with diferent approaches. In our first experiment, we evaluate the impact of learning each structure for estimating the optimal depth of NN layers. As L is 2 in the benchmark case, we set the hidden neurons to α s2 → 822, β s2 → 1096, γ s2 → 400, and δ s2 → 548, respectively. The PSNR result indicates that the performances of emulation approaches by applying the β method increased from 29.55 (the γ method commonly used in emulation) to 32.61 of the average accuracy of emulation. The β structure also shows overall best performance compared to other network structures as shown in the evaluation metric scores. When the number of hidden neurons decreases relative to the input, such as in the γ structure, loss of latent features is inevitable. These losses can cause uncertainties of physics, so the hidden neurons must be greater than or equal to the number of output neurons.
Let us compare the impact of learning according to layer depth and activation function. All networks to which the deep layer was applied except for the single-layer applied the fully connected layer, and then proceeded with setting the activation function and dropout (0.2). The result of approximation errors presented above shows that the shallow NN is capable of providing NN emulations with small error. Especially, single-layer yields best performance compared to other cases. As for the activation, the Leaky ReLU with a bend in the slope shows the worst performance, consistent with previous studies. The results demonstrate that hyperbolic tangent activation for emulating MPS is efective, providing accurate and visually promising results. The swish activation function operated well as hyperbolic tangent with 2.14(· 10− 4) error. Generally, good results are obtained from functions with a soft and high gradient.
Finally, we compare the original KIM MPS output with the NN emulator output. Fig. 1 illustrates the correlations between outputs from the NN-based emulation with the outputs of the MPS. The single-layer-based emulator, which is the best result in Table 3, was used to emulate the output of the MPS. The figure is outputs according to 5-day input data integrated by KIM in consideration of spin-up.
Average of Correlation coeficients is 0.998 (Precipitation: 0.989, Snow: 0.989, Temperature: 0.999, Specific humidity: 0.999, and Ratio: 997). Emulators with shallow NN also verified similar results. KIM MPS based, NN emulation, and their diference in precipitation distribution is shown in Fig. 2. Precipitation simulations of KIM and NN are shown in (a) and (b); (c) is the diference of the simulations, with single-layer emulator. The precipitation distributions for the KIM and NN emulator runs are very similar showing little diference as shown in (c) in Fig. 2.
As information technology evolves, the tasks of accelerating physics parameterization and increasing the resolution of NWP have been gaining importance. Traditional emulator methods tend to design only on the overall composition such as a kind of input data. This paper focused on building an optimal NN targeted according to the detail of network settings for physics emulation. Various experiments were carried out to design the detailed structure of the network, and we tried to design a network that can understand the characteristics of the physics. Specifically, to overcome the drawbacks of existing models, we have attempted methods for optimizing NN details (i.e., neuron structure, number of layers, and activation function). Through experiments parameterization of the MPS using a single layer showed the best performance.
In the case of MPS, the usage of deep-layer does not bring the best result but it may bring great results in other physics parameterizations. The depth of layer can be changed according to physics patterns and other elements (e.g. number of profiles, resolution, parameterization type, so on) used by the NWP model. So we want an emulation that feasible not only single physics, but also across physics module. Ideally, we will achieve unified-physics emulation by considering diferent physical patterns, leveraging dynamic NN modeling.
This work was carried out through the R&D project ”Development of the Nextgeneration Operational System of Korea Institute of Atmospheric Prediction Systems (KIAPS)”, funded by Korea Meteorological Administration (KMA202002213).