Combination of geostatistical interpolation techniques (e.g. kriging) and machine learning (e.g. neural networks) leads to better prediction accuracy and productivity. Application of an arti cial neural network residual kriging (ANNRK) for spatial prediction of soil contamination with Chromium (Cr) is considered in the paper. We examined and compared two neural networks: Generalized Regression Neural Network (GRNN) and Multilayer Perceptron (MLP). We consider them as classes of neural networks widely used for the continuous function mapping, as well as a combined technique Multilayer Perceptron Residual Kriging (MLPRK). The case study is based on the survey on surface contamination by Cr at the subarctic city Noyabrsk, Russia. Structures of used models have been developed using a computer simulation based on a minimization of the RMSE. Each technique has its own bene ts and drawbacks; however both demonstrated fast training and good prediction possibilities. The MLPRK showed the best predictive accuracy.
(OK), which calculates estimates for a poorly sampled element with help of
well-sampled highly correlated elements (co-elements) [
Accuracy of kriging depends on the density and size of the sample grid, since the method is based on interpolation. However, it is not always possible to collect the required number of samples due to time-related or resource-related constraints. To overcome these shortcomings and improve the accuracy, a more e ective method is required.
Currently, one of the applicable methods of prediction is the machine learning and, in particular, the arti cial neural networks (ANN), which provide many powerful techniques for predicting, pattern recognition, data analysis, and many other operations.
Overviews [
The most frequently used ANN in environmental studies is the multi-layer
perceptron (MLP) and generalized regression neural networks (GRNN).
Perceptrons are widely used in studies on soil chemical elements content assessment [
The GRNNs are variations of the radial basis functions (RBF) neural networks, which are based on the kernel regression networks. The GRNNs are used as interpolators and are known as universal function approximators, which can approximate any continuous nonlinear function. The key di erence between GRNNs and MLPs is that the GRNNs do not require the learning process using long-term iterative procedures as back propagation networks (MLPs etc.).
To neutralize weaknesses and to multiply dignities of the mentioned models, it
has been o ered to combine di erent techniques. Researchers successfully exploit
the hybridization of geostatistical and neural approaches, which lead to better
predictions and lower errors [
In this work, we examine two solo ANN-based prediction models (MLP, GRNN), as well as a hybrid model combining the ANN based forecasting and cokriging (MLPRK) for prediction a soil pollutant Cr content at a particular location of the Subarctic Noyabrsk, Russia. We examine the results obtained by applying the models and compare the models output. 2 2.1
Data for the study were obtained from the results of the soil survey in Noyabrsk (N63:1926 , E75:5066 ) and Yamalo-Nenets Autonomous Okrug (YNAO), Russia (see Fig. 1(a)). The area of sampling was approximately 16.5 km2. The detailed spatial location of sampling points is shown in Fig. 1(b). The terrain was at and covered with sandy soil (Cryosols soil type). Totally, 237 topsoil samples at a depth of 0.05 m were collected. Concentration indicators for the elements were obtained by chemical analysis. Preparation of the soil specimens and chemical analysis were conducted in compliance with actual standard requirements. The chemical laboratory involved with the soil sample preparation and analysis passed through the Russian Federal Certi cation System. 2.3
Study Algorithm. To estimate the pollutant content and to predict its distribution at the unknown locations, three competing techniques was applied: two ANN methods (MLP and GRNN), as well as a hybrid ANN-geostatistical model Multi-layer Perceptron Residual Kriging (MLPRK).
The MLPRK is a three-step algorithm combining two di erent interpolation techniques in one ensemble. The rst step implies estimating large-scale nonlinear trends using neural networks (MLP). The second step is analysis of the stationary residuals by ordinary kriging (exponential model), which is able to provide local estimates. The nal step is estimation produced as a sum of the ANN predictions and ordinary kriging estimates of the residuals. In the work, the ANN predictions were carried out in MATLAB; the ArcGIS application was performed to predict the values by kriging.
Since further analysis implies the use of the ANN, a method of selecting input
variables (IVS) based on the estimation of partial mutual information (PMI) was
used [
All the samples were randomly split into independent training and test data sets. The training data set (165 samples) was used for building cokriging, training the networks, and for interpolating the surface pollutant distribution. The test data set (72 samples) was used for testing the models only.
Data Isotropy. Values of the experimental semivariogram were calculated depending on the distance between the points in the pair within the lag h. The lag was selected according to the size of the spatial correlation. With this approach, the semivariogram value did not depend on the orientation of the pair in space, which means isotropy of the structure. The variogram constructed in all directions (omnidirectional) depends on all pairs of points in the domain. To identify di erences in spatial structure depending on the direction the experimental variograms in various directions were applied.
Building ANNs. As rst ANN type, a feed-forward multi-layer perceptron (MLP) with the Levenberg -Marquardt training method was used. The network structure was determined during computer simulation. The input layer of MLP was compiled with sampling points; the hidden layer consists of a several neurons, and the output layer represents the element concentration in the relevant sample. Selection of the number of neurons in the hidden layer was carried out by the lower total root mean squared error (RMSE) (5) of prediction of the element (Cr) concentration for the training (165 samples), test (72 samples), and a complete set of data (237 samples). The number of neurons was varied from 2 to 20. Each network was trained 500 times and the best of them was selected. The network education quality was checked by the Spearmans correlation coe cient, mean absolute error (MAE) (4), and RMSE between the results of the network predictions and the training data set.
As the second ANN, the GRNN was chosen. The rst layer in the GRNN resembles the RBF with the amount of neurons that is equivalent to the quantity of input vectors. Choice of the SPREAD parameter of the RBF, that is known as a smoothing parameter, determines the width of the input area, to which each basis function responds. It is the distance from the center of a Gaussian where the value is one-half of the peak value. The GRNN network had 165 input neurons according to 165 sampling points formed the training data set. During the simulation, the SPREAD parameter varies from 0.01 to 0.30 with step 0.01; totally, 300 simulations were done. 2.4
The starting procedure for the residual kriging is the prediction of residual values by the neural network in the test points. Residuals in the neural network can be de ned as follows: r(xi) = Z(xi)
ZANN(xi); (1) where r(xi) are the residuals of data set (xi), Z(xi) are the measured values, ZANN(xi) are the values estimated by the neural network. The resulting residuals were estimated using kriging. Evaluation in ordinary kriging (OK) is constructed as a linear combination of input data
rOK(x) = X ir(xi); where rOK is the estimated value at the point x using OK, i(x) are the optimal weights with the condition P i = 1, and r(xi) is the residual of the neural network for the point (xi). The nal evaluation of the pollutant content Y (xi) was obtained as the sum of the neural network evaluation and residuals evaluation by kriging
Y (xi) = ZANN(xi) + rOK(xi): 2.5
The performance of prediction models was based on the model error statistics. The predictive accuracy of each selected approach was veri ed by the Spearmans rank correlation coe cient r, MAE (4) and RMSE (5) between the prediction and raw data from the training data set.
M AE =
Pn i=1 jzmod(xi) n
z(xi)j ; RM SE = r Pn i=1(zmod(xi) n z(xi))2 ; where zmod(xi) is a predicted concentration (ANNs, cokriging), z(xi) is a measured concentration, n is a number of points. 3
3.1
During the analysis, the contents of eight elements were obtained (Al, Cr, Mn, Fe, Co, Ni, Zn, Pb). Correlation analysis (Table 1) revealed possible co-elements for Cr (in bold). Three of them (Fe, Co, Ni) we used for the cokriging. (2) (3) (4) (5) The descriptive statistics of Cr and its co-elements concentrations are shown in the Table 2.
SD is a standard deviation; CV is a coe cient of variation; SK is a Skewness; RKu is a Kurtosis; MED is a Median.
From the basic statistics table, it is observed that the element attributes are erratic and positively skewed. The Cr concentrations in all sampling points were from 16.6 to 140 mg/kg, with an average value of 62.4 mg/kg and a standard deviation of 24.2 mg/kg. Due to the skewness of the distribution, the median value (58.8 mg/kg) is more representative of the average Cr content in the study area than the arithmetic mean. Co-elements demonstrate similar characteristics when the medians (12673 mg/kg for Fe, 4.4 mg/kg for Co, 11.0 mg/kg for Ni) are more representative than mean values. 3.2
The probability distribution of the Cr concentration for the training sites is positively skewed and leptokurtic (Table 1). The result of the Chi-Square test shows that this variable is close to normal distribution (p=0.18).
To demonstrate di erences in the spatial structure depending on the direction, variograms are constructed in six directions (0 , 30 , 60 , 90 , 120 and 150 ) (Fig. 2(a)). The anisotropy of the raw data in all these directions is invisible on the direction variograms (Fig. 2(a)) and variogram surface (Fig. 2(b)).
The nal con guration of the MLP network selected was 2-6-1, i.e. the hidden layer contains 6 neurons (see Fig. 3(a)). In our case, 165 sampling points formed the training data set that was applied for networks training.
During the simulation for GRNN building, the SPREAD parameter varies from 0.01 to 0.30 with step 0.01; totally, 300 simulations were done. The minimal RMSE was achieved with the SPREAD parameter of 0.035 (see Fig. 3(b)). (a) (b) Fig. 3. MLP (a) and GRNN (b) frameworks selection based on RMSE minimization: root mean square error (RMSE) of the neural network for test, training, and overall data under di erent neuron number in the hidden layer for Cr 3.3
MLP and MLPRK have shown signi cant increase in modeling accuracy comparing to a geostatistical method (cokriging) and even to GRNN. As Table 3 reveals, the MLP-based models had smaller RMSE than cokriging (9.5% improvement). MAE index of the MLP-based models are about 4% better than cokriging ones. The basic GRNN model demonstrated an unexpectedly low correlation coe cient. This means that the method cannot be applied to modeling in our case.
It is found that application of the hybrid approach (MLPRK) gives an
increase in the accuracy of prediction, which corresponds to the previous suggestion
[
We compared approaches to modeling the spatial distribution of the chemical element concentrations in the surface layer of soil (the geostatistical technique, ANNs, and hybrid model of MLPRK). A quality of models prediction could be analyzed with the help of the test data set, which is not applied to training networks or kriging estimates. Directional variograms of raw data and a variogram surface of training data are shown in Fig. 4. They con rm absence of any anisotropy of data. Table 3 shows the statistical parameters used to assess the performance of the di erent methods (the best values demonstrated by MLP-based models are written in bold).
The MLPRK model reproduces the spatial structure of the Cr distribution quite well. This model has extracted structured information leaving out unexplained noise and local variability. This is shown on directional residuals variograms (Fig.4). The study of the residuals con rms importance of the variography for analysis and modeling of spatial data with using the neural network algorithms. Comparison of di erent approaches to prediction of the chemical elements distribution in the surface layer of soil is carried out. Estimation of the ANN with prediction of residuals by ordinary kriging reduces the ANN prediction errors, and increases accuracy of the models. The results show that the MLP-based models usually are more accurate than the kriging-based ones. In comparison with cokriging, the most signi cant improvement of RMSE (9.5%) is observed in the MLPRK model.
The results con rm possibility of the hybrid ANN-Kriging methods that can be used to improve the accuracy of modeling the spatial distribution of concentrations of chemical elements in the upper layer of soils in urban areas characterising by high heterogeneity. We assume that using (as input) not only spatial coordinates, but also additional variables will improve the predictive ability of ANN based models. This is since the variables have a signi cant correlation with the predicted variable, for example, the concentration of joint elements, geographic data, etc.
Acknowledgment. The reported study was funded by RFBR according to the research project N 18-55-18002.