Combined usage of the optical and radar remote sensing data in territory monitoring tasks V N Kopenkov1 1 Samara National Research University, Moskovskoe Shosse, 34А, Samara, Russia, 443086 e-mail: vkop@geosamara.ru Abstract. At the present time, a lot of problems in a sphere of fundamental sciences as well as technical and applied tasks can be solved only with the use of satellite images, since their usage reduces material, financial and time costs significantly in comparison with traditional methods. One of the modern integrated approach remote sensing processing is to join the measurements obtained from the various sources, such as optical and radar sensors, allowing to achieve a gain in comparison with independent processing due to the extension of the information volume and the opportunities of data acquisition (weather conditions, spectral ranges, etc.). However, methods of digital processing and interpretation of radar data, as well as qualitative and proven methods and algorithms for joint processing of optical and radar satellite images, has not sufficiently been well developed yet. Therefore, the development of new methods and information technology of joint analysis and interpretation of optical and radar data which are a major issue of the current paper, are certainly relevant. The paper presents an information technology for joint processing of optical and radar satellite imagery, based on training the processing procedure based on the reference values of data from sensors of the one type (optical data), followed by applying to both data types: optical and SAR data. 1. Introduction Today, a lot of problems of fundamental sciences, as well as technical and applied tasks can be solved effectively only with help of space vehicles, since their usage significantly reduces the material, financial and time costs compared with traditional methods. Recently, it is possible to receive multi- spectral aerospace information in the optical and radio bands in a regular way. At the same time, the tasks of determining various characteristics of the earth’s surface, monitoring and analyzing of emergency, classifying and analyzing of territories as a whole are becoming increasingly important. A special place is occupied by the task of classifying the underlying surface and species composition. Conventionally, to solve this task images obtained in the optical range of the electromagnetic spectrum used as input data. Classic approaches which are successfully used for the processing of optical images do not give positive results when working with data obtained by synthetic aperture radars. The recognition of vegetation types, borders of territories, the detection and classification of objects on the base of radar images is practically not performed, because approaches to processing and thematic interpretation of radar data are not sufficiently well developed and investigated. For example, all known algorithms for automatic classification are adapted to the Gaussian model of the distribution of additive noise arising only on optical images. High-resolution radar images formed by synthetic V International Conference on "Information Technology and Nanotechnology" (ITNT-2019) Image Processing and Earth Remote Sensing V N Kopenkov aperture radars as a result of irradiating the surface of the Earth with coherent waves are usually distorted by multiplicative noise, known as speckle noise. This kind of noise is a serious problem during the processing of radar images on the base of classic methods. Therefore, we need image processing and analysis methods adapted to the physical and technical features of radar imaging. 2. The scheme of combining the radar and optical data Traditionally, the technology of combining and processing of mixed remote sensing data consists of 3 main subtasks: independent preliminary processing, matching and combining satellite images obtained by various sensors in the optical and microwave ranges, detecting differences and further processing of remote sensing data. Within the framework of the first task, the well-known solutions combine the methods of interpolation of data on a non-uniform grid, image reconstruction frequency methods, adaptive filtering [1], radiometric, geometric and polarimetric calibration (for polarimetric radar data) [2], image enhancement operations [1], reduction of speckle noise on a radar image, orthotransformation taking into account a digital elevation model, etc. Existing approaches ordinarily solve this task using assumptions about the same characteristics of imaging systems, which have similar spatial resolution and the same spectral ranges of signal acquisition and receiving. The task of matching heterogeneous images are needed for the realization of a further goal − joint analysis of radar and optical satellite imagery refers to the data processing and analysis area [3-5]. In this case, special attention should be paid to the choice of the merging method. Because the type and amount of preliminary data preparation depend on it. For example, in the case of point-oriented methods, high-precision geocoding and the further combination of all images is very important. In the case of object processing the preliminary detection, selection and, possibly, recognition of objects on images, as the objects observed on radar images often look completely different than on optical ones, since the scattering of the radio signal and sunlight by objects can differ significantly. Radar and optical observations are complementary, and this fact may have a different nature depending on the problem being solved. Their joint processing improves the ability to detect certain objects and measure the parameters of the underlying territory. The general classical scheme of combining radar and optical data is shown in Figure 1. Herewith, to solve practical tasks, in most cases, the combination of optical and radar data is implemented in two ways. In a situation where optical data is not available, radar data can be a good alternative. For joint processing, in this case, data is orthotransformed, and reduced to comparable resolution (or the optical image may have a lower resolution since the imaging capabilities of such images are higher). After that, points or objects in both images are analyzed to solve the problem of identifying differences and unchanged regions, as well as for further analysis of remote sensing data. The second data matching approach is based on the RGB ↔ HIS conversion, which allows you to combine terrain information obtained in two different ranges on one output image. First, the three most informative spectral channels RGB → HIS (brightness-saturation-hue color) are converted, then the brightness is replaced with a radar image on the resulting image and the inverse transformation is performed, which requires a balanced stretching of the channels to improve the quality of the final representation in RGB. Using such images, it is possible to determine the roughness (low/high vegetation, flooded/not-flooded area), and the structure of the territory, which is necessary for geological tasks. However, both the first and second version of the data combination is not suitable for the qualitative solution of the underlying surface classification and analysis tasks. 3. The technology of joint processing of optical and radar data based on a training dataset Analysis and processing of images obtained by dissimilar remote sensing sensors, with spatial and spectral resolution improving, usually involves the formation of individual models of observation for each of the sensors, which is not always possible and requires a large amount of additional a priori information about the observation model. In addition, it is extremely difficult to develop an automatic V International Conference on "Information Technology and Nanotechnology" (ITNT-2019) 335 Image Processing and Earth Remote Sensing V N Kopenkov adaptive technology for processing dissimilar remote sensing data on the basis of a variety of particular observation models. Figure 1. The classical scheme of combining radar and optical data. Therefore, the methods of nonlinear analysis and data interpretation, such as neural networks or hierarchical structures or Approximate Local Correlation Integral (aLOCI) are of the greatest interest for investigation. These methods can be used without definite assumptions about the source data. In addition, aLOCI uses quadtrees to increase the efficiency of calculations in the multidimensional case. Among these methods in our task we intend to use a multidimensional classification based on optical data, since the solution of the microwave signal analysis problem in the general case can be made in the form of a classification with training, based on features in the form of optical data, observation results and a priori information about the underlying surface [6-7]. At the same time, it is advisable to use standard methods for extracting features and classification algorithms for multidimensional data, because the interest in improving the quality of evaluation of thematic classification. These methods most often include features obtained as a result of the principal component method (PCA) and the Nonparametric Weighted Feature Extraction (NWFE) method, and such classifiers as the Support Vector Method (SVM) and Random Forest (RF) [5-6]. The developed algorithm for the local processing of satellite images based on hierarchical regression [6] allows us to accelerate data processing by using hierarchical data structures and in a multidimensional case, the performance is comparable to or superior to other classification methods, in particular, neural networks. Finally, the assessment of the quality of the results of classification and detection of changes in the territory by received dissimilar satellite images relates to the field of image analysis based on the values of one remote sensing sensor using the values of another sensor (for example, calculating of the regression function) [7,10-11]. Thus, the construction of the technology is supposed to be carried out in three stages: data preparation, combination and analysis of the image sequence. At the first stage, the use of sensor- V International Conference on "Information Technology and Nanotechnology" (ITNT-2019) 336 Image Processing and Earth Remote Sensing V N Kopenkov dependent methods of processing and combining (spatial) data is assumed, i.e. features of the source of satellite images (optical or radar) are taken into account, but it is assumed that the observed territory is the same. After that, the radar images "are brought to the kind" of optical (or optical images "are brought to the kind" of radar, depending on the task) based on the features and non-linear regression models. And finally, a sequence of images with similar spectral characteristics is formed that makes it possible to implement the solution of thematic processing tasks such as classification, change detection and etc. In this paper, to form an aggregate set of satellite images that are suitable as source data for classification and analysis of changes, we propose a technology for converting (point-by-point processing based on reference values) data of the one type (conceivably radar) based on a training sample consisting of a data set of another type (optical images). Regression models of processing based on a decision tree for merging extracted SAR (Synthetic-aperture radar) radars and optical sensor values are assumed to be used here. Hierarchical structures in the form of a decision tree are not parametric and therefore do not require an unambiguous description of the desired classes. Herewith, the final processing procedure consists of a set of explicit hierarchical conditions (rules, thresholds of functions) applied to the input data intervals, in our case SAR and optical functions, as reference values. This allows classifying the pixels (area sampling) of the radar image as a function of optical data. Decision tree classifiers have been successfully applied to classify the underlying surface. The general scheme of data conversion (in the process of applying the processing procedure), oriented on a priori information about the desired processing result (presented in [6, 10]), is illustrated by the diagrams presented in Figure 2. Figure 2. The scheme of training and usage of processing procedures, based on the training dataset. Considering a large number of matched pairs of image pixels (zi, xi) as a training sample, we build K −1 the signal processing procedure Ф, which associates the feature vector { yk ( n,[p])} calculated on k =0 the base of discrete samples of the input signal X, the output value of the signal Z. The sets of features, parameters of the processing procedure construction, as well as the type of functions of the elementary regression in the terminal nodes can be adjusted according to the results of the evaluation of the quality of processing. To evaluate the generalization ability of the constructed procedure, the full sliding control functional is used: 1 N Q st ( µ (Ω), Ω) = ∑ν (µ (Ω ns ), Ω tn ), N n=1 where Ω is the final set of objects for learning (dataset), µ (Ω) is the algorithm (method) of training on a dataset Ω , (Ω ns , Ω tn ), n = 1,2,..., N are the variants of dividing the dataset into training and control parts. V International Conference on "Information Technology and Nanotechnology" (ITNT-2019) 337 Image Processing and Earth Remote Sensing V N Kopenkov ν ( µ (Ω ns ), Ω tn ) is the error rate of the algorithm µ (Ω ns ) , constructed on the base of the set Ω ns and checked by using the set Ω tn . Receiving information from different sensor types leads to a shift in the position of sensor cells relative to objects due to the differences in the orbital parameters of the satellites and due to the difference in the physical laws of pixel formation in the image (resolution is dissimilar for optical and radar data). It is obvious that individual sensors cover the observed area with grids with different parameters that can be described as a relative motion within the coordinate system of the observed scene. As for the coordinate system of the observed scene, you can use some projection coordinate system in which each pixel is represented as an area of finite size, assuming that the coordinates of the image pixel are the coordinates of the center of the pixel. This view does not contradict the processing technology based on decision trees since the latter can use the integral values in the spectral channels over the local window as attributes for the input and output pixels. In this way, the technology of forming a sequence of consistent and combined dissimilar remote sensing images is used to obtain features for the tasks of land use classification according to remote sensing data, as well as for analyzing changes on the sequence of images. 4. Evaluation of the training dataset volume and experimental research Is quite logical is the fact that when working with images solution of the problem of constructing the processing procedures taking into account all combination of training and testing dataset is unrealizable in practice because of the giant of busting on various combinations of datasets. Therefore, it is necessary to develop a method to determine the possibility of stopping a forming, or the need to continue training and testing datasets generation on the base of a finite number of them. It is obvious that for every task there is an optimal complexity of the model, at which the best quality of generalization is achieved. In paper [10], the technology of decision making about stopping the process of generating various combinations of training and control datasets was proposed, according to which to make a decision of stopping generation of various combinations of training and control datasets, and about the transition to the next subset of features it is necessary to calculate confidence intervals for the expectation of a Poisson distribution with parameter λ : Bin(n, λ ) ≈ P (λ ) for the functional full cross-validation on n a datasets N1 and N2 in the form:  τ1−α / 2 λ1 τ λ   τ λ τ λ  λ1 − , λ1 + 1−α / 2 1  λ 2 − 1−α / 2 2 , λ 2 + 1−α / 2 2  ,  N1 N1   N2 N 2  where τ1−α /2 – quantile of distribution N 0,1 for level 1 − α / 2 (α = 1 − γ ). The decision of stopping of a generation of various combinations of datasets and about the transition to the next subset of features is taken at a time when a separation of calculated confidence intervals on adjacent steps is achieved. The presented technology of decision about stopping the brute force bust process allows estimating the amount and possible combinations of training and control data sets used in experimental studies. As data used as training and control datasets, we used sets of images (free data) from Landsat-7 and Sentinel 1-2 satellites for different time periods, as well as Radarsat-2 data available at the center of remote sensing data of Samara University. Datasets from Landsat-7, Spot 6/7, Aist, Resurs-P, Pleiades and coverage with Google and Yandex data were used as reference data in the optical range. Experimental study on the applicability of the technology to dissimilar datasets (optical and radar data) are in progress, so the final results are not presented in this article. Experimental studies of the feasibility of individual stages of processing, classification and analysis of remote sensing data on the basis of hierarchical structures built in accordance with the principles of learning "by precedent" are presented in the author's works [6, 10, 11]. To carry out our experimental studies, we used sets of images from such space vehicles as Spot 7, Aist-2D, Resurs-P, Sentinel-2 in the optical range (more than 40 test images) and radar images from V International Conference on "Information Technology and Nanotechnology" (ITNT-2019) 338 Image Processing and Earth Remote Sensing V N Kopenkov the Radarsat2 and Sentinel-1 satellites (more than 20 scenes). All data were obtained in the spring- summer period in 2010-2018. All images were radiometric and geometrically corrected, brought to a similar resolution and matched (geographically bounded) with an accuracy of about 1 to 2 pixels. All pre-processing and low-level processing task was implemented in ScanEx ImageProcessor software. Examples of images from different spacecraft are shown in Figure 3. Aist-2D Spot-7 Geoton Sentinel-2 04/2016 - 10/2016 05/2016 - 06/2016 05/2016 - 07/2016 04/2018 - 10/2018 Radarsat2 2010-2015 Sentinel-1 04\2018 - 10/2018 Figure 3. Training and testing datasets in a form of images from different spacecrafts. All experimental studies can be divided into 3 groups: 1) Optical data as a training sample, and optical data from the same satellite as test data. 2) Optical data from different satellites as a training sample, and optical data from another satellite as test data. 3) Optical and radar data from different satellites as a training sample, and optical data from another satellite as test data. The results of experimental studies of the quality of the pixel-by-pixel classification are shown in Table 1. In general, it can be noted that the technology of joint processing of optical and local data based on hierarchical structures with training on a priori information is quite efficient. At the same time, we note that if the training sample contains data of the same type as that used as a test data, then the quality of the classification is quite high. It should be noted that the classification based on radar data shows a very low quality of classification since the features in the form of the V International Conference on "Information Technology and Nanotechnology" (ITNT-2019) 339 Image Processing and Earth Remote Sensing V N Kopenkov original image pixels are not sufficiently informative. Therefore, it is necessary to use other features in the form of generalized coefficients over a certain local image area. Selection and optimization of such features is the goal of further work. Table 1. Quality of per-pixel classification. Images for test Training dataset Sentinel-2 Aist-2D Spot-7, Geoton Radarsat2, Sentinel-1 Sentinel-2 96-98% 72-87% 67-79% Sentinel-2 (70%) + Aist-2D 93-98% 90-95% 72-84% Sentinel-2 (50%) + Aist-2D + Spot-7 + 89-97% 86-94% 84-91% Geoton Sentinel-2 (80%) + Radarsat2 + Sentinel-1 93-98% 76-84% 62-78% 46-72% Optical satellites (80%) + Radarsat data 90-97% 87-92% 78-84% 52-74% (20%) Optical satellites (50%) + Radarsat data 86-92% 76-82% 76-80% 57-82% (50%) No experiments were carried out to restore (classify) pixels of optical images by using the pixel values only of radar images because it is impossible to obtain adequate results for various spectral channels (RGB) from radar data only. 5. Conclusion This article presents a new information technology for joint processing of optical and radar satellite imagery. The technology is based on learning procedure that uses the data from sensors of the one type (optical in the basic approximation) as reference values and then the technology is applied to the data of both types: the same one (optical) and the another data type (radar data). Experiments on testing of the developed regression models of processing based on a decision tree for joint processing of data obtained in the optical and SAR ranges confirm the efficiency and effectiveness of the technology. That, as a result, allows classifying the samples (areas of pixels) of the radar image as a function of optical data. The earlier experiments [6, 10, 11], as well as experiments in the framework implemented in this paper, showed the efficiency of using decision trees as classifiers of the underlying surface and demonstrated a high quality of classification, which suggests the possibility of applying this approach to the problem of joint analysis of optical and radar data. 6. References [1] Wan X, Yang Y 2010 Super-resolution image reconstruction Computer Application and System Modeling (ICCASM) 8 351-355 [2] Nikolsky D B 2016 Methods for processing radar data Geomatics URL: http:// geomatica.ru/clauses/48/ (10/01/2018) [3] Kopenkov V N, Sergeev V V, Soifer V A and Chernov A V 2017 Remote sensing of the Earth and geographic information systems Bulletin RFBR 95(3) 78-96 DOI: 10.22204/2410-4639- 2017-095-03-78-96 [4] Zakharova L N, Zakharov A I, Sorochinsky M V, Ryabokon G P and Leonov V M 2011 Joint data analysis of optical and radar sensors: opportunities. restrictions, prospects Radio engineering and electronics 56(1) 5-19 [5] Schmitt M, Tupin F and Xiang Zhu X 2017 Fusion of SAR and optical remote sensing data — Challenges and recent trends IEEE International Geoscience and Remote Sensing Symposium (IGARSS) DOI: 10.1109/IGARSS.2017.8128239 [6] Kopenkov V N 2013 Designing a procedure for local image processing based on hierarchical regression designed for processing satellite images tasks Collection of scientific articles "Modern problems of remote sensing of the Earth from space" 10 2284-2295 V International Conference on "Information Technology and Nanotechnology" (ITNT-2019) 340 Image Processing and Earth Remote Sensing V N Kopenkov [7] Hughes L H, Auer S and Schmitt M 2018 Investigation of joint visibility between SAR and optical images of urban environments ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences (Riva del Garda, Italy) IV-2 [8] Mandanici E, Bitelli G 2016 Preliminary Comparison of Sentinel-2 and Landsat 8 Imagery for a Combined Use Remote Sensing 8(12) 1014 [9] Kopenkov V N 2014 The method of stopping the process of building hierarchical regression in the implementation of computational procedures for local image processing Computer Optics 38(2) 351-355 [10] Bagautdinov R S, Kopenkov V N, Myshkin V N, Sergeev V V and Tribunsky S A 2015 Investigation of the possibility of using satellite images to identify archaeological objects Computer Optics 39(3) 439-444 DOI: 10.18287/0134-2452-2015-39-3-439-444 [11] Kopenkov V N, Myasnikov V V 2016 Development of Algorithm for automatic construction of computational procedure of local image processing, based on the usage of hierarchical regression Computer Optics 40(5) 713-720 DOI: 10.18287/2412-6179-2016-40-5-713-720 Acknowledgments The reported study was funded by RFBR according to the research project № 16-29-09494 ofi_m, 17- 29-03190 ofi, 18-07-00748, as well as the Ministry of Science and Higher Education of the Russian Federation as part of the work on the State task of the Federal Research Center "Crystallography and Photonics" of the Russian Academy of Sciences (agreement № 007-GZ/Ch3363/26). V International Conference on "Information Technology and Nanotechnology" (ITNT-2019) 341