At the present time, a lot of problems in a sphere of fundamental sciences as well as technical and applied tasks can be solved only with the use of satellite images, since their usage reduces material, financial and time costs significantly in comparison with traditional methods. One of the modern integrated approach remote sensing processing is to join the measurements obtained from the various sources, such as optical and radar sensors, allowing to achieve a gain in comparison with independent processing due to the extension of the information volume and the opportunities of data acquisition (weather conditions, spectral ranges, etc.). However, methods of digital processing and interpretation of radar data, as well as qualitative and proven methods and algorithms for joint processing of optical and radar satellite images, has not sufficiently been well developed yet. Therefore, the development of new methods and information technology of joint analysis and interpretation of optical and radar data which are a major issue of the current paper, are certainly relevant. The paper presents an information technology for joint processing of optical and radar satellite imagery, based on training the processing procedure based on the reference values of data from sensors of the one type (optical data), followed by applying to both data types: optical and SAR data.
Today, a lot of problems of fundamental sciences, as well as technical and applied tasks can be solved effectively only with help of space vehicles, since their usage significantly reduces the material, financial and time costs compared with traditional methods. Recently, it is possible to receive multispectral aerospace information in the optical and radio bands in a regular way. At the same time, the tasks of determining various characteristics of the earth’s surface, monitoring and analyzing of emergency, classifying and analyzing of territories as a whole are becoming increasingly important. A special place is occupied by the task of classifying the underlying surface and species composition. Conventionally, to solve this task images obtained in the optical range of the electromagnetic spectrum used as input data. Classic approaches which are successfully used for the processing of optical images do not give positive results when working with data obtained by synthetic aperture radars. The recognition of vegetation types, borders of territories, the detection and classification of objects on the base of radar images is practically not performed, because approaches to processing and thematic interpretation of radar data are not sufficiently well developed and investigated. For example, all known algorithms for automatic classification are adapted to the Gaussian model of the distribution of additive noise arising only on optical images. High-resolution radar images formed by synthetic aperture radars as a result of irradiating the surface of the Earth with coherent waves are usually distorted by multiplicative noise, known as speckle noise. This kind of noise is a serious problem during the processing of radar images on the base of classic methods. Therefore, we need image processing and analysis methods adapted to the physical and technical features of radar imaging.
Traditionally, the technology of combining and processing of mixed remote sensing data consists of 3 main subtasks: independent preliminary processing, matching and combining satellite images obtained by various sensors in the optical and microwave ranges, detecting differences and further processing of remote sensing data.
Within the framework of the first task, the well-known solutions combine the methods of
interpolation of data on a non-uniform grid, image reconstruction frequency methods, adaptive
filtering [
The task of matching heterogeneous images are needed for the realization of a further goal − joint
analysis of radar and optical satellite imagery refers to the data processing and analysis area [
Herewith, to solve practical tasks, in most cases, the combination of optical and radar data is implemented in two ways.
In a situation where optical data is not available, radar data can be a good alternative. For joint processing, in this case, data is orthotransformed, and reduced to comparable resolution (or the optical image may have a lower resolution since the imaging capabilities of such images are higher). After that, points or objects in both images are analyzed to solve the problem of identifying differences and unchanged regions, as well as for further analysis of remote sensing data.
The second data matching approach is based on the RGB ↔ HIS conversion, which allows you to combine terrain information obtained in two different ranges on one output image. First, the three most informative spectral channels RGB → HIS (brightness-saturation-hue color) are converted, then the brightness is replaced with a radar image on the resulting image and the inverse transformation is performed, which requires a balanced stretching of the channels to improve the quality of the final representation in RGB. Using such images, it is possible to determine the roughness (low/high vegetation, flooded/not-flooded area), and the structure of the territory, which is necessary for geological tasks.
However, both the first and second version of the data combination is not suitable for the qualitative solution of the underlying surface classification and analysis tasks.
Analysis and processing of images obtained by dissimilar remote sensing sensors, with spatial and spectral resolution improving, usually involves the formation of individual models of observation for each of the sensors, which is not always possible and requires a large amount of additional a priori information about the observation model. In addition, it is extremely difficult to develop an automatic adaptive technology for processing dissimilar remote sensing data on the basis of a variety of particular observation models.
Therefore, the methods of nonlinear analysis and data interpretation, such as neural networks or
hierarchical structures or Approximate Local Correlation Integral (aLOCI) are of the greatest interest
for investigation. These methods can be used without definite assumptions about the source data. In
addition, aLOCI uses quadtrees to increase the efficiency of calculations in the multidimensional case.
Among these methods in our task we intend to use a multidimensional classification based on optical
data, since the solution of the microwave signal analysis problem in the general case can be made in
the form of a classification with training, based on features in the form of optical data, observation
results and a priori information about the underlying surface [
At the same time, it is advisable to use standard methods for extracting features and classification
algorithms for multidimensional data, because the interest in improving the quality of evaluation of
thematic classification. These methods most often include features obtained as a result of the principal
component method (PCA) and the Nonparametric Weighted Feature Extraction (NWFE) method, and
such classifiers as the Support Vector Method (SVM) and Random Forest (RF) [
Thus, the construction of the technology is supposed to be carried out in three stages: data preparation, combination and analysis of the image sequence. At the first stage, the use of sensordependent methods of processing and combining (spatial) data is assumed, i.e. features of the source of satellite images (optical or radar) are taken into account, but it is assumed that the observed territory is the same. After that, the radar images "are brought to the kind" of optical (or optical images "are brought to the kind" of radar, depending on the task) based on the features and non-linear regression models. And finally, a sequence of images with similar spectral characteristics is formed that makes it possible to implement the solution of thematic processing tasks such as classification, change detection and etc.
In this paper, to form an aggregate set of satellite images that are suitable as source data for
classification and analysis of changes, we propose a technology for converting (point-by-point
processing based on reference values) data of the one type (conceivably radar) based on a training
sample consisting of a data set of another type (optical images). Regression models of processing
based on a decision tree for merging extracted SAR (Synthetic-aperture radar) radars and optical
sensor values are assumed to be used here. Hierarchical structures in the form of a decision tree are not
parametric and therefore do not require an unambiguous description of the desired classes. Herewith,
the final processing procedure consists of a set of explicit hierarchical conditions (rules, thresholds of
functions) applied to the input data intervals, in our case SAR and optical functions, as reference
values. This allows classifying the pixels (area sampling) of the radar image as a function of optical
data. Decision tree classifiers have been successfully applied to classify the underlying surface. The
general scheme of data conversion (in the process of applying the processing procedure), oriented on a
priori information about the desired processing result (presented in [
Considering a large number of matched pairs of image pixels (zi, xi) as a training sample, we build the signal processing procedure Ф, which associates the feature vector { yk ( n,[p])}kK=−01 calculated on the base of discrete samples of the input signal X, the output value of the signal Z. The sets of features, parameters of the processing procedure construction, as well as the type of functions of the elementary regression in the terminal nodes can be adjusted according to the results of the evaluation of the quality of processing.
To evaluate the generalization ability of the constructed procedure, the full sliding control functional is used:
Q st (µ (Ω), Ω) = 1 ∑Nν (µ (Ωns ), Ωtn ),
N n=1 where Ω is the final set of objects for learning (dataset), µ (Ω) is the algorithm (method) of training on a dataset Ω , (Ωns , Ωtn ), n = 1,2,..., N are the variants of dividing the dataset into training and control parts. ν (µ (Ωns ), Ωtn ) is the error rate of the algorithm µ (Ωns ) , constructed on the base of the set Ωns and checked by using the set Ωtn .
Receiving information from different sensor types leads to a shift in the position of sensor cells relative to objects due to the differences in the orbital parameters of the satellites and due to the difference in the physical laws of pixel formation in the image (resolution is dissimilar for optical and radar data). It is obvious that individual sensors cover the observed area with grids with different parameters that can be described as a relative motion within the coordinate system of the observed scene. As for the coordinate system of the observed scene, you can use some projection coordinate system in which each pixel is represented as an area of finite size, assuming that the coordinates of the image pixel are the coordinates of the center of the pixel. This view does not contradict the processing technology based on decision trees since the latter can use the integral values in the spectral channels over the local window as attributes for the input and output pixels.
In this way, the technology of forming a sequence of consistent and combined dissimilar remote sensing images is used to obtain features for the tasks of land use classification according to remote sensing data, as well as for analyzing changes on the sequence of images.
Is quite logical is the fact that when working with images solution of the problem of constructing the processing procedures taking into account all combination of training and testing dataset is unrealizable in practice because of the giant of busting on various combinations of datasets. Therefore, it is necessary to develop a method to determine the possibility of stopping a forming, or the need to continue training and testing datasets generation on the base of a finite number of them. It is obvious that for every task there is an optimal complexity of the model, at which the best quality of generalization is achieved.
In paper [
The decision of stopping of a generation of various combinations of datasets and about the transition to the next subset of features is taken at a time when a separation of calculated confidence intervals on adjacent steps is achieved. The presented technology of decision about stopping the brute force bust process allows estimating the amount and possible combinations of training and control data sets used in experimental studies.
As data used as training and control datasets, we used sets of images (free data) from Landsat-7 and Sentinel 1-2 satellites for different time periods, as well as Radarsat-2 data available at the center of remote sensing data of Samara University. Datasets from Landsat-7, Spot 6/7, Aist, Resurs-P, Pleiades and coverage with Google and Yandex data were used as reference data in the optical range.
Experimental study on the applicability of the technology to dissimilar datasets (optical and radar
data) are in progress, so the final results are not presented in this article. Experimental studies of the
feasibility of individual stages of processing, classification and analysis of remote sensing data on the
basis of hierarchical structures built in accordance with the principles of learning "by precedent" are
presented in the author's works [
To carry out our experimental studies, we used sets of images from such space vehicles as Spot 7, Aist-2D, Resurs-P, Sentinel-2 in the optical range (more than 40 test images) and radar images from the Radarsat2 and Sentinel-1 satellites (more than 20 scenes). All data were obtained in the springsummer period in 2010-2018. All images were radiometric and geometrically corrected, brought to a similar resolution and matched (geographically bounded) with an accuracy of about 1 to 2 pixels. All pre-processing and low-level processing task was implemented in ScanEx ImageProcessor software. Examples of images from different spacecraft are shown in Figure 3.
04/2016 - 10/2016
Radarsat2 2010-2015 Sentinel-1 04\2018 - 10/2018 Figure 3. Training and testing datasets in a form of images from different spacecrafts.
1) Optical data as a training sample, and optical data from the same satellite as test data. 2) Optical data from different satellites as a training sample, and optical data from another satellite as test data.
3) Optical and radar data from different satellites as a training sample, and optical data from another satellite as test data.
The results of experimental studies of the quality of the pixel-by-pixel classification are shown in Table 1. In general, it can be noted that the technology of joint processing of optical and local data based on hierarchical structures with training on a priori information is quite efficient.
At the same time, we note that if the training sample contains data of the same type as that used as a test data, then the quality of the classification is quite high. It should be noted that the classification based on radar data shows a very low quality of classification since the features in the form of the original image pixels are not sufficiently informative. Therefore, it is necessary to use other features in the form of generalized coefficients over a certain local image area. Selection and optimization of such features is the goal of further work.
No experiments were carried out to restore (classify) pixels of optical images by using the pixel values only of radar images because it is impossible to obtain adequate results for various spectral channels (RGB) from radar data only.
This article presents a new information technology for joint processing of optical and radar satellite
imagery. The technology is based on learning procedure that uses the data from sensors of the one type
(optical in the basic approximation) as reference values and then the technology is applied to the data
of both types: the same one (optical) and the another data type (radar data). Experiments on testing of
the developed regression models of processing based on a decision tree for joint processing of data
obtained in the optical and SAR ranges confirm the efficiency and effectiveness of the technology.
That, as a result, allows classifying the samples (areas of pixels) of the radar image as a function of
optical data. The earlier experiments [
The reported study was funded by RFBR according to the research project № 16-29-09494 ofi_m, 1729-03190 ofi, 18-07-00748, as well as the Ministry of Science and Higher Education of the Russian Federation as part of the work on the State task of the Federal Research Center "Crystallography and Photonics" of the Russian Academy of Sciences (agreement № 007-GZ/Ch3363/26).