=Paper=
{{Paper
|id=Vol-2391/paper46
|storemode=property
|title=Combined usage of the optical and radar remote sensing data in territory monitoring tasks
|pdfUrl=https://ceur-ws.org/Vol-2391/paper46.pdf
|volume=Vol-2391
|authors=Vasiliy Kopenkov
}}
==Combined usage of the optical and radar remote sensing data in territory monitoring tasks ==
https://ceur-ws.org/Vol-2391/paper46.pdf
Combined usage of the optical and radar remote sensing data
in territory monitoring tasks
V N Kopenkov1
1
Samara National Research University, Moskovskoe Shosse, 34А, Samara, Russia, 443086
e-mail: vkop@geosamara.ru
Abstract. At the present time, a lot of problems in a sphere of fundamental sciences as well as
technical and applied tasks can be solved only with the use of satellite images, since their usage
reduces material, financial and time costs significantly in comparison with traditional methods.
One of the modern integrated approach remote sensing processing is to join the measurements
obtained from the various sources, such as optical and radar sensors, allowing to achieve a gain
in comparison with independent processing due to the extension of the information volume and
the opportunities of data acquisition (weather conditions, spectral ranges, etc.). However,
methods of digital processing and interpretation of radar data, as well as qualitative and proven
methods and algorithms for joint processing of optical and radar satellite images, has not
sufficiently been well developed yet. Therefore, the development of new methods and
information technology of joint analysis and interpretation of optical and radar data which are a
major issue of the current paper, are certainly relevant. The paper presents an information
technology for joint processing of optical and radar satellite imagery, based on training the
processing procedure based on the reference values of data from sensors of the one type
(optical data), followed by applying to both data types: optical and SAR data.
1. Introduction
Today, a lot of problems of fundamental sciences, as well as technical and applied tasks can be solved
effectively only with help of space vehicles, since their usage significantly reduces the material,
financial and time costs compared with traditional methods. Recently, it is possible to receive multi-
spectral aerospace information in the optical and radio bands in a regular way. At the same time, the
tasks of determining various characteristics of the earth’s surface, monitoring and analyzing of
emergency, classifying and analyzing of territories as a whole are becoming increasingly important. A
special place is occupied by the task of classifying the underlying surface and species composition.
Conventionally, to solve this task images obtained in the optical range of the electromagnetic spectrum
used as input data. Classic approaches which are successfully used for the processing of optical images
do not give positive results when working with data obtained by synthetic aperture radars. The
recognition of vegetation types, borders of territories, the detection and classification of objects on the
base of radar images is practically not performed, because approaches to processing and thematic
interpretation of radar data are not sufficiently well developed and investigated. For example, all
known algorithms for automatic classification are adapted to the Gaussian model of the distribution of
additive noise arising only on optical images. High-resolution radar images formed by synthetic
V International Conference on "Information Technology and Nanotechnology" (ITNT-2019)
Image Processing and Earth Remote Sensing
V N Kopenkov
aperture radars as a result of irradiating the surface of the Earth with coherent waves are usually
distorted by multiplicative noise, known as speckle noise. This kind of noise is a serious problem
during the processing of radar images on the base of classic methods. Therefore, we need image
processing and analysis methods adapted to the physical and technical features of radar imaging.
2. The scheme of combining the radar and optical data
Traditionally, the technology of combining and processing of mixed remote sensing data consists of 3
main subtasks: independent preliminary processing, matching and combining satellite images obtained
by various sensors in the optical and microwave ranges, detecting differences and further processing
of remote sensing data.
Within the framework of the first task, the well-known solutions combine the methods of
interpolation of data on a non-uniform grid, image reconstruction frequency methods, adaptive
filtering [1], radiometric, geometric and polarimetric calibration (for polarimetric radar data) [2],
image enhancement operations [1], reduction of speckle noise on a radar image, orthotransformation
taking into account a digital elevation model, etc. Existing approaches ordinarily solve this task using
assumptions about the same characteristics of imaging systems, which have similar spatial resolution
and the same spectral ranges of signal acquisition and receiving.
The task of matching heterogeneous images are needed for the realization of a further goal − joint
analysis of radar and optical satellite imagery refers to the data processing and analysis area [3-5]. In
this case, special attention should be paid to the choice of the merging method. Because the type and
amount of preliminary data preparation depend on it. For example, in the case of point-oriented
methods, high-precision geocoding and the further combination of all images is very important. In the
case of object processing the preliminary detection, selection and, possibly, recognition of objects on
images, as the objects observed on radar images often look completely different than on optical ones,
since the scattering of the radio signal and sunlight by objects can differ significantly. Radar and
optical observations are complementary, and this fact may have a different nature depending on the
problem being solved. Their joint processing improves the ability to detect certain objects and measure
the parameters of the underlying territory. The general classical scheme of combining radar and
optical data is shown in Figure 1.
Herewith, to solve practical tasks, in most cases, the combination of optical and radar data is
implemented in two ways.
In a situation where optical data is not available, radar data can be a good alternative. For joint
processing, in this case, data is orthotransformed, and reduced to comparable resolution (or the optical
image may have a lower resolution since the imaging capabilities of such images are higher). After
that, points or objects in both images are analyzed to solve the problem of identifying differences and
unchanged regions, as well as for further analysis of remote sensing data.
The second data matching approach is based on the RGB ↔ HIS conversion, which allows you to
combine terrain information obtained in two different ranges on one output image. First, the three
most informative spectral channels RGB → HIS (brightness-saturation-hue color) are converted, then
the brightness is replaced with a radar image on the resulting image and the inverse transformation is
performed, which requires a balanced stretching of the channels to improve the quality of the final
representation in RGB. Using such images, it is possible to determine the roughness (low/high
vegetation, flooded/not-flooded area), and the structure of the territory, which is necessary for
geological tasks.
However, both the first and second version of the data combination is not suitable for the
qualitative solution of the underlying surface classification and analysis tasks.
3. The technology of joint processing of optical and radar data based on a training dataset
Analysis and processing of images obtained by dissimilar remote sensing sensors, with spatial and
spectral resolution improving, usually involves the formation of individual models of observation for
each of the sensors, which is not always possible and requires a large amount of additional a priori
information about the observation model. In addition, it is extremely difficult to develop an automatic
V International Conference on "Information Technology and Nanotechnology" (ITNT-2019) 335
Image Processing and Earth Remote Sensing
V N Kopenkov
adaptive technology for processing dissimilar remote sensing data on the basis of a variety of
particular observation models.
Figure 1. The classical scheme of combining radar and optical data.
Therefore, the methods of nonlinear analysis and data interpretation, such as neural networks or
hierarchical structures or Approximate Local Correlation Integral (aLOCI) are of the greatest interest
for investigation. These methods can be used without definite assumptions about the source data. In
addition, aLOCI uses quadtrees to increase the efficiency of calculations in the multidimensional case.
Among these methods in our task we intend to use a multidimensional classification based on optical
data, since the solution of the microwave signal analysis problem in the general case can be made in
the form of a classification with training, based on features in the form of optical data, observation
results and a priori information about the underlying surface [6-7].
At the same time, it is advisable to use standard methods for extracting features and classification
algorithms for multidimensional data, because the interest in improving the quality of evaluation of
thematic classification. These methods most often include features obtained as a result of the principal
component method (PCA) and the Nonparametric Weighted Feature Extraction (NWFE) method, and
such classifiers as the Support Vector Method (SVM) and Random Forest (RF) [5-6]. The developed
algorithm for the local processing of satellite images based on hierarchical regression [6] allows us to
accelerate data processing by using hierarchical data structures and in a multidimensional case, the
performance is comparable to or superior to other classification methods, in particular, neural
networks. Finally, the assessment of the quality of the results of classification and detection of changes
in the territory by received dissimilar satellite images relates to the field of image analysis based on
the values of one remote sensing sensor using the values of another sensor (for example, calculating of
the regression function) [7,10-11].
Thus, the construction of the technology is supposed to be carried out in three stages: data
preparation, combination and analysis of the image sequence. At the first stage, the use of sensor-
V International Conference on "Information Technology and Nanotechnology" (ITNT-2019) 336
Image Processing and Earth Remote Sensing
V N Kopenkov
dependent methods of processing and combining (spatial) data is assumed, i.e. features of the source
of satellite images (optical or radar) are taken into account, but it is assumed that the observed territory
is the same. After that, the radar images "are brought to the kind" of optical (or optical images "are
brought to the kind" of radar, depending on the task) based on the features and non-linear regression
models. And finally, a sequence of images with similar spectral characteristics is formed that makes it
possible to implement the solution of thematic processing tasks such as classification, change
detection and etc.
In this paper, to form an aggregate set of satellite images that are suitable as source data for
classification and analysis of changes, we propose a technology for converting (point-by-point
processing based on reference values) data of the one type (conceivably radar) based on a training
sample consisting of a data set of another type (optical images). Regression models of processing
based on a decision tree for merging extracted SAR (Synthetic-aperture radar) radars and optical
sensor values are assumed to be used here. Hierarchical structures in the form of a decision tree are not
parametric and therefore do not require an unambiguous description of the desired classes. Herewith,
the final processing procedure consists of a set of explicit hierarchical conditions (rules, thresholds of
functions) applied to the input data intervals, in our case SAR and optical functions, as reference
values. This allows classifying the pixels (area sampling) of the radar image as a function of optical
data. Decision tree classifiers have been successfully applied to classify the underlying surface. The
general scheme of data conversion (in the process of applying the processing procedure), oriented on a
priori information about the desired processing result (presented in [6, 10]), is illustrated by the
diagrams presented in Figure 2.
Figure 2. The scheme of training and usage of processing procedures, based on the training dataset.
Considering a large number of matched pairs of image pixels (zi, xi) as a training sample, we build
K −1
the signal processing procedure Ф, which associates the feature vector { yk ( n,[p])} calculated on
k =0
the base of discrete samples of the input signal X, the output value of the signal Z. The sets of features,
parameters of the processing procedure construction, as well as the type of functions of the elementary
regression in the terminal nodes can be adjusted according to the results of the evaluation of the
quality of processing.
To evaluate the generalization ability of the constructed procedure, the full sliding control
functional is used:
1 N
Q st ( µ (Ω), Ω) = ∑ν (µ (Ω ns ), Ω tn ),
N n=1
where Ω is the final set of objects for learning (dataset), µ (Ω) is the algorithm (method) of training
on a dataset Ω , (Ω ns , Ω tn ), n = 1,2,..., N are the variants of dividing the dataset into training and control
parts.
V International Conference on "Information Technology and Nanotechnology" (ITNT-2019) 337
Image Processing and Earth Remote Sensing
V N Kopenkov
ν ( µ (Ω ns ), Ω tn ) is the error rate of the algorithm µ (Ω ns ) , constructed on the base of the set Ω ns and
checked by using the set Ω tn .
Receiving information from different sensor types leads to a shift in the position of sensor cells
relative to objects due to the differences in the orbital parameters of the satellites and due to the
difference in the physical laws of pixel formation in the image (resolution is dissimilar for optical and
radar data). It is obvious that individual sensors cover the observed area with grids with different
parameters that can be described as a relative motion within the coordinate system of the observed
scene. As for the coordinate system of the observed scene, you can use some projection coordinate
system in which each pixel is represented as an area of finite size, assuming that the coordinates of the
image pixel are the coordinates of the center of the pixel. This view does not contradict the processing
technology based on decision trees since the latter can use the integral values in the spectral channels
over the local window as attributes for the input and output pixels.
In this way, the technology of forming a sequence of consistent and combined dissimilar remote
sensing images is used to obtain features for the tasks of land use classification according to remote
sensing data, as well as for analyzing changes on the sequence of images.
4. Evaluation of the training dataset volume and experimental research
Is quite logical is the fact that when working with images solution of the problem of constructing the
processing procedures taking into account all combination of training and testing dataset is
unrealizable in practice because of the giant of busting on various combinations of datasets. Therefore,
it is necessary to develop a method to determine the possibility of stopping a forming, or the need to
continue training and testing datasets generation on the base of a finite number of them. It is obvious
that for every task there is an optimal complexity of the model, at which the best quality of
generalization is achieved.
In paper [10], the technology of decision making about stopping the process of generating various
combinations of training and control datasets was proposed, according to which to make a decision of
stopping generation of various combinations of training and control datasets, and about the transition
to the next subset of features it is necessary to calculate confidence intervals for the expectation of a
Poisson distribution with parameter λ : Bin(n, λ ) ≈ P (λ ) for the functional full cross-validation on
n
a datasets N1 and N2 in the form:
τ1−α / 2 λ1 τ λ τ λ τ λ
λ1 − , λ1 + 1−α / 2 1 λ 2 − 1−α / 2 2 , λ 2 + 1−α / 2 2 ,
N1 N1 N2 N 2
where τ1−α /2 – quantile of distribution N 0,1 for level 1 − α / 2 (α = 1 − γ ).
The decision of stopping of a generation of various combinations of datasets and about the
transition to the next subset of features is taken at a time when a separation of calculated confidence
intervals on adjacent steps is achieved. The presented technology of decision about stopping the brute
force bust process allows estimating the amount and possible combinations of training and control data
sets used in experimental studies.
As data used as training and control datasets, we used sets of images (free data) from Landsat-7 and
Sentinel 1-2 satellites for different time periods, as well as Radarsat-2 data available at the center of
remote sensing data of Samara University. Datasets from Landsat-7, Spot 6/7, Aist, Resurs-P, Pleiades
and coverage with Google and Yandex data were used as reference data in the optical range.
Experimental study on the applicability of the technology to dissimilar datasets (optical and radar
data) are in progress, so the final results are not presented in this article. Experimental studies of the
feasibility of individual stages of processing, classification and analysis of remote sensing data on the
basis of hierarchical structures built in accordance with the principles of learning "by precedent" are
presented in the author's works [6, 10, 11].
To carry out our experimental studies, we used sets of images from such space vehicles as Spot 7,
Aist-2D, Resurs-P, Sentinel-2 in the optical range (more than 40 test images) and radar images from
V International Conference on "Information Technology and Nanotechnology" (ITNT-2019) 338
Image Processing and Earth Remote Sensing
V N Kopenkov
the Radarsat2 and Sentinel-1 satellites (more than 20 scenes). All data were obtained in the spring-
summer period in 2010-2018. All images were radiometric and geometrically corrected, brought to a
similar resolution and matched (geographically bounded) with an accuracy of about 1 to 2 pixels. All
pre-processing and low-level processing task was implemented in ScanEx ImageProcessor software.
Examples of images from different spacecraft are shown in Figure 3.
Aist-2D Spot-7 Geoton Sentinel-2
04/2016 - 10/2016 05/2016 - 06/2016 05/2016 - 07/2016 04/2018 - 10/2018
Radarsat2 2010-2015 Sentinel-1 04\2018 - 10/2018
Figure 3. Training and testing datasets in a form of images from different spacecrafts.
All experimental studies can be divided into 3 groups:
1) Optical data as a training sample, and optical data from the same satellite as test data.
2) Optical data from different satellites as a training sample, and optical data from another satellite
as test data.
3) Optical and radar data from different satellites as a training sample, and optical data from
another satellite as test data.
The results of experimental studies of the quality of the pixel-by-pixel classification are shown in
Table 1. In general, it can be noted that the technology of joint processing of optical and local data
based on hierarchical structures with training on a priori information is quite efficient.
At the same time, we note that if the training sample contains data of the same type as that used as
a test data, then the quality of the classification is quite high. It should be noted that the classification
based on radar data shows a very low quality of classification since the features in the form of the
V International Conference on "Information Technology and Nanotechnology" (ITNT-2019) 339
Image Processing and Earth Remote Sensing
V N Kopenkov
original image pixels are not sufficiently informative. Therefore, it is necessary to use other features in
the form of generalized coefficients over a certain local image area. Selection and optimization of such
features is the goal of further work.
Table 1. Quality of per-pixel classification.
Images for test
Training dataset Sentinel-2 Aist-2D Spot-7, Geoton Radarsat2, Sentinel-1
Sentinel-2 96-98% 72-87% 67-79%
Sentinel-2 (70%) + Aist-2D 93-98% 90-95% 72-84%
Sentinel-2 (50%) + Aist-2D + Spot-7 + 89-97% 86-94% 84-91%
Geoton
Sentinel-2 (80%) + Radarsat2 + Sentinel-1 93-98% 76-84% 62-78% 46-72%
Optical satellites (80%) + Radarsat data 90-97% 87-92% 78-84% 52-74%
(20%)
Optical satellites (50%) + Radarsat data 86-92% 76-82% 76-80% 57-82%
(50%)
No experiments were carried out to restore (classify) pixels of optical images by using the pixel
values only of radar images because it is impossible to obtain adequate results for various spectral
channels (RGB) from radar data only.
5. Conclusion
This article presents a new information technology for joint processing of optical and radar satellite
imagery. The technology is based on learning procedure that uses the data from sensors of the one type
(optical in the basic approximation) as reference values and then the technology is applied to the data
of both types: the same one (optical) and the another data type (radar data). Experiments on testing of
the developed regression models of processing based on a decision tree for joint processing of data
obtained in the optical and SAR ranges confirm the efficiency and effectiveness of the technology.
That, as a result, allows classifying the samples (areas of pixels) of the radar image as a function of
optical data. The earlier experiments [6, 10, 11], as well as experiments in the framework implemented
in this paper, showed the efficiency of using decision trees as classifiers of the underlying surface and
demonstrated a high quality of classification, which suggests the possibility of applying this approach
to the problem of joint analysis of optical and radar data.
6. References
[1] Wan X, Yang Y 2010 Super-resolution image reconstruction Computer Application and System
Modeling (ICCASM) 8 351-355
[2] Nikolsky D B 2016 Methods for processing radar data Geomatics URL: http://
geomatica.ru/clauses/48/ (10/01/2018)
[3] Kopenkov V N, Sergeev V V, Soifer V A and Chernov A V 2017 Remote sensing of the Earth
and geographic information systems Bulletin RFBR 95(3) 78-96 DOI: 10.22204/2410-4639-
2017-095-03-78-96
[4] Zakharova L N, Zakharov A I, Sorochinsky M V, Ryabokon G P and Leonov V M 2011 Joint
data analysis of optical and radar sensors: opportunities. restrictions, prospects Radio
engineering and electronics 56(1) 5-19
[5] Schmitt M, Tupin F and Xiang Zhu X 2017 Fusion of SAR and optical remote sensing data —
Challenges and recent trends IEEE International Geoscience and Remote Sensing Symposium
(IGARSS) DOI: 10.1109/IGARSS.2017.8128239
[6] Kopenkov V N 2013 Designing a procedure for local image processing based on hierarchical
regression designed for processing satellite images tasks Collection of scientific articles
"Modern problems of remote sensing of the Earth from space" 10 2284-2295
V International Conference on "Information Technology and Nanotechnology" (ITNT-2019) 340
Image Processing and Earth Remote Sensing
V N Kopenkov
[7] Hughes L H, Auer S and Schmitt M 2018 Investigation of joint visibility between SAR and
optical images of urban environments ISPRS Annals of the Photogrammetry, Remote Sensing
and Spatial Information Sciences (Riva del Garda, Italy) IV-2
[8] Mandanici E, Bitelli G 2016 Preliminary Comparison of Sentinel-2 and Landsat 8 Imagery for a
Combined Use Remote Sensing 8(12) 1014
[9] Kopenkov V N 2014 The method of stopping the process of building hierarchical regression in
the implementation of computational procedures for local image processing Computer Optics
38(2) 351-355
[10] Bagautdinov R S, Kopenkov V N, Myshkin V N, Sergeev V V and Tribunsky S A 2015
Investigation of the possibility of using satellite images to identify archaeological objects
Computer Optics 39(3) 439-444 DOI: 10.18287/0134-2452-2015-39-3-439-444
[11] Kopenkov V N, Myasnikov V V 2016 Development of Algorithm for automatic construction of
computational procedure of local image processing, based on the usage of hierarchical
regression Computer Optics 40(5) 713-720 DOI: 10.18287/2412-6179-2016-40-5-713-720
Acknowledgments
The reported study was funded by RFBR according to the research project № 16-29-09494 ofi_m, 17-
29-03190 ofi, 18-07-00748, as well as the Ministry of Science and Higher Education of the Russian
Federation as part of the work on the State task of the Federal Research Center "Crystallography and
Photonics" of the Russian Academy of Sciences (agreement № 007-GZ/Ch3363/26).
V International Conference on "Information Technology and Nanotechnology" (ITNT-2019) 341